Training methods for video redirection models, video redirection methods and devices

By adjusting the network model to adapt to the video redirection model of different target objects, the problem of weak generalization ability in the existing technology is solved, and effective adaptation to different target objects and video quality improvement is achieved.

CN114302023BActive Publication Date: 2024-11-26TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202111189312.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-10-12
Publication Date
2024-11-26
Estimated Expiration
2041-10-12

AI Technical Summary

Technical Problem

The existing video redirection model has weak generalization ability, which leads to the need to train different models for different target objects, which is expensive and less practical.

Method used

By obtaining data of sample objects, including reference action images, reference pose images, action videos and pose videos, the network model is adjusted to generate a video redirection model with strong generalization capabilities, which can be applied to different target objects.

Benefits of technology

It saves training costs, improves the practicality of the model and the quality of video redirection, and achieves effective adaptation to different target objects.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114302023B_ABST
    Figure CN114302023B_ABST
Patent Text Reader

Abstract

The present application discloses a training method, a video redirection method and a device for a video redirection model, and belongs to the field of artificial intelligence technology. The method includes: obtaining a reference action image, a reference posture image, a first action video, a first posture video, a second action video and a second posture video; determining a third action video based on the reference action image, the reference posture image and the first posture video based on the first network model; adjusting the first network model based on the first action video and the third action video to obtain a second network model; determining a fourth action video based on the reference action image, the reference posture image and the second posture video based on the second network model; adjusting the second network model based on the second action video and the fourth action video to obtain a third network model; in response to meeting the training end condition, using the third network model as a video redirection model. The generalization ability and practicality of the video redirection model are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and in particular to a video redirection model training method, a video redirection method and a device. Background Art

[0002] With the rise of video applications, various video processing technologies have emerged, among which video retargeting technology is a widely-watched video processing technology. Video retargeting technology, also known as motion transfer technology, is a technology that transfers the motion of the original object in the original video to the target object, so that the target object in the target video imitates the motion of the original object.

[0003] The related technology uses a video redirection model to achieve video redirection. When training the video redirection model, first, a long video containing several thousand frames of images of the target object is collected, and the long video is used as training data. A video redirection model is trained based on the training data. The video redirection model is a personalized model of the target object, which can obtain the target video of the target object based on the original video.

[0004] Since the video redirection model in the related art is a personalized model of the target object, the generalization ability of the video redirection model is weak. Therefore, different video redirection models need to be trained for different target objects, resulting in high overhead and low practicality. Summary of the invention

[0005] The embodiments of the present application provide a video redirection model training method, a video redirection method and a device, which can be used to solve the problems of weak generalization ability and low practicality of the video redirection model in the related art. The technical solution includes the following contents.

[0006] On the one hand, an embodiment of the present application provides a method for training a video redirection model, the method comprising:

[0007] Acquire data of a sample object, wherein the data of the sample object includes a reference action image, a reference posture image, a first action video, a first posture video, a second action video, and a second posture video;

[0008] Determining a third action video based on a first network model according to the reference action image, the reference posture image and the first posture video;

[0009] Adjusting the first network model based on the first action video and the third action video to obtain a second network model;

[0010] Determining a fourth action video based on the second network model according to the reference action image, the reference posture image and the second posture video;

[0011] Adjusting the second network model based on the second action video and the fourth action video to obtain a third network model;

[0012] In response to satisfying the training end condition, the third network model is used as a video redirection model.

[0013] On the other hand, an embodiment of the present application provides a video redirection method, the method comprising:

[0014] Acquire a reference action image of a target object, a reference posture image of the target object, a first action video of the target object, a posture video of the target object, and a posture video of an original object;

[0015] Determining a second action video of the target object based on a video redirection model according to the reference action image of the target object, the reference posture image of the target object and the posture video of the target object, wherein the video redirection model is obtained according to any of the above-mentioned training methods for the video redirection model;

[0016] Adjusting the video redirection model based on the first action video of the target object and the second action video of the target object to obtain the target model;

[0017] Based on the reference action image of the target object, the reference posture image of the target object and the posture video of the original object, a target video is determined based on the target model.

[0018] On the other hand, an embodiment of the present application provides a training device for a video redirection model, the device comprising:

[0019] An acquisition module, configured to acquire data of a sample object, wherein the data of the sample object includes a reference action image, a reference posture image, a first action video, a first posture video, a second action video, and a second posture video;

[0020] a determination module, configured to determine a third action video based on a first network model according to the reference action image, the reference posture image and the first posture video;

[0021] an adjustment module, configured to adjust the first network model based on the first action video and the third action video to obtain a second network model;

[0022] The determination module is further configured to determine a fourth action video based on the second network model according to the reference action image, the reference posture image and the second posture video;

[0023] The adjustment module is further used to adjust the second network model based on the second action video and the fourth action video to obtain a third network model;

[0024] The determination module is further configured to use the third network model as a video redirection model in response to satisfying a training end condition.

[0025] In a possible implementation, the first posture video includes a plurality of posture images;

[0026] The determination module is used to determine the action image corresponding to the first posture image based on the first network model according to the reference action image, the reference posture image and the first posture image in the first posture video; for any posture image in the first posture video except the first posture image, the action image corresponding to any posture image is determined according to the reference action image, the reference posture image, any posture image, the previous posture image of any posture image and the action image corresponding to the previous posture image.

[0027] In one possible implementation, the determination module is used to determine a first action image based on the reference action image, the reference posture image and any posture image; determine a second action image based on the reference action image, any posture image, a previous posture image of any posture image and an action image corresponding to the previous posture image; and determine an action image corresponding to any posture image based on the first action image and the second action image.

[0028] In a possible implementation, the determination module is used to determine a fused posture image according to the reference posture image and any one of the posture images; and to determine the first action image according to the reference action image and the fused posture image.

[0029] In one possible implementation, the determination module is used to extract features of the reference action image, features of any one of the posture images, features of a posture image previous to any one of the posture images, and features of the action image corresponding to the previous posture image; determine fusion features based on the features of the reference action image, features of any one of the posture images, features of a posture image previous to any one of the posture images, and features of the action image corresponding to the previous posture image; and determine the second action image based on the fusion features.

[0030] In a possible implementation, the first action video includes action images corresponding to each posture image in the first posture video, and the third action video includes action images corresponding to each posture image in the first posture video;

[0031] The adjustment module is used to determine the loss value of each first image set, wherein the first image set includes a third action image, a fourth action image, a fifth action image and a sixth action image, wherein the third action image and the fourth action image are action images corresponding to the first adjacent posture images in the first action video, the fifth action image and the sixth action image are action images corresponding to the first adjacent posture images in the third action video, and the first adjacent posture images are two adjacent posture images in the first posture video; based on the loss values ​​of each first image set, the first network model is adjusted to obtain a second network model.

[0032] In a possible implementation, the third action image and the fifth action image are action images corresponding to the same action image in the first action video, and the fourth action image and the sixth action image are action images corresponding to the same action image in the first action video;

[0033] The adjustment module is used to determine, for any first image set, a first loss value based on the third action image and the fifth action image in the any first image set; determine a second loss value based on the fourth action image and the sixth action image in the any first image set; determine a third loss value based on the third action image, the fourth action image, the fifth action image and the sixth action image in the any first image set; and determine the loss value of any first image set based on the first loss value, the second loss value and the third loss value.

[0034] In a possible implementation, the second posture video includes a plurality of posture images;

[0035] The determination module is used to determine the action image corresponding to the first posture image based on the second network model according to the reference action image, the reference posture image and the first posture image in the second posture video; for any posture image in the second posture video except the first posture image, the action image corresponding to any posture image is determined according to the reference action image, the reference posture image, any posture image, the previous posture image of any posture image and the action image corresponding to the previous posture image.

[0036] In a possible implementation, the second action video includes action images corresponding to each posture image in the second posture video, and the fourth action video includes action images corresponding to each posture image in the second posture video;

[0037] The adjustment module is used to determine the loss value of each second image set, where the second image set includes a seventh action image, an eighth action image, a ninth action image and a tenth action image, where the seventh action image and the eighth action image are action images corresponding to the second adjacent posture images in the second action video, the ninth action image and the tenth action image are action images corresponding to the second adjacent posture images in the fourth action video, and the second adjacent posture images are two adjacent posture images in the second posture video; based on the loss value of each second image set and the parameters of the first network model, the second network model is adjusted to obtain a third network model.

[0038] In a possible implementation, the acquisition module is used to acquire an action video of a sample object; select the reference action image, the first action video, and the second action video from the action video of the sample object; acquire the reference posture image based on the reference action image; acquire the first posture video based on the first action video; and acquire the second posture video based on the second action video.

[0039] In one possible implementation, the determination module is further used to, in response to the training end condition not being met, determine a fourth network model based on the third network model according to the data of the sample object; in response to the training end condition being met, use the fourth network model as a video redirection model.

[0040] On the other hand, an embodiment of the present application provides a video redirection device, the device comprising:

[0041] An acquisition module, used for acquiring a reference action image of a target object, a reference posture image of the target object, a first action video of the target object, a posture video of the target object, and a posture video of an original object;

[0042] a determination module, configured to determine a second action video of the target object based on a video redirection model according to a reference action image of the target object, a reference posture image of the target object, and a posture video of the target object, wherein the video redirection model is obtained according to any of the above-mentioned training methods for the video redirection model;

[0043] an adjustment module, configured to adjust the video redirection model based on the first action video of the target object and the second action video of the target object to obtain the target model;

[0044] The determination module is further configured to determine a target video based on the target model based on a reference action image of the target object, a reference posture image of the target object and a posture video of the original object.

[0045] In a possible implementation, the acquisition module is used to acquire the first action video of the target object and the action video of the original object;

[0046] acquiring a posture video of the target object based on a first action video of the target object, wherein the first action video of the target object includes a reference action image of the target object, and the posture video of the target object includes a reference posture image of the target object;

[0047] A posture video of the original object is acquired based on the action video of the original object.

[0048] On the other hand, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores at least one program code, and the at least one program code is loaded and executed by the processor so that the electronic device implements any of the above-mentioned video redirection model training methods or video redirection methods.

[0049] On the other hand, a computer-readable storage medium is also provided, in which at least one program code is stored, and the at least one program code is loaded and executed by a processor so that a computer implements any of the above-mentioned video redirection model training methods or video redirection methods.

[0050] On the other hand, a computer program or a computer program product is also provided, wherein at least one computer instruction is stored in the computer program or the computer program product, and the at least one computer instruction is loaded and executed by a processor so that the computer implements any of the above-mentioned video redirection model training methods or video redirection methods.

[0051] The technical solution provided by the embodiments of the present application brings at least the following beneficial effects:

[0052] The technical solution provided by the embodiment of the present application is to obtain a video redirection model using the data of the sample object. The video redirection model has strong generalization ability and can be used to generate target videos of different target objects, saving costs and improving practicality. The data of the sample object includes two action videos. In the process of obtaining the video redirection model, the network model is adjusted twice for these two action videos, which strengthens the learning ability of the model and improves the quality of the video output by the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0054] Figure 1 It is a schematic diagram of a training method for a video redirection model and an implementation environment of the video redirection method provided in an embodiment of the present application;

[0055] Figure 2 It is a flow chart of a training method of a video redirection model provided in an embodiment of the present application;

[0056] Figure 3 is a schematic diagram of generating an action image in a third action video provided by an embodiment of the present application;

[0057] Figure 4 is a schematic diagram of determining an action image corresponding to any posture image provided in an embodiment of the present application;

[0058] Figure 5 is a schematic diagram of generating an action image in a fourth action video provided by an embodiment of the present application;

[0059] Figure 6 is a schematic diagram of a training method for a video redirection model provided in an embodiment of the present application;

[0060] Figure 7 is a flow chart of a video redirection method provided by an embodiment of the present application;

[0061] Figure 8 It is a structural schematic diagram of a training device for a video redirection model provided in an embodiment of the present application;

[0062] Fig. 9 It is a structural schematic diagram of a video redirection device provided in an embodiment of the present application;

[0063] Fig.10 It is a structural diagram of a terminal device provided in an embodiment of the present application;

[0064] Fig.11 It is a structural diagram of a server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical solution and advantages of the present application clearer, the implementation mode of the present application will be further described in detail below in conjunction with the accompanying drawings. First, the terms involved in the embodiments of the present application are explained and illustrated.

[0066] Meta-learning: Training a model by learning various tasks so that the model can solve new tasks using only a small number of training samples.

[0067] Few-shot Learning: Training a model to solve a new task with only a small number of training samples.

[0068] See also Figure 1 , Figure 1 is a schematic diagram of a training method for a video redirection model and an implementation environment of the video redirection method provided in an embodiment of the present application, such as Figure 1 The implementation environment shown includes an electronic device 11, and the video redirection model training method and the video redirection method in the embodiment of the present application can be executed by the electronic device 11. Exemplarily, the electronic device 11 can include at least one of a terminal device or a server.

[0069] The terminal device can be at least one of a smart phone, a game console, a desktop computer, a tablet computer, an e-book reader, an MP3 (Moving Picture Experts Group Audio Layer III) player, an MP4 (Moving Picture Experts Group Audio Layer IV) player and a laptop computer.

[0070] The server may be a single server, or a server cluster consisting of multiple servers, or any one of a cloud computing platform and a virtualization center, which is not limited in the embodiments of the present application. The server may be connected to the terminal device through a wired network or a wireless network. The server may have functions such as data processing, data storage, and data transmission and reception, which are not limited in the embodiments of the present application.

[0071] The training method of the video redirection model and the video redirection method provided in the embodiments of the present application are implemented based on artificial intelligence technology. Artificial Intelligence (AI) is a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that the machines have the functions of perception, reasoning and decision-making.

[0072] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. Basic artificial intelligence technologies generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operating / interactive systems, mechatronics and other technologies. Artificial intelligence software technologies mainly include computer vision technology, speech processing technology, natural language processing technology, as well as machine learning / deep learning, autonomous driving, smart transportation and other major directions.

[0073] Computer vision technology (CV) is a science that studies how to make machines "see". To put it more specifically, it refers to the use of cameras and computers to replace human eyes to identify, track and measure targets, and further perform image processing to make computer processing into images that are more suitable for human eye observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and mapping, automatic driving, smart transportation and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0074] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, drones, robots, smart medical care, smart customer service, Internet of Vehicles, automatic driving, smart transportation, etc. I believe that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0075] The solution provided in the embodiments of the present application involves technologies such as a training method for an artificial intelligence video redirection model and a video redirection method, which are specifically illustrated by the following embodiments.

[0076] Based on the above implementation environment, the present application embodiment provides a training method for a video redirection model. Figure 2 Taking the flowchart of a video redirection model training method provided in the embodiment of the present application as an example, the method can be performed by Figure 1 The electronic device 11 in the embodiment is executed. Figure 2 As shown, the method includes steps 201 to 206.

[0077] Step 201 , obtaining data of a sample object, where the data of the sample object includes a reference action image, a reference posture image, a first action video, a first posture video, a second action video, and a second posture video.

[0078] In the embodiment of the present application, the type and quantity of sample objects are not limited. For example, the sample objects are humans, animals, robots, etc., and the number of sample objects is multiple (e.g., 100,000). The data of any sample object consists of six parts, which are reference action image, reference posture image, first action video, first posture video, second action video, and second posture video.

[0079] The reference action image is an action image, the first action video and the second action video each include at least two action images, the action image includes the actual action of the sample object, such as each frame image in the dance video includes the actual action of a person dancing, therefore, each frame image in the dance video is an action image. The reference posture image is a posture image, the first posture video and the second posture video each include at least two posture images, the posture image includes the action of the skeleton model (such as a stickman model) corresponding to the sample object, such as the posture image includes the action of the stickman model dancing.

[0080] Optionally, obtaining data of the sample object includes: obtaining an action video of the sample object; selecting a reference action image, a first action video, and a second action video from the action video of the sample object; obtaining a reference posture image based on the reference action image; obtaining a first posture video based on the first action video; and obtaining a second posture video based on the second action video.

[0081] In the embodiment of the present application, action videos of multiple sample objects are obtained, and the action video of any sample object includes multiple action images, and one sample object corresponds to at least one action video. For the action video of any sample object, a reference action image, a first action video, and a second action video are selected from the action video, wherein the reference action image is any action image in the action video, and the first action video and the second action video are two video clips in the action video.

[0082] Optionally, the first action video and the second action video are two discontinuous video clips in the action video, that is, there is at least one image between the first action video and the second action video in the action video to ensure that the first action video and the second action video contain different actions.

[0083] Afterwards, based on the reference action image, a posture extractor is used to obtain a reference posture image. Similarly, based on the first action video, a posture extractor is used to obtain posture images corresponding to each action image in the first action video to obtain the first posture video. Based on the second action video, a posture extractor is used to obtain posture images corresponding to each action image in the second action video to obtain the second posture video. The posture extractor is not limited in the embodiments of the present application.

[0084] In the embodiment of the present application, the first action video and the second action video extracted from the action video of a sample object are small sample videos with a small number of video frames. The video redirection model is obtained by training with small sample videos, so that the model can be effectively adjusted by a few frames of action of the target object, thereby improving the generalization ability of the model. The model is obtained by training with data of multiple sample objects, so that the model can learn the time series data of each sample object, and the video redirection model is obtained by training with time series-aware meta-learning, thereby improving the generalization ability of the model.

[0085] Step 202, determining a third action video based on a first network model according to the reference action image, the reference posture image and the first posture video.

[0086] In the embodiment of the present application, the data of any sample object includes but is not limited to a reference action image, a reference posture image, and a first posture video. For the data of any sample object, the reference action image, the reference posture image, and the first posture video are input into the first network model, and the first network model outputs the third action video. In this way, the third action video of each sample object is obtained. Among them, the structure and size of the first network model are not limited. The third action video includes at least two action images.

[0087] In one possible implementation, the first posture video includes multiple posture images, and the third action video includes action images corresponding to each posture image in the first posture video; the third action video is determined based on the first network model according to the reference action image, the reference posture image and the first posture video, including: determining the action image corresponding to the first posture image based on the first network model according to the reference action image, the reference posture image and the first posture image in the first posture video; for any posture image except the first posture image in the first posture video, determine the action image corresponding to any posture image according to the reference action image, the reference posture image, any posture image, the previous posture image of any posture image and the action image corresponding to the previous posture image.

[0088] In the embodiment of the present application, after the reference action image, the reference posture image and the first posture video are input into the first network model, for the first posture image in the first posture video, the first network model determines the action image corresponding to the first posture image based on the reference action image, the reference posture image and the first posture image. For the second posture image in the first posture video, the first network model determines the action image corresponding to the second posture image based on the reference action image, the reference posture image, the second posture image, the first posture image (i.e., the previous posture image of the second posture image) and the action image corresponding to the first posture image. For the third posture image in the first posture video, the first network model determines the action image corresponding to the third posture image based on the reference action image, the reference posture image, the third posture image, the second posture image (i.e., the previous posture image of the third posture image) and the action image corresponding to the second posture image. In this way, by analogy, the action images corresponding to each posture image in the first posture video can be obtained, that is, the third action video is obtained, wherein the number of posture images in the first posture video is equal to the number of action images in the third action video.

[0089] For example, Figure 3 As shown, Figure 3 : is a schematic diagram of generating an action image in a third action video provided by an embodiment of the present application. Wherein, I0 is a reference action image, P0 is a reference posture image, and the first posture video includes three consecutive posture images P s-1 , P s and P s+1 After the first posture video is input into the first network model, the first network model is based on the reference action image I0, the reference posture image P0, and the posture image P s , pose image P s-1 and pose image P s-1 Corresponding action image I * s-1, determine the pose image P s Corresponding action image I * s ,like Figure 3 The first network model is also based on the reference action image I0, the reference posture image P0, and the posture image P s+1 , pose image P s and pose image P s Corresponding action image I * s , determine the pose image P s+1 Corresponding action image I * s+1 ,like Figure 3 As shown by the dotted line in .

[0090] It should be noted that Figure 3 Medium Action Image I s and I s+1 is the pose image P s and P s+1 The corresponding action image in the first action video, action image I * s-1 ,I * s and I * s+1 is the pose image P s and P s+1 The corresponding action image in the third action video.

[0091] It can be understood that when s=1, P s-1 =P0, at this time, the first network model determines the action image I corresponding to the posture image P1 according to the reference action image I0, the reference posture image P0 and the posture image P1. * 1, that is, the first network model mentioned above determines the action image corresponding to the first posture image according to the reference action image, the reference posture image and the first posture image.

[0092] In a possible implementation, an action image corresponding to any posture image is determined based on a reference action image, a reference posture image, any posture image, a previous posture image of any posture image, and an action image corresponding to the previous posture image, including: determining a first action image based on the reference action image, the reference posture image, and any posture image; determining a second action image based on the reference action image, any posture image, a previous posture image of any posture image, and an action image corresponding to the previous posture image; determining an action image corresponding to any posture image based on the first action image and the second action image.

[0093] In the embodiment of the present application, the first network model determines the first action image based on the reference action image, the reference posture image and any posture image. The first network model determines the second action image based on the reference action image, any posture image, the previous posture image of any posture image and the action image corresponding to the previous posture image.

[0094] The first motion image and the second motion image are determined based on different methods. Optionally, the first motion image is determined based on an image fusion method, which is recorded as method A1, and the second motion image is determined based on a feature fusion method, which is recorded as method A2. Methods A1 and A2 will be described in detail below.

[0095] Method A1, determining a first action image based on a reference action image, a reference posture image and any posture image, includes: determining a fused posture image based on the reference posture image and any posture image; determining the first action image based on the reference action image and the fused posture image.

[0096] In an embodiment of the present application, the first network model first extracts features of a reference posture image and features of any posture image. Optionally, the features of the reference posture image and the features of any posture image are both pyramid-level features.

[0097] Then, the first network model fuses the features of the reference pose image with the features of any pose image to fuse the appearance features of the skeleton model in the reference pose image and the action features of the skeleton model in any pose image to obtain fused features, and obtains a fused pose image based on the fused features. The fused features are the features of the fused pose image, and in this way, the fused pose image contains the appearance of the skeleton model in the reference pose image and the action of the skeleton model in any pose image.

[0098] Afterwards, the first network model performs image fusion on the reference action image and the fused posture image to fuse the appearance, action, etc. of the skeleton model in the fused posture image with the appearance of the sample object in the reference action image to obtain a first action image. The action of the sample object in the first action image is the same as the action of the skeleton model in any posture image.

[0099] Method A2, determining a second action image based on a reference action image, any posture image, a posture image previous to any posture image, and an action image corresponding to the previous posture image, includes: extracting features of the reference action image, features of any posture image, features of a posture image previous to any posture image, and features of the action image corresponding to the previous posture image; determining fusion features based on features of the reference action image, features of any posture image, features of a posture image previous to any posture image, and features of the action image corresponding to the previous posture image; and determining the second action image based on the fusion features.

[0100] In an embodiment of the present application, the first network model first extracts features of a reference action image, features of any posture image, features of a posture image previous to any posture image, and features of an action image corresponding to the previous posture image. Optionally, the features of the reference action image, features of any posture image, features of a posture image previous to any posture image, and features of an action image corresponding to the previous posture image are all pyramid-level features.

[0101] Then, the first network model fuses the features of the reference action image, the features of any posture image, the features of the previous posture image of any posture image, and the features of the action image corresponding to the previous posture image to obtain a fused feature. After that, the first network model decodes the fused feature to obtain a second action image, in which the action of the sample object in the second action image is the same as the action of the skeleton model in any posture image.

[0102] Through method A1 and method A2, a first action image and a second action image corresponding to any posture image are obtained. Afterwards, the first network model generates a weight image according to the first action image and the second action image, and the weight image includes the weight of each pixel in the first action image or the weight of each pixel in the second action image.

[0103] Exemplarily, the weight image includes the weight of each pixel, and the weight of any pixel is 0 or 1. When the weight of any pixel is 0, the weight of the pixel in the first action image is 0 (equivalent to the weight of the pixel in the second action image is 1). When the weight of any pixel is 1, the weight of the pixel in the first action image is 1 (equivalent to the weight of the pixel in the second action image is 0).

[0104] Afterwards, the first network model determines the action graph corresponding to any posture image according to the first action image, the second action image and the weight image. For any pixel point in the first action image, the first network model determines a pixel point in the action image corresponding to any posture image according to the pixel point in the first action image, the pixel point in the second action image corresponding to the pixel point, and the weight in the weight image corresponding to the pixel point.

[0105] Exemplarily, the weight image includes the weight of each pixel in the first action image. For any pixel in the first action image, when the weight in the weight image corresponding to the pixel is 1, the pixel is a pixel in the action image corresponding to any posture image; when the weight in the weight image corresponding to the pixel is 0, the pixel in the second action image corresponding to the pixel is a pixel in the action image corresponding to any posture image.

[0106] The present application embodiment provides a detailed example of determining the action image corresponding to any posture image. Figure 4 As shown, Figure 4 It is a schematic diagram of determining an action image corresponding to any posture image provided by an embodiment of the present application. The first network model includes a feature extraction network for posture images, a feature extraction network for action images, an optical flow network, a modulation network, a decoding network, a distortion network and a weight network.

[0107] The reference pose image P0, any pose image P s , the previous pose image P of any pose image s-1 , reference action image I0 and previous pose image P s-1 Corresponding action image I * s-1 , is input to the first network model, and the feature extraction network of the posture image extracts the features of the reference posture image P0 and any posture image P s The features and the previous pose image P s-1 The feature extraction network of the action image extracts the features of the reference action image I0 and the action image I * s-1 characteristics.

[0108] On the one hand, the features of the reference pose image P0 and any pose image P s The features of the reference pose image P0 and any pose image P s The features of are fused to obtain the fused features, and the fused posture image P is obtained based on the fused features. * s, realize the feature based on the reference pose image P0 and any pose image P s features, predict the reference pose image P0 and any pose image P s The optical flow between them is used to deform the reference pose image P0 to obtain the fused pose image P * s After that, the fused posture image P * s The reference action image I0 and the fusion posture image P are input into the distortion network, and the distortion network transforms the reference action image I0 and the fusion posture image P * s Perform image fusion to obtain the first action image I * s1 .

[0109] On the other hand, any pose image P s The features and the previous pose image P s-1 The features of the reference action image I0 and the action image I * s-1 The features of are input into the modulation network, and the modulation network is used to modulate any posture image P s The features and the previous pose image P s-1 The features of the reference action image I0 and the action image I * s-1 The fused features are input to the decoding network for decoding, and the decoding network outputs the second action image I * s2 .

[0110] Then, the first action image I * s1 and the second action image I * s2 Input to the weight network, the weight network according to the first action image I * s1 and the second action image I * s2 Generate a weight image. Then, the weight network is based on the first action image I * s1 , second action image I * s2 And the weight image, determine any pose image P s Corresponding action image I * s .

[0111] It can be understood that for the first posture image in the first posture video, since the first posture image does not have the previous posture image, the first network model determines the action image corresponding to the first posture image based on the reference action image, the reference posture image and the first posture image. Among them, on the one hand, the first network model determines the first action image based on the reference action image, the reference posture image and the first posture image, and on the other hand, the first network model determines the second action image based on the reference action image and the first posture image. Afterwards, the first network model determines the action image corresponding to the first posture image based on the first action image and the second action image.

[0112] When the first network model determines the first action image based on the reference action image, the reference posture image and the first posture image, the first network model first determines the fused posture image based on the reference posture image and the first posture image, and then determines the first action image based on the reference action image and the fused posture image. For details, please refer to the relevant description of method A1. The implementation principles of the two are the same and will not be repeated here.

[0113] When the first network model determines the second action image based on the reference action image and the first posture image, the first network model first extracts the features of the reference action image and the features of the first posture image, and then determines the fusion features based on the features of the reference action image and the features of the first posture image, and then determines the second action image based on the fusion features. For details, please refer to the relevant description of method A2. The implementation principles of the two are the same and will not be repeated here.

[0114] When the first network model determines the action image corresponding to the first posture image based on the first action image and the second action image, the first network model first generates a weight image based on the first action image and the second action image, and then determines the action graph corresponding to the first posture image based on the first action image, the second action image and the weight image. For details, please refer to the relevant description of determining the action image corresponding to any posture image based on the first network model according to the first action image and the second action image. The implementation principles of the two are the same and will not be repeated here.

[0115] Through the above method, the action image corresponding to each posture image in the first posture video can be determined, that is, the third action video can be obtained.

[0116] Step 203: Adjust the first network model based on the first action video and the third action video to obtain a second network model.

[0117] In an embodiment of the present application, a loss value of the first network model is calculated based on the first action video and the third action video, and model parameters of the first network model are adjusted based on the loss value of the first network model to obtain the first network model after adjusting the model parameters, that is, to obtain the second network model.

[0118] It can be understood that there are multiple sample objects, and the embodiment of the present application adjusts the first network model based on the first action video of each sample object and the third action video of each sample object to obtain the second network model.

[0119] In a possible implementation, the first action video includes action images corresponding to each posture image in the first posture video, and the third action video includes action images corresponding to each posture image in the first posture video; the first network model is adjusted based on the first action video and the third action video to obtain the second network model, including: determining the loss value of each first image set, the first image set includes a third action image, a fourth action image, a fifth action image and a sixth action image, the third action image and the fourth action image are action images corresponding to the first adjacent posture images in the first action video, the fifth action image and the sixth action image are action images corresponding to the first adjacent posture images in the third action video, and the first adjacent posture images are two adjacent posture images in the first posture video; based on the loss value of each first image set, the first network model is adjusted to obtain the second network model.

[0120] It should be noted that the posture extractor is used to obtain the posture images corresponding to each action image in the first action video to obtain the first posture video. Therefore, the first action video includes the action images corresponding to each posture image in the first posture video. Since the action images corresponding to each posture image in the first posture video are determined, the third action video is obtained. Therefore, the third action video includes the action images corresponding to each posture image in the first posture video. That is, the first action video and the third action video include the same number of action images.

[0121] In the embodiment of the present application, any two adjacent posture images in the first posture video are taken as a posture image pair, and the two action images in the first action video corresponding to the posture image pair (which can be taken as an action image pair, including the third action image and the fourth action image) and the two action images in the third action video corresponding to the posture image pair (which can be taken as another action image pair, including the fifth action image and the sixth action image) are taken as a first image set. In other words, one posture image pair corresponds to one first image set, and one first image set includes four action images, which are the third action image, the fourth action image, the fifth action image, and the sixth action image.

[0122] For example, the first posture video includes three consecutive posture images P s-1 , P s and P s+1 The three action images in the first action video corresponding to these three posture images are I s-1 ,I s and I s+1 , the three action images in the third action video corresponding to these three posture images are and Then the pose image pair {P s-1 , P s The first image set corresponding to} is Among them, I s-1 and I s are the third action image and the fourth action image respectively, and are the fifth action image and the sixth action image respectively. s , P s+1 The first image set corresponding to} is Among them, I s and I s+1 are the third action image and the fourth action image respectively, and They are the fifth motion image and the sixth motion image respectively.

[0123] For any first image set, the loss value of the first image set is determined using the third motion image, the fourth motion image, the fifth motion image and the sixth motion image of the first image set.

[0124] Optionally, the third action image and the fifth action image are action images corresponding to the same posture image in the first posture video, and the fourth action image and the sixth action image are action images corresponding to the same posture image in the first posture video; determining the loss value of each first image set includes: for any first image set, determining a first loss value based on the third action image and the fifth action image in any first image set; determining a second loss value based on the fourth action image and the sixth action image in any first image set; determining a third loss value based on the third action image, the fourth action image, the fifth action image and the sixth action image in any first image set; determining the loss value of any first image set based on the first loss value, the second loss value and the third loss value.

[0125] For any first image set, the third action image and the fifth action image in the first image set are action images corresponding to the same posture image, and the fourth action image and the sixth action image are action images corresponding to the same posture image.

[0126] For example, the first posture video includes a posture image P s and P s+1 , the two action images in the first action video corresponding to these two posture images are I s and I s+1 , the two action images in the third action video corresponding to these two posture images are and Then the pose image pair {P s , P s+1 The first image set corresponding to} is Among them, I s is the third action image, I s+1 is the fourth action image, is the fifth action image, This is the sixth action image.

[0127] In an embodiment of the present application, for any first image set, a first loss value is determined based on the third action image, the fifth action image and the model parameters of the first network model in the first image set. A second loss value is determined based on the fourth action image, the sixth action image and the model parameters of the first network model in the first image set. A third loss value is determined based on the third action image, the fourth action image, the fifth action image, the sixth action image and the model parameters of the first network model in the first image set. Thereafter, the loss value of the first image set is determined based on the first loss value, the second loss value and the third loss value. The loss value of the first image set is shown in formula (1).

[0128]

[0129] in, is the loss value of the first image set, represents the first image set, θ is the model parameter of the first network model, is the first loss value, is the second loss value, is the third loss value, I s is the third action image, I s+1 is the fourth action image, is the fifth action image, This is the sixth action image.

[0130] Through the above method, the loss value of each first image set can be determined. Afterwards, based on the loss value of each first image set, the loss value of the first network model is determined. Using the loss value of the first network model, the model parameters of the first network model are adjusted to obtain the adjusted model parameters of the first network model, that is, the model parameters of the second network model are obtained, which is equivalent to obtaining the second network model. The model parameters of the second network model are shown in formula (2).

[0131]

[0132] Among them, θ′ is the model parameter of the second network model, θ is the model parameter of the first network model, is the loss value of the first network model, α is the learning rate, is the gradient symbol, is the sum of the loss values ​​of each first image set.

[0133] In an embodiment of the present application, by treating every two adjacent posture images in the first posture video as a posture image pair, the loss value of the first image set (including two action image pairs) corresponding to each posture image pair is calculated, so as to utilize the loss values ​​of several action image pairs to adjust the model parameters, enhance the model's ability to learn the timing characteristics of actions, and improve the quality of the video output by the model.

[0134] Step 204 , determining a fourth action video based on the second network model according to the reference action image, the reference posture image and the second posture video.

[0135] In the embodiment of the present application, the data of any sample object includes but is not limited to a reference action image, a reference posture image, and a second posture video. For the data of any sample object, the reference action image, the reference posture image, and the second posture video are input into the second network model, and the second network model outputs a fourth action video, which includes at least two action images.

[0136] In one possible implementation, the second posture video includes multiple posture images, and the fourth action video includes action images corresponding to each posture image in the second posture video; determining the fourth action video based on the second network model according to the reference action image, the reference posture image and the second posture video, including: determining the action image corresponding to the first posture image based on the second network model according to the reference action image, the reference posture image and the first posture image in the second posture video; for any posture image except the first posture image in the second posture video, determining the action image corresponding to any posture image according to the reference action image, the reference posture image, any posture image, the previous posture image of any posture image and the action image corresponding to the previous posture image.

[0137] In the embodiment of the present application, after the reference action image, the reference posture image and the second posture video are input into the second network model, for the first posture image in the second posture video, the second network model determines the action image corresponding to the first posture image based on the reference action image, the reference posture image and the first posture image. For the second posture image in the second posture video, the second network model determines the action image corresponding to the second posture image based on the reference action image, the reference posture image, the second posture image, the first posture image (i.e., the previous posture image of the second posture image) and the action image corresponding to the first posture image. For the third posture image in the second posture video, the second network model determines the action image corresponding to the third posture image based on the reference action image, the reference posture image, the third posture image, the second posture image (i.e., the previous posture image of the third posture image) and the action image corresponding to the second posture image. In this way, by analogy, the action images corresponding to each posture image in the second posture video can be obtained, that is, the fourth action video is obtained, wherein the number of posture images in the second posture video is equal to the number of action images in the fourth action video.

[0138] For example, Figure 5 As shown, Figure 5 : is a schematic diagram of generating an action image in a fourth action video provided by an embodiment of the present application. Wherein, I0 is a reference action image, P0 is a reference posture image, and the second posture video includes three consecutive posture images P q-1 , P q and P q+1 After the second posture video is input into the second network model, the second network model is based on the reference action image I0, the reference posture image P0, and the posture image P q , pose image P q-1 and pose image P q-1 Corresponding action image I * q-1 , determine the pose image P q Corresponding action image I * q ,like Figure 5 The second network model is also based on the reference action image I0, the reference posture image P0, and the posture image P q+1 , pose image P q and pose image P q Corresponding action image I * q , determine the pose image P q+1 Corresponding action image I * q+1 ,like Figure 5 As shown by the dotted line in .

[0139] It should be noted that Figure 3 Medium Action Image I q and I q+1 is the pose image P q and P q+1 The corresponding action image in the second action video, action image I * q-1 ,I * q and I * q+1 is the pose image P q and P q+1 The corresponding action image in the fourth action video.

[0140] It can be understood that when q = 1, P q-1 =P0, at this time, the second network model determines the action image I corresponding to the posture image P1 according to the reference action image I0, the reference posture image P0 and the posture image P1. * 1, that is, the second network model mentioned above determines the action image corresponding to the first posture image according to the reference action image, the reference posture image and the first posture image.

[0141] It can be understood that, since the model parameters of the first network model are further adjusted to obtain the second network model, the first network model and the second network model only have different model parameters. The second network model determines the fourth action video based on the reference action image, the reference posture image and the second posture video, in the same way as the first network model determines the third action video based on the reference action image, the reference posture image and the first posture video. For details, please refer to the relevant description of step 202. The implementation principles of the two are the same and will not be repeated here.

[0142] Step 205: Adjust the second network model based on the second action video and the fourth action video to obtain a third network model.

[0143] In an embodiment of the present application, a loss value of the second network model is calculated based on the second action video and the fourth action video, and the model parameters of the second network model are adjusted based on the loss value of the second network model to obtain the second network model after adjusting the model parameters, that is, to obtain the third network model.

[0144] In one possible implementation, the second action video includes action images corresponding to each posture image in the second posture video, and the fourth action video includes action images corresponding to each posture image in the second posture video; the second network model is adjusted based on the second action video and the fourth action video to obtain a third network model, including: determining the loss value of each second image set, the second image set includes a seventh action image, an eighth action image, a ninth action image and a tenth action image, the seventh action image and the eighth action image are action images corresponding to the second adjacent posture images in the second action video, the ninth action image and the tenth action image are action images corresponding to the second adjacent posture images in the fourth action video, and the second adjacent posture images are two adjacent posture images in the second posture video; based on the loss value of each second image set and the parameters of the first network model, the second network model is adjusted to obtain the third network model.

[0145] It should be noted that the posture extractor is used to obtain the posture images corresponding to each action image in the second action video to obtain the second posture video. Therefore, the second action video includes the action images corresponding to each posture image in the second posture video. Since the action images corresponding to each posture image in the second posture video are determined, the fourth action video is obtained. Therefore, the fourth action video includes the action images corresponding to each posture image in the second posture video. That is, the second action video and the fourth action video include the same number of action images.

[0146] In the embodiment of the present application, any two adjacent posture images in the second posture video are taken as a posture image pair, and the two action images in the second action video corresponding to the posture image pair (i.e., the seventh action image and the eighth action image) and the two action images in the fourth action video corresponding to the posture image pair (i.e., the ninth action image and the tenth action image) are taken as a second image set. In other words, one posture image pair corresponds to one second image set, and one second image set includes four action images, which are the seventh action image, the eighth action image, the ninth action image, and the tenth action image.

[0147] For example, the second posture video includes three consecutive posture images P q-1 , P q and P q+1 The three action images in the second action video corresponding to these three posture images are I q-1 ,I q and I q+1 , the three action images in the fourth action video corresponding to these three posture images are and Then the pose image pair {P q-1 , P qThe second image set corresponding to} is Among them, I q-1 and I q are the seventh action image and the eighth action image respectively, and are the ninth action image and the tenth action image respectively. q , P q+1 The second image set corresponding to} is Among them, I q and I q+1 are the seventh action image and the eighth action image respectively, and They are the ninth action image and the tenth action image respectively.

[0148] For any second image set, the loss value of the second image set is determined using the seventh motion image, the eighth motion image, the ninth motion image and the tenth motion image of the second image set.

[0149] Optionally, the seventh action image and the ninth action image are action images corresponding to the same posture image, and the eighth action image and the tenth action image are action images corresponding to the same posture image; determining the loss value of each second image set includes: for any second image set, determining a fourth loss value based on the seventh action image and the ninth action image in any second image set; determining a fifth loss value based on the eighth action image and the tenth action image in any second image set; determining a sixth loss value based on the seventh action image, the eighth action image, the ninth action image and the tenth action image in any second image set; determining the loss value of any second image set based on the fourth loss value, the fifth loss value and the sixth loss value.

[0150] For any second image set, the seventh action image and the ninth action image in the second image set are action images corresponding to the same posture image, and the eighth action image and the tenth action image are action images corresponding to the same posture image.

[0151] For example, the second posture video includes a posture image P q and P q+1 , the two action images in the second action video corresponding to these two posture images are I q and I q+1 , the two action images in the fourth action video corresponding to these two posture images are and Then the pose image pair {P q , P q+1 The second image set corresponding to} is Among them, I qFor the seventh action image, I q+1 For the eighth action image, For the ninth action image, This is the tenth action image.

[0152] In an embodiment of the present application, for any second image set, a fourth loss value is determined based on the seventh action image, the ninth action image and the model parameters of the second network model in the second image set. A fifth loss value is determined based on the eighth action image, the tenth action image and the model parameters of the second network model in the second image set. A sixth loss value is determined based on the seventh action image, the eighth action image, the ninth action image, the tenth action image and the model parameters of the second network model in the second image set. Thereafter, the loss value of the second image set is determined based on the fourth loss value, the fifth loss value and the sixth loss value. The loss value of the second image set is shown in formula (3).

[0153]

[0154] in, is the loss value of the second image set, represents the second image set, θ′ is the model parameter of the second network model, is the fourth loss value, is the fifth loss value, is the sixth loss value, I q For the seventh action image, I q+1 For the eighth action image, For the ninth action image, This is the tenth action image.

[0155] Through the above method, the loss value of each second image set can be determined. Afterwards, based on the loss value of each second image set, the loss value of the second network model is determined. Using the loss value of the second network model, the model parameters of the second network model are adjusted to obtain the adjusted model parameters of the second network model, that is, the model parameters of the third network model are obtained, which is equivalent to obtaining the third network model. The model parameters of the third network model are shown in formula (4).

[0156]

[0157] Among them, the θ before “←” is the model parameter of the third network model, the θ after “←” is the model parameter of the first network model, and θ′ is the model parameter of the second network model. is the loss value of the second network model, β is the learning rate, is the gradient symbol, is the sum of the loss values ​​of each second image set.

[0158] It should be noted that the loss functions shown in the above formulas (1) to (4) are only exemplary. When applied, they can be flexibly changed according to the application scenario, manual experience, etc. For example, in order to improve the temporal stability of the video output by the model, the model parameters of the model can be adjusted based on the temporal loss function. The temporal loss function is not limited here.

[0159] In an embodiment of the present application, a fourth action video is determined based on the second network model according to the reference action image, the reference posture image and the second posture video, and the quality of the video output by the model and the timing performance of the model are evaluated through the fourth action video, and the parameters of the model are further updated to enhance the learning ability of the model and improve the quality of the video output by the model and the timing performance of the model.

[0160] Step 206: In response to satisfying the training end condition, using the third network model as a video redirection model.

[0161] When the training end condition is met, the third network model is a video redirection model. There is no limitation on the training end condition. For example, the training end condition is to reach the target number of training times. There is no limitation on the value of the target number of training times. It is flexibly set according to manual experience or scenarios. For example, the target number of training times is 500 times.

[0162] In an embodiment of the present application, a video redirection model is obtained by utilizing data of multiple sample objects after multiple trainings. The video redirection model is a universal model with strong generalization ability. It can be effectively adjusted by the action video of the target object, so that the model can quickly adapt to the target object and become a personalized model of the target object, which is convenient for outputting the target video of the target object.

[0163] In one possible implementation, after adjusting the second network model based on the second action video and the fourth action video to obtain the third network model, it also includes: in response to not meeting the training end condition, determining the fourth network model based on the third network model according to the data of the sample object; in response to meeting the training end condition, using the fourth network model as a video redirection model.

[0164] When the training end condition is not met, a fourth network model is determined based on the third network model according to the data of the plurality of sample objects, wherein the data of any sample object includes a reference action image of the sample object, a reference posture image of the sample object, a first action video of the sample object, a first posture video of the sample object, a second action video of the sample object, and a second posture video of the sample object.

[0165] When determining the fourth network model based on the third network model according to the data of the sample object, first determine the fifth action video based on the third network model according to the reference action image, the reference posture image and the first posture video. Then adjust the third network model based on the first action video and the fifth action video. Then, determine the sixth action video based on the adjusted third network model according to the reference action image, the reference posture image and the second posture video. Afterwards, adjust the adjusted third network model based on the second action video and the sixth action video to obtain the fourth network model. For details, please refer to the relevant descriptions of steps 202 to 205. The implementation principles of the two are the same and will not be repeated here.

[0166] The above method uses the data of the sample object to obtain the video redirection model. The video redirection model has strong generalization ability and can be used to generate target videos of different target objects, saving costs and improving practicality. The data of the sample object includes two action videos. In the process of obtaining the video redirection model, the network model is adjusted twice for these two action videos, which strengthens the learning ability of the model and improves the quality of the video output by the model.

[0167] The above describes in detail the training method of the video redirection model of the embodiment of the present application from the perspective of method steps. The following will further illustrate the training method of the video redirection model in combination with a detailed scenario. The scenario is a dancing video retargeting scenario. Dancing video retargeting is to transfer the dance movements of the original object in the original video to the target object to obtain a target video, in which the target object in the target video imitates the dance movements of the original object. In this scenario, the first action video is a dancing action video {I s}, the second action video is a dancing action video {I q}, the third action video is a dancing action video {I * s}, the fourth action video is a dancing action video {I * q}, the first posture video is a dancing posture video {P s}, the second posture video is a dancing posture video {P q}, the reference action image is I0, and the reference posture image is P0, wherein the dance action video includes at least two action images, and the dance posture video includes at least two posture images. Figure 6 As shown, Figure 6 It is a schematic diagram of a training method for a video redirection model provided in an embodiment of the present application.

[0168] In the embodiment of the present application, the dancing posture video {P s}, the reference action image I0 and the reference posture image P0 are input to the first network model, and the first network model outputs the dance action video {I * s}, where the model parameter of the first network model is θ, and the first network model outputs a dance action video {I * s}, see Figure 3 and Figure 4 The relevant description of is not repeated here. Then, using the dance action video {I * s} and dance moves videos {I s}, according to the above-mentioned formula (1) and formula (2), the loss value of the first network model is calculated, the model parameter θ of the first network model is adjusted, and the model parameter θ′ of the second network model is obtained, thereby obtaining the second network model.

[0169] After that, the dancing posture video {P q}, the reference action image I0 and the reference posture image P0 are input to the second network model, and the second network model outputs the dance action video {I * q}, where the model parameter of the second network model is θ′, and the second network model outputs a dance action video {I * q}, see Figure 4 and Figure 5 The relevant description will not be repeated here.

[0170] Then, using the dance action video {I * q} and dance moves videos {I q}, according to the above-mentioned formula (3) and formula (4), the loss value of the second network model is calculated, the model parameter θ′ of the second network model is adjusted, and the model parameter θ of the third network model is obtained, thereby obtaining the third network model.

[0171] When the training end condition is not met, the third network model can be used as the first network model and re-executed. Figure 6 The training method shown is performed until the training end condition is met to obtain a video redirection model. When the training end condition is met, the third network model is used as the video redirection model. In this way, the video redirection model in the embodiment of the present application is obtained.

[0172] Based on the above implementation environment, the embodiment of the present application provides a video redirection method. Figure 7 As an example, the flowchart of a video redirection method provided in the embodiment of the present application is shown in FIG. Figure 1 The electronic device 11 in the embodiment is executed. Figure 7As shown, the method includes steps 701 to 704.

[0173] Step 701 , obtaining a reference action image of a target object, a reference posture image of a target object, a first action video of the target object, a posture video of the target object, and a posture video of an original object.

[0174] In the embodiments of the present application, the target object and the original object are not limited. For example, the target object and the original object are both humans, or the target object and the original object are two animals of different species, etc.

[0175] The reference action image of the target object is an action image including the target object. The first action video of the target object includes at least two action images, each of which includes the actual action of the target object. The reference posture image of the target object is a posture image including the target object. The posture video of the target object includes at least two posture images, each of which includes the action of the skeleton model corresponding to the target object. The posture video of the original object includes at least two posture images, each of which includes the action of the skeleton model corresponding to the original object.

[0176] Optionally, obtaining a reference action image of the target object, a reference posture image of the target object, a first action video of the target object, a posture video of the target object, and a posture video of the original object includes: obtaining the first action video of the target object and the action video of the original object; obtaining the posture video of the target object based on the first action video of the target object, the first action video of the target object including the reference action image of the target object, and the posture video of the target object including the reference posture image of the target object; obtaining the posture video of the original object based on the action video of the original object.

[0177] A first action video of the target object and an action video of the original object are obtained, wherein the action video of the original object includes at least two action images, and each of the at least two action images includes an actual action of the original object.

[0178] In the embodiment of the present application, a reference action image of the target object is first selected from the first action video of the target object, and the reference action image of the target object is any action image in the first action video of the target object. Afterwards, based on the first action video of the target object, a posture extractor is used to obtain posture images corresponding to each action image in the first action video to obtain a posture video of the target object, wherein the posture image in the posture video of the target object corresponding to the reference action image of the target object is the reference posture image of the target object. Based on the action video of the original object, a posture extractor is used to obtain posture images corresponding to each action image in the action video to obtain a posture video of the original object. The posture extractor is not limited in the embodiment of the present application.

[0179] Step 702 : determining a second action video of the target object based on a video redirection model according to a reference action image of the target object, a reference posture image of the target object, and a posture video of the target object.

[0180] Among them, the video redirection model is obtained according to the training method of the video redirection model mentioned above. The training method of the video redirection model is detailed in the relevant description above and will not be repeated here.

[0181] In an embodiment of the present application, a reference action image of the target object, a reference posture image of the target object, and a posture video of the target object are input into a video redirection model, and the video redirection model outputs a second action video of the target object. The second action video of the target object includes at least two action images. Please refer to the relevant description of step 202 for details. Since the first network model and the video redirection model in step 202 have the same structure and size, the two only differ in model parameters. Therefore, the implementation principle is the same and will not be repeated here.

[0182] Step 703: Adjust the video redirection model based on the first action video of the target object and the second action video of the target object to obtain a target model.

[0183] In the embodiment of the present application, the loss value of the video redirection model is calculated based on the first action video of the target object and the second action video of the target object, and the model parameters of the video redirection model are adjusted based on the loss value of the video redirection model to obtain the video redirection model after adjusting the model parameters, that is, to obtain the target model. Please refer to the relevant description of step 203 for details. Since the first network model and the video redirection model in step 203 have the same structure and size, the two only differ in model parameters, and the second network model and the target model in step 203 have the same structure and size, the two only differ in model parameters, therefore, the implementation principle is the same and will not be repeated here.

[0184] Step 704 , determining a target video based on a target model based on the reference action image of the target object, the reference posture image of the target object and the posture video of the original object.

[0185] In an embodiment of the present application, a reference action image of the target object, a reference posture image of the target object, and a posture video of the original object are input into the target model, and the target model outputs a target video. The target video includes at least two action images, and the target video is a video of the target object imitating the actual action of the original object. Please refer to the relevant description of step 204 for details. Since the second network model in step 204 and the target model have the same structure and size, the two only differ in model parameters. Therefore, the implementation principle is the same and will not be repeated here.

[0186] The above method can adjust the video redirection model based on the reference action image of the target object, the reference posture image of the target object and the posture video of the target object, which helps the model capture the static appearance features of the target object and the action timing features of the target object between adjacent frames, and improves the quality of the video output by the model. In addition, since the video redirection model is trained based on the training method of the video redirection model provided in the above embodiment, the video redirection model has a strong generalization ability. Therefore, even if the posture video of the target object has only a few frames, the adjusted video redirection model can have a strong video redirection ability, allowing the model to quickly adapt to the target object and become a personalized model of the target object, thereby ensuring the quality of the video output by the model.

[0187] Figure 8 FIG. 1 is a schematic diagram of a structure of a training device for a video redirection model provided in an embodiment of the present application. Figure 8 As shown, the device includes an acquisition module 801, a determination module 802 and an adjustment module 803.

[0188] An acquisition module 801 is used to acquire data of a sample object, where the data of the sample object includes a reference action image, a reference posture image, a first action video, a first posture video, a second action video, and a second posture video;

[0189] A determination module 802 is used to determine a third action video based on a first network model according to the reference action image, the reference posture image and the first posture video;

[0190] An adjustment module 803, configured to adjust the first network model based on the first action video and the third action video to obtain a second network model;

[0191] The determination module 802 is further configured to determine a fourth action video based on the second network model according to the reference action image, the reference posture image and the second posture video;

[0192] The adjustment module 803 is further used to adjust the second network model based on the second action video and the fourth action video to obtain a third network model;

[0193] The determination module 802 is further configured to use the third network model as a video redirection model in response to satisfying a training end condition.

[0194] In a possible implementation, the first posture video includes a plurality of posture images;

[0195] Determination module 802 is used to determine the action image corresponding to the first posture image based on the first network model according to the reference action image, the reference posture image and the first posture image in the first posture video; for any posture image except the first posture image in the first posture video, determine the action image corresponding to any posture image according to the reference action image, the reference posture image, any posture image, the previous posture image of any posture image and the action image corresponding to the previous posture image.

[0196] In one possible implementation, the determination module 802 is used to determine a first action image based on a reference action image, a reference posture image and any posture image; determine a second action image based on the reference action image, any posture image, a previous posture image of any posture image and an action image corresponding to the previous posture image; and determine an action image corresponding to any posture image based on the first action image and the second action image.

[0197] In a possible implementation, the determination module 802 is configured to determine a fused posture image according to a reference posture image and any one posture image; and determine a first action image according to the reference action image and the fused posture image.

[0198] In one possible implementation, the determination module 802 is used to extract features of a reference action image, features of any posture image, features of a posture image previous to any posture image, and features of an action image corresponding to the previous posture image; determine fusion features based on the features of the reference action image, features of any posture image, features of a posture image previous to any posture image, and features of an action image corresponding to the previous posture image; and determine a second action image based on the fusion features.

[0199] In a possible implementation, the first action video includes action images corresponding to each posture image in the first posture video, and the third action video includes action images corresponding to each posture image in the first posture video;

[0200] The adjustment module 803 is used to determine the loss value of each first image set, where the first image set includes a third action image, a fourth action image, a fifth action image and a sixth action image, where the third action image and the fourth action image are action images corresponding to the first adjacent posture images in the first action video, the fifth action image and the sixth action image are action images corresponding to the first adjacent posture images in the third action video, and the first adjacent posture images are two adjacent posture images in the first posture video; based on the loss value of each first image set, the first network model is adjusted to obtain a second network model.

[0201] In a possible implementation, the third action image and the fifth action image are action images corresponding to the same action image in the first action video, and the fourth action image and the sixth action image are action images corresponding to the same action image in the first action video;

[0202] The adjustment module 803 is used to determine, for any first image set, a first loss value based on the third action image and the fifth action image in any first image set; determine a second loss value based on the fourth action image and the sixth action image in any first image set; determine a third loss value based on the third action image, the fourth action image, the fifth action image and the sixth action image in any first image set; and determine the loss value of any first image set based on the first loss value, the second loss value and the third loss value.

[0203] In a possible implementation, the second posture video includes a plurality of posture images;

[0204] Determination module 802 is used to determine the action image corresponding to the first posture image based on the second network model according to the reference action image, the reference posture image and the first posture image in the second posture video; for any posture image in the second posture video except the first posture image, determine the action image corresponding to any posture image according to the reference action image, the reference posture image, any posture image, the previous posture image of any posture image and the action image corresponding to the previous posture image.

[0205] In a possible implementation, the second action video includes action images corresponding to each posture image in the second posture video, and the fourth action video includes action images corresponding to each posture image in the second posture video;

[0206] The adjustment module 803 is used to determine the loss value of each second image set, the second image set includes the seventh action image, the eighth action image, the ninth action image and the tenth action image, the seventh action image and the eighth action image are action images corresponding to the second adjacent posture images in the second action video, the ninth action image and the tenth action image are action images corresponding to the second adjacent posture images in the fourth action video, and the second adjacent posture images are two adjacent posture images in the second posture video; based on the loss value of each second image set and the parameters of the first network model, the second network model is adjusted to obtain the third network model.

[0207] In one possible implementation, the acquisition module 801 is used to acquire an action video of a sample object; select a reference action image, a first action video, and a second action video from the action video of the sample object; acquire a reference posture image based on the reference action image; acquire a first posture video based on the first action video; and acquire a second posture video based on the second action video.

[0208] In one possible implementation, the determination module 802 is further used to determine a fourth network model based on the third network model according to data of the sample object in response to not meeting the training end condition; and to use the fourth network model as a video redirection model in response to meeting the training end condition.

[0209] The above device uses the data of the sample object to obtain a video redirection model. The video redirection model has strong generalization ability and can be used to generate target videos of different target objects, saving costs and improving practicality. The data of the sample object includes two action videos. In the process of obtaining the video redirection model, the network model is adjusted twice for these two action videos, which strengthens the learning ability of the model and improves the quality of the video output by the model.

[0210] It should be understood that the above Figure 8 When the device provided realizes its functions, only the division of the above-mentioned functional modules is used as an example for illustration. In practical applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0211] Fig. 9 FIG. 1 is a schematic diagram of the structure of a video redirection device provided in an embodiment of the present application. Fig. 9 As shown, the device includes an acquisition module 901 , a determination module 902 and an adjustment module 903 .

[0212] An acquisition module 901 is used to acquire a reference action image of a target object, a reference posture image of a target object, a first action video of a target object, a posture video of a target object, and a posture video of an original object;

[0213] A determination module 902 is used to determine a second action video of the target object based on a video redirection model according to a reference action image of the target object, a reference posture image of the target object and a posture video of the target object, wherein the video redirection model is obtained according to any of the above-mentioned training methods of the video redirection model;

[0214] An adjustment module 903 is used to adjust the video redirection model based on the first action video of the target object and the second action video of the target object to obtain a target model;

[0215] The determination module 902 is further configured to determine the target video based on the target model based on the reference action image of the target object, the reference posture image of the target object and the posture video of the original object.

[0216] In a possible implementation, the acquisition module 901 is used to acquire a first action video of the target object and an action video of the original object;

[0217] acquiring a posture video of the target object based on a first action video of the target object, wherein the first action video of the target object includes a reference action image of the target object, and the posture video of the target object includes a reference posture image of the target object;

[0218] A posture video of the original object is obtained based on the action video of the original object.

[0219] The above device can adjust the video redirection model based on the reference action image of the target object, the reference posture image of the target object and the posture video of the target object, which helps the model capture the static appearance features of the target object and the action timing features of the target object between adjacent frames, and improves the quality of the video output by the model. In addition, since the video redirection model is trained based on the training method of the video redirection model provided by the above embodiment, the video redirection model has a strong generalization ability. Therefore, even if the posture video of the target object has only a few frames, the adjusted video redirection model can have a strong video redirection ability, allowing the model to quickly adapt to the target object and become a personalized model of the target object, thereby ensuring the quality of the video output by the model.

[0220] It should be understood that the above Fig. 9When the device provided realizes its functions, only the division of the above-mentioned functional modules is used as an example for illustration. In practical applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0221] Fig.10 The structure block diagram of a terminal device 1000 provided by an exemplary embodiment of the present application is shown. The terminal device 1000 may be a portable mobile terminal, such as a smart phone, a tablet computer, an MP3 (Moving Picture Experts Group Audio Layer III, dynamic image expert compression standard audio layer 3) player, an MP4 (Moving Picture Experts Group Audio Layer IV, dynamic image expert compression standard audio layer 4) player, a notebook computer or a desktop computer. The terminal device 1000 may also be called a user device, a portable terminal, a laptop terminal, a desktop terminal or other names.

[0222] Typically, the terminal device 1000 includes: a processor 1001 and a memory 1002 .

[0223] The processor 1001 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 1001 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 1001 may also include a main processor and a coprocessor. The main processor is a processor for processing data in an awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 1001 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1001 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0224] The memory 1002 may include one or more computer-readable storage media, which may be non-transitory. The memory 1002 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1002 is used to store at least one instruction, which is used to be executed by the processor 1001 to implement the training method of the video redirection model or the video redirection method provided in the method embodiment of the present application.

[0225] In some embodiments, the terminal device 1000 may also optionally include: a peripheral device interface 1003 and at least one peripheral device. The processor 1001, the memory 1002 and the peripheral device interface 1003 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 1003 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 1004, a display screen 1005, a camera assembly 1006, an audio circuit 1007, a positioning assembly 1008 and a power supply 1009.

[0226] The peripheral device interface 1003 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 1001 and the memory 1002. In some embodiments, the processor 1001, the memory 1002, and the peripheral device interface 1003 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 1001, the memory 1002, and the peripheral device interface 1003 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0227] The radio frequency circuit 1004 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 1004 communicates with the communication network and other communication devices through electromagnetic signals. The radio frequency circuit 1004 converts the electrical signal into an electromagnetic signal for transmission, or converts the received electromagnetic signal into an electrical signal. Optionally, the radio frequency circuit 1004 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The radio frequency circuit 1004 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes but is not limited to: the World Wide Web, a metropolitan area network, an intranet, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 1004 may also include circuits related to NFC (Near Field Communication), which is not limited in this application.

[0228] The display screen 1005 is used to display a UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 1005 is a touch display screen, the display screen 1005 also has the ability to collect touch signals on the surface or above the surface of the display screen 1005. The touch signal can be input to the processor 1001 as a control signal for processing. At this time, the display screen 1005 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 1005 can be one, arranged on the front panel of the terminal device 1000; in other embodiments, the display screen 1005 can be at least two, respectively arranged on different surfaces of the terminal device 1000 or in a folding design; in other embodiments, the display screen 1005 can be a flexible display screen, arranged on a curved surface or a folding surface of the terminal device 1000. Even, the display screen 1005 can also be arranged as a non-rectangular irregular figure, that is, a special-shaped screen. The display screen 1005 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0229] The camera assembly 1006 is used to capture images or videos. Optionally, the camera assembly 1006 includes a front camera and a rear camera. Typically, the front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize the panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 1006 may also include a flash. The flash can be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0230] The audio circuit 1007 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them into the processor 1001 for processing, or input them into the radio frequency circuit 1004 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there may be multiple microphones, which are respectively arranged at different parts of the terminal device 1000. The microphone may also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 1001 or the radio frequency circuit 1004 into sound waves. The speaker may be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 1007 may also include a headphone jack.

[0231] The positioning component 1008 is used to locate the current geographical location of the terminal device 1000 to implement navigation or LBS (Location Based Service). The positioning component 1008 can be a positioning component based on the US GPS (Global Positioning System), China's Beidou system or Russia's Galileo system.

[0232] The power supply 1009 is used to power various components in the terminal device 1000. The power supply 1009 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 1009 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery charged through a wired line, and a wireless rechargeable battery is a battery charged through a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0233] In some embodiments, the terminal device 1000 further includes one or more sensors 1010 , including but not limited to: an acceleration sensor 1011 , a gyroscope sensor 1012 , a pressure sensor 1013 , a fingerprint sensor 1014 , an optical sensor 1015 , and a proximity sensor 1016 .

[0234] The acceleration sensor 1011 can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established by the terminal device 1000. For example, the acceleration sensor 1011 can be used to detect the components of gravity acceleration on the three coordinate axes. The processor 1001 can control the display screen 1005 to display the user interface in a horizontal view or a vertical view according to the gravity acceleration signal collected by the acceleration sensor 1011. The acceleration sensor 1011 can also be used to collect game or user motion data.

[0235] The gyro sensor 1012 can detect the body direction and rotation angle of the terminal device 1000, and the gyro sensor 1012 can cooperate with the acceleration sensor 1011 to collect the user's 3D actions on the terminal device 1000. The processor 1001 can implement the following functions based on the data collected by the gyro sensor 1012: motion sensing (such as changing the UI according to the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

[0236] The pressure sensor 1013 can be set on the side frame of the terminal device 1000 and / or the lower layer of the display screen 1005. When the pressure sensor 1013 is set on the side frame of the terminal device 1000, the user's holding signal of the terminal device 1000 can be detected, and the processor 1001 performs left and right hand recognition or shortcut operations according to the holding signal collected by the pressure sensor 1013. When the pressure sensor 1013 is set on the lower layer of the display screen 1005, the processor 1001 controls the operability controls on the UI interface according to the user's pressure operation on the display screen 1005. The operability controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0237] The fingerprint sensor 1014 is used to collect the user's fingerprint, and the processor 1001 identifies the user's identity based on the fingerprint collected by the fingerprint sensor 1014, or the fingerprint sensor 1014 identifies the user's identity based on the collected fingerprint. When the user's identity is identified as a trusted identity, the processor 1001 authorizes the user to perform relevant sensitive operations, including unlocking the screen, viewing encrypted information, downloading software, paying, and changing settings. The fingerprint sensor 1014 can be set on the front, back, or side of the terminal device 1000. When a physical button or a manufacturer logo is set on the terminal device 1000, the fingerprint sensor 1014 can be integrated with the physical button or the manufacturer logo.

[0238] The optical sensor 1015 is used to collect the ambient light intensity. In one embodiment, the processor 1001 can control the display brightness of the display screen 1005 according to the ambient light intensity collected by the optical sensor 1015. Specifically, when the ambient light intensity is high, the display brightness of the display screen 1005 is increased; when the ambient light intensity is low, the display brightness of the display screen 1005 is reduced. In another embodiment, the processor 1001 can also dynamically adjust the shooting parameters of the camera assembly 1006 according to the ambient light intensity collected by the optical sensor 1015.

[0239] The proximity sensor 1016, also called a distance sensor, is usually arranged on the front panel of the terminal device 1000. The proximity sensor 1016 is used to collect the distance between the user and the front of the terminal device 1000. In one embodiment, when the proximity sensor 1016 detects that the distance between the user and the front of the terminal device 1000 is gradually decreasing, the processor 1001 controls the display screen 1005 to switch from the screen-on state to the screen-off state; when the proximity sensor 1016 detects that the distance between the user and the front of the terminal device 1000 is gradually increasing, the processor 1001 controls the display screen 1005 to switch from the screen-off state to the screen-on state.

[0240] Those skilled in the art will understand that Fig.10 The structure shown in the figure does not constitute a limitation on the terminal device 1000, and may include more or fewer components than shown in the figure, or combine certain components, or adopt a different component arrangement.

[0241] Fig.11This is a schematic diagram of the structure of the server provided in the embodiment of the present application. The server 1100 may have relatively large differences due to different configurations or performances, and may include one or more processors 1101 and one or more memories 1102, wherein the one or more memories 1102 store at least one program code, and the at least one program code is loaded and executed by the one or more processors 1101 to implement the training method of the video redirection model or the video redirection method provided in the above-mentioned various method embodiments. Exemplarily, the processor 1101 is a CPU. Of course, the server 1100 may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output. The server 1100 may also include other components for implementing device functions, which will not be described in detail here.

[0242] In an exemplary embodiment, a computer-readable storage medium is also provided, in which at least one program code is stored. The at least one program code is loaded and executed by a processor to enable an electronic device to implement any of the above-mentioned video redirection model training methods or video redirection methods.

[0243] Optionally, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0244] In an exemplary embodiment, a computer program or a computer program product is also provided, in which at least one computer instruction is stored, and the at least one computer instruction is loaded and executed by a processor to enable a computer to implement any of the above-mentioned video redirection model training methods or video redirection methods.

[0245] It should be understood that the "plurality" mentioned in this article refers to two or more. "And / or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0246] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0247] The above description is only an exemplary embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A training method for a video redirection model, It is characterized in that The method comprises: Acquire data of a sample object, wherein the data of the sample object includes a reference action image, a reference posture image, a first action video, a first posture video, a second action video, and a second posture video; Determining a third action video based on a first network model according to the reference action image, the reference posture image and the first posture video; Adjusting the first network model based on the first action video and the third action video to obtain a second network model; Determining a fourth action video based on the second network model according to the reference action image, the reference posture image and the second posture video; Adjusting the second network model based on the second action video and the fourth action video to obtain a third network model; In response to satisfying the training end condition, the third network model is used as a video redirection model.

2. The method according to claim 1, It is characterized in that The first posture video includes a plurality of posture images; The determining a third action video based on a first network model according to the reference action image, the reference posture image and the first posture video includes: Determining, based on the first network model, an action image corresponding to the first posture image according to the reference action image, the reference posture image, and a first posture image in the first posture video; For any posture image in the first posture video except the first posture image, the action image corresponding to any posture image is determined based on the reference action image, the reference posture image, any posture image, the previous posture image of any posture image and the action image corresponding to the previous posture image.

3. The method according to claim 2, It is characterized in that The step of determining the action image corresponding to any one of the posture images according to the reference action image, the reference posture image, any one of the posture images, a previous posture image of any one of the posture images, and the action image corresponding to the previous posture image comprises: Determine a first action image according to the reference action image, the reference posture image and any one posture image; determining a second action image according to the reference action image, any one of the posture images, a previous posture image of any one of the posture images, and an action image corresponding to the previous posture image; The action image corresponding to any one of the posture images is determined according to the first action image and the second action image.

4. The method according to claim 3, It is characterized in that The step of determining a first action image according to the reference action image, the reference posture image and any one posture image comprises: Determine a fused posture image according to the reference posture image and any one of the posture images; The first action image is determined according to the reference action image and the fused posture image.

5. The method according to claim 3, It is characterized in that The determining of the second action image according to the reference action image, any one of the posture images, a previous posture image of any one of the posture images, and an action image corresponding to the previous posture image includes: Extracting features of the reference action image, features of any one of the posture images, features of a posture image previous to any one of the posture images, and features of the action image corresponding to the previous posture image; Determining a fusion feature according to the feature of the reference action image, the feature of any one of the posture images, the feature of a previous posture image of any one of the posture images, and the feature of the action image corresponding to the previous posture image; The second action image is determined according to the fusion feature.

6. The method according to claim 1, It is characterized in that The first action video includes action images corresponding to each posture image in the first posture video, and the third action video includes action images corresponding to each posture image in the first posture video; The adjusting the first network model based on the first action video and the third action video to obtain a second network model includes: Determining a loss value of each first image set, the first image set includes a third action image, a fourth action image, a fifth action image, and a sixth action image, the third action image and the fourth action image are action images corresponding to first adjacent posture images in the first action video, the fifth action image and the sixth action image are action images corresponding to the first adjacent posture images in the third action video, and the first adjacent posture images are two adjacent posture images in the first posture video; Based on the loss values ​​of the first image sets, the first network model is adjusted to obtain a second network model.

7. The method according to claim 6, It is characterized in that The third action image and the fifth action image are action images corresponding to the same action image in the first action video, and the fourth action image and the sixth action image are action images corresponding to the same action image in the first action video; The determining of the loss value of each first image set comprises: For any one of the first image sets, determining a first loss value based on a third motion image and a fifth motion image in the any one of the first image sets; determining a second loss value based on a fourth motion image and a sixth motion image in any one of the first image sets; determining a third loss value based on a third motion image, a fourth motion image, a fifth motion image, and a sixth motion image in any one of the first image sets; Based on the first loss value, the second loss value and the third loss value, a loss value of any one of the first image sets is determined.

8. The method according to any one of claims 1 to 7, It is characterized in that The second posture video includes a plurality of posture images; The step of determining a fourth action video based on the second network model according to the reference action image, the reference posture image and the second posture video includes: Determining, based on the second network model, an action image corresponding to the first posture image according to the reference action image, the reference posture image, and the first posture image in the second posture video; For any posture image in the second posture video except the first posture image, the action image corresponding to any posture image is determined based on the reference action image, the reference posture image, any posture image, the previous posture image of any posture image and the action image corresponding to the previous posture image.

9. The method according to any one of claims 1 to 7, It is characterized in that The second action video includes action images corresponding to each posture image in the second posture video, and the fourth action video includes action images corresponding to each posture image in the second posture video; The adjusting the second network model based on the second action video and the fourth action video to obtain a third network model includes: determining a loss value of each second image set, wherein the second image set includes a seventh action image, an eighth action image, a ninth action image, and a tenth action image, wherein the seventh action image and the eighth action image are action images corresponding to second adjacent posture images in the second action video, the ninth action image and the tenth action image are action images corresponding to the second adjacent posture images in the fourth action video, and the second adjacent posture images are two adjacent posture images in the second posture video; Based on the loss values ​​of the respective second image sets and the parameters of the first network model, the second network model is adjusted to obtain a third network model.

10. The method according to any one of claims 1 to 7, It is characterized in that The step of obtaining data of the sample object includes: Get action videos of sample objects; selecting the reference action image, the first action video, and the second action video from the action videos of the sample object; acquiring the reference posture image based on the reference action image; Acquire the first posture video based on the first action video; The second posture video is obtained based on the second action video.

11. The method according to any one of claims 1 to 7, It is characterized in that After the second network model is adjusted based on the second action video and the fourth action video to obtain the third network model, the method further includes: In response to not meeting the training end condition, determining a fourth network model based on the third network model according to the data of the sample object; In response to the training end condition being met, the fourth network model is used as a video redirection model.

12. A video redirection method, It is characterized in that The method comprises: Acquire a reference action image of a target object, a reference posture image of the target object, a first action video of the target object, a posture video of the target object, and a posture video of an original object; Determining a second action video of the target object based on a video redirection model according to a reference action image of the target object, a reference posture image of the target object, and a posture video of the target object, wherein the video redirection model is obtained according to any one of the methods described in claims 1 to 11; Adjusting the video redirection model based on the first action video of the target object and the second action video of the target object to obtain a target model; Based on the reference action image of the target object, the reference posture image of the target object and the posture video of the original object, a target video is determined based on the target model.

13. The method according to claim 12, It is characterized in that The step of acquiring a reference action image of the target object, a reference posture image of the target object, a first action video of the target object, a posture video of the target object, and a posture video of the original object includes: Acquire a first action video of the target object and an action video of the original object; acquiring a posture video of the target object based on a first action video of the target object, wherein the first action video of the target object includes a reference action image of the target object, and the posture video of the target object includes a reference posture image of the target object; A posture video of the original object is acquired based on the action video of the original object.

14. A training device for a video redirection model, It is characterized in that The device comprises: An acquisition module, configured to acquire data of a sample object, wherein the data of the sample object includes a reference action image, a reference posture image, a first action video, a first posture video, a second action video, and a second posture video; a determination module, configured to determine a third action video based on a first network model according to the reference action image, the reference posture image and the first posture video; an adjustment module, configured to adjust the first network model based on the first action video and the third action video to obtain a second network model; The determination module is further configured to determine a fourth action video based on the second network model according to the reference action image, the reference posture image and the second posture video; The adjustment module is further used to adjust the second network model based on the second action video and the fourth action video to obtain a third network model; The determination module is further configured to use the third network model as a video redirection model in response to satisfying a training end condition.

15. A video redirection device, It is characterized in that The device comprises: An acquisition module, used for acquiring a reference action image of a target object, a reference posture image of the target object, a first action video of the target object, a posture video of the target object, and a posture video of an original object; a determination module, configured to determine a second action video of the target object based on a video redirection model according to a reference action image of the target object, a reference posture image of the target object, and a posture video of the target object, wherein the video redirection model is obtained according to any one of the methods described in claims 1 to 11; An adjustment module, configured to adjust the video redirection model based on the first action video of the target object and the second action video of the target object to obtain a target model; The determination module is further configured to determine a target video based on the target model based on a reference action image of the target object, a reference posture image of the target object and a posture video of the original object.

16. An electronic device, It is characterized in that The electronic device includes a processor and a memory, wherein at least one program code is stored in the memory, and the at least one program code is loaded and executed by the processor so that the electronic device implements the training method of the video redirection model as described in any one of claims 1 to 11, or implements the video redirection method as described in claim 12 or 13.

17. A computer-readable storage medium, It is characterized in that At least one program code is stored in the computer-readable storage medium, and the at least one program code is loaded and executed by the processor so that the computer implements the training method of the video redirection model as described in any one of claims 1 to 11, or implements the video redirection method as described in claim 12 or 13.

18. A computer program product, It is characterized in that The computer program product stores at least one computer instruction, and the at least one computer instruction is loaded and executed by a processor so that the computer implements the training method of the video redirection model described in any one of 1 to 11 above, or implements the video redirection method described in claim 12 or 13.

Citation Information

Patent Citations

  • Video live-broadcasting data transmission method and apparatus, storage medium and electronic apparatus

    CN107623723A

  • Expression redirection training method and device, electronic equipment and readable storage medium

    CN113408452A

  • Activity Surface Detection, Display and Enhancement of a Virtual Scene

    US20170206693A1