Video generation method and device, equipment, storage medium and program product

Abstract

The embodiment of the invention provides a video generation method and device, equipment, a storage medium and a program product. The method includes determining a three-dimensional attitude representation of a target object based on a source image of the target object. And based on the target audio, determining a reference attitude action sequence corresponding to the target audio. And adjusting the reference attitude action sequence based on the three-dimensional attitude representation to obtain a target attitude action sequence corresponding to the target object. And based on the source image of the target object and the target attitude action sequence, generating a target video of the target object, the target video representing that the target object executes an attitude action corresponding to the target attitude action sequence while speaking the target audio. Therefore, the video generation quality can be improved.

Description

Technical Field

[0001] Example embodiments of the present disclosure generally relate to the field of computers, and more particularly, to methods, devices, apparatuses, storage media, and program products for video generation. Background Art

[0002] With the continuous development of speech-driven image generation technology, this technology has shown wide potential in application scenarios such as virtual character generation, video conferencing, and smart assistants. Traditional technologies usually focus on image generation related to facial movements. However, facial movements alone often cannot convey accurate emotions and ideas, thus affecting the quality of the video. Summary of the invention

[0003] In a first aspect of the present disclosure, a method for video generation is provided. The method may include: determining a three-dimensional posture representation of a target object based on a source image of the target object. Determining a reference posture action sequence corresponding to the target audio based on a target audio. Adjusting the reference posture action sequence based on the three-dimensional posture representation to obtain a target posture action sequence corresponding to the target object. Generating a target video of the target object based on the source image of the target object and the target posture action sequence, the target video indicating that the target object performs a posture action corresponding to the target posture action sequence while speaking the target audio.

[0004] In a second aspect of the present disclosure, a device for video generation is provided. The device may include: a three-dimensional posture representation determination module, configured to determine the three-dimensional posture representation of the target object based on the source image of the target object. A reference posture action sequence determination module, configured to determine the reference posture action sequence corresponding to the target audio based on the target audio. The target posture action sequence determination module is configured to adjust the reference posture action sequence based on the three-dimensional posture representation to obtain a target posture action sequence corresponding to the target object. A video generation module, configured to generate a target video of the target object based on the source image of the target object and the target posture action sequence, the target video indicating that the target object performs a posture action corresponding to the target posture action sequence while speaking the target audio.

[0005] In a third aspect of the present disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory, the at least one memory is coupled to the at least one processing unit and stores instructions for execution by the at least one processing unit. When the instructions are executed by the at least one processing unit, the electronic device executes the method of the first aspect.

[0006] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein a computer program is stored on the medium, and when the computer program is executed by a processor, the method of the first aspect is implemented.

[0007] In a fifth aspect of the present disclosure, a computer program product is provided, which includes computer executable instructions, and when the computer executable instructions are executed by a processor, the method of the first aspect is implemented.

[0008] It should be understood that the content described in this section is not intended to limit the key features or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0010] Figure 1 A schematic diagram showing an example environment in which embodiments of the present disclosure can be implemented;

[0011] Figure 2 A flowchart of a method for video generation according to some embodiments of the present disclosure is shown;

[0012] Figure 3A A schematic diagram showing a method for video generation according to some embodiments of the present disclosure is shown;

[0013] Figure 3B An example principle schematic diagram of a rendering model according to some embodiments of the present disclosure is shown;

[0014] Figure 4A An example diagram of a video coding model according to some embodiments of the present disclosure is shown;

[0015] Figure 4B A schematic diagram of the pre-training principle of a video coding model according to some embodiments of the present disclosure is shown;

[0016] Figure 5A A schematic diagram of training a rendering model according to some embodiments of the present disclosure is shown;

[0017] Figure 5B A schematic diagram of training a weight determination model according to some embodiments of the present disclosure is shown;

[0018] Figure 6 A schematic structural block diagram of an apparatus for video generation according to some embodiments of the present disclosure is shown; and

[0019] Figure 7 A block diagram of an electronic device in which one or more embodiments of the present disclosure may be implemented is shown. DETAILED DESCRIPTION

[0020] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0021] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below.

[0022] Herein, unless explicitly stated, executing a step “in response to A” does not mean executing the step immediately after “A” but may include one or more intermediate steps.

[0023] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition, use, storage or deletion of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0024] It is understandable that before using the technical solutions disclosed in the various embodiments of the present disclosure, the types, scopes of use, usage scenarios, etc. of the information involved in the present disclosure should be informed to relevant users and their authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations. The relevant users may include any type of right holders, such as individuals, enterprises, and groups.

[0025] For example, in response to receiving an active request from a user, a prompt message is sent to the relevant user to clearly prompt the relevant user that the operation requested to be performed will require obtaining and using the information of the relevant user, so that the relevant user can independently choose whether to provide information to software or hardware such as an electronic device, application, server or storage medium that executes the operation of the technical solution of the present disclosure based on the prompt message.

[0026] As an optional but non-limiting implementation, in response to receiving an active request from a relevant user, a prompt message is sent to the relevant user, for example, in the form of a pop-up window, in which the prompt message may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to select "agree" or "disagree" to provide information to the electronic device.

[0027] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0028] As used herein, the term "model" can learn the association between the corresponding input and output from the training data, so that after the training is completed, the corresponding output can be generated for a given input. The generation of the model can be based on machine learning technology. Deep learning is a machine learning algorithm that processes inputs and provides corresponding outputs by using multi-layer processing units. A neural network model is an example of a model based on deep learning. In this article, "model" may also be referred to as "machine learning model", "learning model", "machine learning network" or "learning network", and these terms are used interchangeably in this article.

[0029] Figure 1 1 is a schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented. Figure 1 As shown, environment 100 may include electronic device 110 .

[0030] In the example environment 100, the electronic device 110 can obtain input information 102. The input information 102 includes at least a target audio 114 and a source image 113 of a target object. As an example, the target object may include humans, animals, cartoon characters, and virtual characters, etc. The electronic device 110 can generate a three-dimensional gesture representation corresponding to the target object using a target model 115 based on the source image 113 of the target object. In addition, the electronic device 110 can also generate a reference gesture action sequence 307 synchronized with the target audio 114 using a target model 115 based on the target audio 114. The reference gesture action sequence 307 corresponds to the target audio and is used to represent gestures, postures, etc. corresponding to the target audio. Based on the reference gesture action sequence 307 and the three-dimensional gesture representation corresponding to the target object, a gesture action sequence corresponding to the target object can be obtained. The source image 113 of the target object is mapped to the gesture action sequence corresponding to the target object, and the target video 104 of the target object can be obtained. The target video 104 can show gesture actions corresponding to the reference gesture action sequence 307 in synchronization with the target audio 114. Figure 1 Only one target model 115 is shown as an example. In practice, multiple different target models 115 may collaborate to complete the generation of the video.

[0031] The electronic device 110 can be any type of mobile terminal, fixed terminal or portable terminal, including a mobile phone, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a media computer, a multimedia tablet, a personal communication system (PCS) device, a personal navigation device, a personal digital assistant (PDA), an audio / video player, a digital camera / camcorder, a television receiver, a radio broadcast receiver, an e-book device, a gaming device, or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof. In some embodiments, the electronic device 110 can also support any type of interface for the user (such as a "wearable" circuit, etc.). The server device (not shown) can be various types of computing systems / servers that can provide computing power, including but not limited to mainframes, edge computing nodes, computing devices in cloud environments, and the like. The server device can, for example, provide background services for the application of the electronic device 110.

[0032] It should be understood that the structure and function of the various elements in the environment 100 are described for exemplary purposes only and do not imply any limitation on the scope of the present disclosure.

[0033] In conversations or speeches, speakers often use co-articulating gestures to express their thoughts and emotions. These co-articulating gestures help bridge the communication gap and enhance mutual intimacy and the credibility of speech. Therefore, in speech-driven image generation technology, giving virtual objects co-articulating movements is of great significance for human-computer interaction and also helps to enhance the realism of virtual reality environments. Therefore, some related technologies explore the generation of co-articulating gestures by generating gesture sequences aligned with speech. However, although these generated gesture sequences can generate movements, they cannot reflect the personalized appearance characteristics of the speaker, thus limiting their potential applications. Some studies overcome this limitation by synthesizing gesture sequences and specific speaker videos, but these studies are limited to specific speakers and lack generalization.

[0034] In an embodiment of the present disclosure, an improved scheme for video generation is proposed. In the scheme, the electronic device determines a three-dimensional posture representation of the target object based on a source image of the target object. Based on the target audio, a reference posture action sequence corresponding to the target audio is determined. The reference posture action sequence is adjusted based on the three-dimensional posture representation to obtain a posture action sequence corresponding to the target object. Based on the source image of the target object and the target posture action sequence, a target video of the target object is generated, and the video represents that the target object performs a posture action corresponding to the target posture action sequence while speaking the target audio.

[0035] Through the above process, the source image is used to determine the three-dimensional posture representation of the target object, so that the generated video can reflect the personalized appearance characteristics of the target object, avoiding the limitation of only generating posture sequences without appearance information. Secondly, by generating a synchronized baseline posture action sequence based on the target audio, it is ensured that the actions of the target object in the video can reflect the ideas and emotions of the target audio, enhancing the naturalness and expressiveness of the generated video. In addition, the improved scheme also shows good generalization. It only needs to provide source images of different target objects to generate corresponding personalized videos, which can adapt to the generation needs of multiple speakers. Therefore, the scheme shows a wider potential and application value in application scenarios such as virtual character generation and intelligent assistants, and effectively improves the quality and applicability of video generation.

[0036] Figure 2 FIG. 2 shows an example process of a method 200 for video generation according to some embodiments of the present disclosure. For ease of discussion, reference will be made to Figure 1 In the environment 100, the video generation can be completed by the electronic device 110, but some operations can be performed by requesting a server device (not shown) (for example, the determination of the three-dimensional posture representation, the determination of the reference posture action sequence, the generation of the video, or the training process of part of the model can be implemented at the server device).

[0037] The electronic device 110 may obtain input information for generating a video as a trigger condition for generating a video. The input information may include at least a source image 113 containing a target object and a target audio 114. As an example, the target object may include humans, animals, cartoon characters, and virtual characters, etc. In block 201, the electronic device 110 determines a three-dimensional posture representation of the target object based on the source image of the target object.

[0038] Figure 3A A process diagram 300A of a method 300 for video generation according to some embodiments of the present disclosure is shown. The source image 113 containing the target object may be a static image, or a video image obtained by performing frame extraction from a video containing the target object.

[0039] The source image 113 including the target object may be used to determine the 3D pose representation 304 of the target object. For example, the source image 113 including the target object may be processed using the 3D pose generation model 320 to obtain the 3D pose representation 304 of the target object. The 3D pose generation model 320 may detect and locate the skeletal key points of the target object in the source image 113, and map the skeletal key points to the 3D space to determine the 3D pose representation 304 of the target object. As an example, the 3D pose generation model 320 may correspond to an Expressive Human Pose and Shape Predictor (EHPS).

[0040] In block 202 , the electronic device 110 may first perform feature extraction on the target audio 114 to obtain a feature representation corresponding to the audio, and then process the feature representation to generate a reference gesture action sequence 307 .

[0041] Each reference gesture in the reference gesture sequence 307 is a normalized gesture, corresponding to the target audio 114. The reference gesture is a general gesture representation method, which defines the movement direction and movement amplitude of the target object in three-dimensional space, without being restricted by specific body shape, proportion and other characteristics.

[0042] The versatility of this gesture representation method makes it applicable to different target objects, and in subsequent steps, it can be adjusted to adapt to the specific gesture features of different objects. As an example, each reference gesture action in the reference gesture action sequence 307 can be in a three-dimensional form or a two-dimensional form. In the present application example, each reference gesture action in the reference gesture action sequence 307 is in a three-dimensional form as an example to be explained and introduced.

[0043] In block 203, the electronic device 110 adjusts the reference gesture action sequence 307 based on the three-dimensional gesture representation 304 to obtain a target gesture action sequence 308 corresponding to the target object. As mentioned above, the reference gesture actions in the reference gesture action sequence 307 are not limited by specific body shape, proportions, etc. Therefore, in order to better express the target object, the reference gesture action sequence 307 can be adjusted based on the three-dimensional gesture representation 304.

[0044] The three-dimensional posture representation 304 of the target object may indicate the corresponding bone proportions and body shape features of the target object. The three-dimensional posture representation 304 of the target object is used to adapt and adjust each reference posture action in the reference posture action sequence 307, thereby generating a target posture action sequence 308 corresponding to the bone proportions and body shape features of the target object. As an example, the adaptation and adjustment may include adjusting the lengths and angles between each key point in the reference posture action.

[0045] This adjustment ensures that the target posture action sequence 308 generated in the end can not only express the posture action corresponding to the speech, but also match the body shape and body proportions of the target object, ensuring that the generated video has personalized appearance features and natural action performance.

[0046] In box 204 , the electronic device 110 generates a target video 350 of the target object based on the source image 113 of the target object and the target gesture sequence 308 , wherein the target video 350 shows that the target object performs a gesture corresponding to the target gesture sequence 308 while speaking the target audio 114 .

[0047] The electronic device 110 may determine a visual feature representation of the target object using the source image 113 of the target object. The visual feature representation may indicate the appearance characteristics of the target object, such as facial features, body shape, clothing, etc. These appearance characteristics will be used to ensure that the generated target video retains the personalized visual effect of the target object.

[0048] The electronic device 110 combines the target gesture action sequence 308 with the visual feature representation of the target object, and generates a dynamic video frame corresponding to the gesture action through a rendering process. In the generated target video 350, the target object not only makes a sound synchronously according to the target audio 114, but also accompanies the gestures and actions associated with the audio content of the target audio 114, showing a natural and personalized performance.

[0049] The above process allows the facial expressions, body movements and voice of the target object to be closely integrated, thereby presenting a more vivid and realistic effect in the target video, meeting the needs of application scenarios such as virtual character generation and intelligent assistants. In addition, the above process also shows good generalization. It only needs to provide source images of different target objects to generate corresponding personalized target videos, which can meet the generation needs of various speakers. Therefore, this solution shows a wider potential and application value in application scenarios such as virtual character generation and intelligent assistants, and effectively improves the quality and applicability of video generation.

[0050] In the process of determining the target gesture action sequence 308 corresponding to the target audio 114, the electronic device 110 can first determine the feature representation of the target audio 114. Based on the feature representation of the target audio 114, the reference gesture action sequence 307 corresponding to the target audio 114 is determined using the action generation model 311. In some embodiments, the action generation model 311 can be implemented based on a diffusion model, and therefore can also be referred to as a diffusion action generation model. Of course, other types of model structures can also be selected to construct the action generation model, as long as the model can construct a gesture action from the audio features.

[0051] The electronic device 110 may process the target audio 114 using an audio encoding model (Audio Encoder) 310 to obtain a feature representation of the target audio 114. The feature representation of the target audio 114 may indicate semantic information, speech rhythm, and emotional changes corresponding to the target audio 114, and thus may be used as reference information for generating a reference gesture action sequence 307.

[0052] By processing the feature representation of the target audio 114 using the action generation model 311, a reference gesture action sequence 307 corresponding to the target audio 114 can be generated. The reference gesture action sequence 307 is a standardized gesture action representation. Since the reference gesture action sequence 307 is generated based on the feature representation of the target audio 114, it can reflect the gesture actions related to the content expressed by the target audio 114. The reference gesture actions in the reference gesture action sequence 307 are standardized and do not yet reflect the personalized characteristics of the target object, such as the skeletal proportions, body shape features, etc. of the target object. The reference gesture action sequence 307 can include multiple reference gesture actions, which correspond to each audio unit in the target audio (for example, with audio frames as audio units). Each reference gesture action corresponds to a specific segment in the audio, reflecting the general body movements that should be performed under the audio segment.

[0053] Through the above process, the audio coding model can capture the rhythm and emotional characteristics of the target audio 114, so that the reference posture action sequence 307 can be highly consistent with the content, emotional expression, etc. of the target audio 114.

[0054] As mentioned above, the reference gesture action sequence 307 is standardized. In order to obtain the gesture action corresponding to a specific target object, it is necessary to adjust the reference gesture action sequence 307 using the three-dimensional gesture representation 304 of the target object, so as to obtain the target gesture action sequence corresponding to the target object, so as to reflect the visual characteristics of the gesture of the target object. Based on this, the electronic device 110 can determine the three-dimensional space coordinates corresponding to each key point in the three-dimensional gesture representation 304 of the target object. Based on the three-dimensional space coordinates corresponding to each key point, the angle representation and length representation between each key point are determined. Using the angle representation and length representation between each key point, each reference gesture action included in the reference gesture action sequence 307 is adjusted to obtain the target gesture action sequence 308.

[0055] Each key point in the three-dimensional posture representation 304 of the target object can correspond to a joint in the target object's skeletal system, such as the joints at the shoulder, elbow, wrist, and five fingers. Based on the three-dimensional posture representation 304 of the target object, the electronic device 110 can determine the coordinate position of each key point of the target object in the three-dimensional space. Based on the coordinate position of these key points in the three-dimensional space, the electronic device 110 can calculate the angle representation and length representation between each key point. These angle representations and length representations reflect the skeletal proportions and body structure of the target object, such as the presentation state of the five fingers, the extension angle of the arms, the extension state of the five fingers, the length of the legs, etc. Through this information, the spatial relationship between the various parts of the target object can be accurately determined.

[0056] Based on the angle representation and length representation between each key point in the three-dimensional posture representation 304 of the target object, each reference gesture action in the reference gesture action sequence 307 can be adjusted. Specifically, each reference gesture action in the reference gesture action sequence 307 will be scaled and adjusted (such as length adjustment, fatness and thinness adjustment) to obtain a target gesture action sequence 308 that conforms to the skeletal proportions and body shape characteristics of the target object. Each gesture action in the target gesture action sequence 308 will be consistent with the personalized visual features of the target object, thereby ensuring that in the generated target video 350, the gesture action of the target object can be synchronized with the audio content of the target audio 114 and can also be visually matched with the body features of the target object.

[0057] As mentioned above, each reference gesture action in the reference gesture action sequence 307 can be in a three-dimensional form. After adjusting each reference gesture action in the reference gesture action sequence 307 using the three-dimensional gesture representation 304, a gesture action sequence in a three-dimensional form can be obtained. Based on this, the gesture action sequence in a three-dimensional form can also be converted into a two-dimensional representation (a target gesture action sequence 308 in a two-dimensional form) using camera parameters to adapt to the two-dimensional characteristics of the final video generation. The camera parameters are used to determine the viewing angle and projection mode in the three-dimensional space, thereby ensuring that the generated target gesture action sequence 308 in a two-dimensional form can accurately reflect the action and characteristics of the target object in the three-dimensional space.

[0058] Through this adjustment method, the generated target video 350 can reflect the real appearance and natural movements of the target object, making the performance of the target object in the target video 350 more realistic and credible, and enhancing the personalization and visual effects of the video.

[0059] After adjusting the reference gesture action sequence 307 based on the three-dimensional gesture representation 304 to obtain the target gesture action sequence 308 corresponding to the target object, the target gesture action sequence 308 can be rendered based on the source image 113 of the target object to generate the target video 350 of the target object. As an example, the electronic device 110 can determine the visual feature representation of the target object based on the source image 113 of the target object. Based on the visual feature representation of the target object, each gesture action in the target gesture action sequence 308 is rendered to obtain the target video 350 of the target object.

[0060] The electronic device 110 can extract visual feature representations of the target object based on the source image 113 containing the target object. These visual feature representations can include, but are not limited to, information such as facial features, body appearance, clothing color and texture of the target object. With the help of the visual feature representations, it can be ensured that the subsequently generated target video 350 visually accurately reflects the personalized characteristics of the target object.

[0061] The electronic device 110 represents each gesture in the target gesture action sequence 308 based on the visual feature of the target object. During the rendering process, the electronic device 110 combines the visual features of the target object with each gesture in the target gesture action sequence 308 to generate corresponding image frames, so that the target object can show the action corresponding to the target gesture action sequence 308 in each image frame. In this way, the target object presented in the target video 350 can not only move according to the specified gesture action, but also retain its appearance characteristics and dynamic details. In order to enhance the visual effect of the video, the rendering process can also apply the technology of illumination processing, texture detail enhancement, etc. to improve the realism and expressiveness of the target video 350. Finally, the electronic device 110 can combine these rendered image frames into a continuous target video 350, so that the target object shows the gesture and action corresponding to the target audio 114 in the target video 350.

[0062] In some embodiments of the present disclosure, the source image of the target object includes multiple source images 113, and the presentation angles of the target object in each source image 113 are different from each other. In this case, taking a given gesture action in the target gesture action sequence 308 as an example, the generation process of the target video 350 is introduced. For a given gesture action in the target gesture action sequence 308, the electronic device 110 determines the weight parameters corresponding to each of the multiple source images based on the action difference and the given gesture action, wherein the action difference is determined based on the given gesture action and the source gesture actions corresponding to the multiple source images 113. Based on the multiple source images 113 and the weight parameters corresponding to each of the multiple source images 113, a visual feature representation of the target object is determined. The given gesture action is rendered based on the visual feature representation of the target object to obtain a video clip corresponding to the given gesture action, and the video clip is a part of the target video of the target object.

[0063] Figure 3B A rendering process schematic diagram 300B of a rendering model 330 according to some embodiments of the present disclosure is shown. For multiple source images 113, the source gesture action corresponding to the target object in each source image 113 can be first determined, and the gesture action feature representation 113-a of the source gesture action can be obtained. Taking 4 source images as an example, they correspond to the first source image to the fourth source image respectively. As an example, the front image, back image, and side image (left side, right side) of the target object can be presented in multiple source images respectively.

[0064] For a given gesture action 308-1 in the target gesture action sequence 308, the electronic device 110 can extract a motion difference feature representation 362 corresponding to each source image from the given gesture action and the source gesture actions corresponding to the plurality of source images, the motion difference feature representation indicating the motion difference. Based on the motion difference feature representation corresponding to each source image, a weight parameter corresponding to each source image is determined.

[0065] For example, a gesture action feature representation 308-a of a given gesture action 308-1 and a gesture action feature representation 113-a of a source gesture action can be determined. As an example, the gesture action feature representation can indicate information such as coordinates of key points. Taking the first source image as an example, the action difference determination model 360 can compare the action difference between the given gesture action 308-1 and the source gesture action in the first source image based on the gesture action feature representation. Based on the action difference, an action difference feature representation 362 of the first source image can be determined. The action difference can indicate information such as gesture angles and joint position differences. Similarly, the action difference determination model 360 can obtain action difference feature representations 362 of other source images respectively.

[0066] Based on the action difference feature representation 362 of each source image, the electronic device 110 uses the weight determination model 364 to evaluate the importance of each source image to determine the weight of each source image in generating the visual feature representation of the target object. For example, if the action difference between the given gesture action 308-1 and the source gesture action in the first source image is very large (for example, the given gesture action needs to present the left face, but the source gesture action in the first source image correspondingly presents the right face), then the weight determination model 364 can determine that the first source image reflects less of the given gesture action, so the weight parameter of the first source image is lower. For another example, if the action difference between the given gesture action 308-1 and the source gesture action in the second source image is very small, then the weight determination model 364 can determine that the weight parameter of the second source image is higher.

[0067] The weight parameter can indicate the relative importance of each source image in the process of generating the visual feature representation of the target object. The higher the weight parameter, the more the source image meets the requirements in reflecting the specific posture or perspective of the target object. Therefore, the reference degree can be understood as a measure of the contribution of each source image in synthesizing the final visual features. Specifically, the reference degree can be further refined to the pixel level. The information contained in different pixels in each source image is also of different importance in generating the visual features of the target object. Therefore, the electronic device 110 generates a pixel-level weight representation based on the importance of each pixel in the multiple source images. These pixel-level weights determine the contribution of a certain pixel of a certain source image to the visual features of the target object when synthesizing the visual feature representation.

[0068] For example, when the target object's posture needs to turn to the left, the left side information in a certain source image may be more important than other parts, and the pixel weight of the left side area will be higher. For the back part of the target object in the same source image, since this part is not the focus in this posture action, its pixel weight will be lower. Under such pixel-level weight control, the image decoder 369 can ensure that in the process of determining the visual features of the target object, the feature information that best reflects the specific posture and viewing angle of the target object can be accurately selected.

[0069] By processing each source image 113 using the image encoder 366, a feature encoding representation of the source image 113 can be obtained. By processing a given gesture action 308-1 using the gesture encoder 368, a motion feature encoding representation of the given gesture action 308-1 can be obtained.

[0070] The feature coding representation of the source image 113, the action feature coding representation of the given gesture action 308-1, and the weight of each source image are used as input information of the image decoder 369, and the image decoder 369 can obtain the video segment 350-1 corresponding to the given gesture action 308-1. Similarly, when rendering gesture actions, the electronic device 110 renders each gesture action in the target gesture action sequence 308 frame by frame. Each gesture action corresponds to a video segment, and each gesture action is rendered in sequence, and finally a complete target video 350 is generated by combining them.

[0071] Through the above process, the visual feature representation of the target object is determined based on each source image and its corresponding weight parameters to ensure that the visual features can fully combine the source image information from different angles, thereby making up for the shortcomings of a single image perspective and obtaining a more complete visual feature representation of the target object.

[0072] Combination Figure 3A As shown, generating the reference posture action sequence 307 involves an audio coding model 310 and an action generation model 311. The audio coding model 310 can be trained by pre-training.

[0073] After pre-training is completed, the audio coding model 310 and the action generation model 311 are jointly trained by fine-tuning. The fine-tuning process includes processing the (first or second) video sample, extracting the audio sample and the benchmark action sequence sample from the video sample. The audio coding model 310 processes the audio sample to extract the audio features. The action generation model 311 generates the corresponding action sequence prediction results based on the audio features. Then, the action sequence prediction results are compared with the benchmark action sequence samples, and the loss function is calculated. The loss function is used to jointly tune the parameters of the audio coding model 310 and the action generation model 311.

[0074] The following first introduces the tuning process of the action generation model 311. The action generation model 311 is used to generate an action sequence x under the condition of a given audio feature sequence a. The action generation model 311 generates a reference gesture action by step-by-step denoising. The generation process starts with random noise, and by gradually applying diffusion steps, the audio features are gradually guided to gesture actions that reflect the target audio content. The training goal of the action generation model 311 is to minimize the mean square error in the noise space. The training goal of the action generation model 311 can be expressed as follows:

[0075]

[0076] ∈ can represent noise, which conforms to the distribution of random noise, that is, ∈~N(0,I). ∈ θ can represent the denoising network. t can represent a random time step, x tmay represent a noisy gesture sequence in time step t. c may represent an input obtained by processing an audio sequence through the audio coding model 310. The training goal is to minimize the noise ∈ to ensure that the generated gesture sequence is consistent with the distribution of the real gesture sequence.

[0077] The pre-training process of the audio coding model 310 is described in detail below. The electronic device 110 extracts an audio sample and a benchmark action sequence sample from a first video sample containing a first object sample. Construct a plurality of positive samples, each of which includes a first audio segment and a first benchmark action sequence, the first audio segment is obtained by segmenting the audio sample, the first benchmark action sequence segment is obtained by segmenting the benchmark action sequence sample, and the first audio segment and the first benchmark action sequence segment are aligned in time. Construct a plurality of negative samples, each of which includes a second audio segment and a second benchmark action sequence segment, the second audio segment is obtained by segmenting the audio sample, the second benchmark action sequence segment is obtained by segmenting the benchmark action sequence sample, and the second audio segment and the second benchmark action sequence segment are not aligned in time. Use a plurality of positive samples and a plurality of negative samples to pre-train the audio coding model through comparative learning.

[0078] The electronic device 110 can extract audio samples and reference action sequence samples from the first video sample containing the first object sample. These audio samples and reference action sequence samples reflect the synchronization relationship between the audio content and the gesture action during the speaking process of the first object sample.

[0079] Figure 4A A schematic diagram 400A of a principle of an audio coding model 310 according to some embodiments of the present disclosure is shown. The audio coding model 310 includes a convolutional layer 405, which is used to process an audio sample 401 to obtain a first audio feature. In addition, the audio coding model 310 also includes a coding layer 402 and a feature extraction layer 404. The coding layer 402 is used to process the audio sample 401 to obtain a coded representation of the audio sample 401. The feature extraction layer 404 processes the coded representation of the audio sample 401 to obtain a second audio feature. By fusing the first audio feature and the second audio feature, an audio feature representation of the audio sample 401 can be obtained.

[0080] In some embodiments, the encoding layer 402 may use a Hubert feature encoder, which may encode the audio sample 401 into a series of semantic tags. As an example, the audio sample 401 may be represented as a series of numbers such as "32 67 354..." after being processed by the Hubert feature encoder. In some embodiments, the feature extraction layer 404 may be a context encoder (Context Transformer), which performs further feature extraction on the result output by the encoding layer 402.

[0081] Figure 4B Schematic diagram 400B of the pre-training principle of the audio coding model 310 according to some embodiments of the present disclosure is shown. The pre-training process of the audio coding model 310 may include training of the coding layer 402 and the feature extraction layer 404. Before training, the electronic device 110 may first construct positive samples and negative samples for training. For example, the audio sample 401 and the reference action sequence sample 403 may first be segmented into equal lengths to obtain multiple audio segments a1, a2, ..., a n and the corresponding multiple benchmark action sequence fragments x1,x2,...,x n The audio segment a1 and the reference action sequence segment x1 are of equal length and aligned on the time axis. Similarly, the audio segment a2 and the reference action sequence segment a2 are of equal length and aligned on the time axis, and so on to the audio segment a n and the benchmark action sequence fragment x n Equal length and aligned on the time axis.

[0082] For each audio clip, we can get the corresponding audio feature representation (T1, T2, ..., T n ). Similarly, by processing each benchmark action sequence segment using the action encoder 407, the corresponding action feature representation (S1, S2, ..., S n ). The determination of the audio feature representation and the action feature representation can be expressed as follows:

[0083]

[0084] LN can represent layer normalization, L speech and L gesture can represent linear transformation processing respectively. speech and f speech can represent the processing of the semantic encoder (encoding layer 402 and feature extraction layer 404) and the motion encoder 407 respectively. a can represent an audio sample, a={a1, a2, ..., a n}. x can represent the benchmark action sequence sample x = {x1, x2, ..., x n}.

[0085] The multiple positive samples may be audio feature representations and action feature representations aligned on the time axis. For example, in the similarity matrix 410, the feature pair consisting of the corresponding audio feature representation T1 of the first audio segment and the corresponding action feature representation S1 of the first benchmark action sequence segment may correspond to the first positive sample. The feature pair consisting of the corresponding audio feature representation T2 of the second audio segment and the corresponding action feature representation S2 of the second benchmark action sequence segment may correspond to the second positive sample. With such pairs, the corresponding audio feature representation T1 of the nth audio segment and the corresponding action feature representation S2 of the second benchmark action sequence segment may correspond to the second positive sample. nThe action feature representation S corresponding to the nth benchmark action sequence segment n The composed feature pair can correspond to the nth positive sample.

[0086] The multiple negative samples may be audio feature representations and action feature representations that are not aligned on the time axis. For example, in the similarity matrix 410, all feature pairs other than the positive samples may be used as negative samples. For example, the audio feature representation T1 corresponding to the first audio segment and the action feature representation S2 corresponding to the second benchmark action sequence segment may constitute a negative sample.

[0087] In the pre-training stage, the contrastive learning method can be used to train the audio coding model 310 to ensure that it can learn the close correspondence between audio and action. The training goal of the audio coding model 310 can be expressed as follows:

[0088] L CL =0.5×(l speech (τ·C ges )+l gesture (τ·C ges )) (4)

[0089] τ can represent the parameter that needs to be adjusted. C ges The similarity matrix 410 can be represented as follows. speech It can represent the contrast loss of audio features calculated based on the similarity between audio feature representation and action feature representation, l gesture It can represent the contrast loss of action features calculated based on the similarity between audio feature representation and action feature representation. The purpose of contrastive learning is to make the audio feature representation and action feature representation (positive sample) that are the same (aligned) on the time axis have a higher value in the similarity matrix, while the audio feature representation and action feature representation (negative sample) that are not the same (unaligned) on the time axis have a lower similarity.

[0090] The pre-trained audio coding model 310 can more accurately capture the relationship between audio and gestures, so that the baseline gesture sequence can be generated more naturally and in sync with the audio content. At the same time, this training method based on contrastive learning enables the audio coding model 310 to have stronger generalization ability when facing diverse voice inputs, so that expressive gestures that match the voice can be generated in various scenarios.

[0091] In the process of generating the target video, the rendering model 330 is also involved. Still taking the training process performed by the electronic device 110 as an example, the training process of the rendering model 330 is introduced. The electronic device 110 selects a source image sample 511 and a target image sample 512 containing a second object sample from a second video sample containing a second object sample. Based on the source image sample, a visual feature representation of the second object sample is determined. Based on the second video sample, a gesture action sequence sample related to the second object sample is determined. Based on the visual feature representation, a given action sample in the gesture action sequence sample is rendered using the rendering model to obtain a predicted image of the second object sample, and the given action sample is extracted from the target image sample. Based on the difference between the predicted image and the target image sample, the parameters of the rendering model are adjusted.

[0092] Figure 5A A schematic diagram 500A of the training principle of the rendering model 330 according to some embodiments of the present disclosure is shown. First, taking the training of the rendering model 330 using a single source image sample as an example, the training process of the rendering model 330 is introduced.

[0093] The electronic device 110 selects a source image sample 511 and a target image sample 512 including a second object sample from a second video sample including a second object sample. The source image sample 511 is used to determine a visual feature representation of the second object sample.

[0094] In addition, the electronic device 110 may also determine a gesture action sequence sample related to the second object sample from the second video sample containing the second object sample. The given action sample 506 in the gesture action sequence sample is extracted from the target image sample 512 .

[0095] The target image sample 512 corresponds to the given action sample 506 in the gesture action sequence sample, that is, the given action sample 506 is extracted from the target image sample 512 .

[0096] The source image sample 511 is processed by the image encoder 366 to obtain a feature encoding representation of the source image sample 511. The given action sample 506 is processed by the posture encoder 368 to obtain a motion feature encoding representation of the given action sample 506. The feature encoding representation of the source image sample 511 and the motion feature encoding representation of the given action sample 506 are used as input information of the image decoder 369, and the image decoder 369 can obtain a predicted image 520. Based on the difference between the predicted image 520 and the target image sample 512, the parameters of the rendering model 330 can be adjusted.

[0097] The training objective of the rendering model 330 can be expressed as follows:

[0098]

[0099] i can represent the number of layers of the loss-aware model, V i It can represent the output feature map of the i-th layer of the loss-aware model. j can represent the number of downsampling. j It can be represented that the target image sample 512 is downsampled j times before being input into the loss-aware model. It can be indicated that the predicted image 520 is downsampled j times before being input into the loss-aware model. The loss-aware model is used to calculate the feature differences at different levels between the generated predicted image 520 and the target image sample 512. Based on the feature differences at different levels between the predicted image 520 and the target image sample 512, the electronic device 110 can continuously adjust the parameters of the rendering model 330 until the training cutoff condition is met (such as the feature difference meets the requirements, such as the training time reaches a given time, etc.).

[0100] It should be noted that, since the current embodiment uses a single source image sample to train the rendering model 330 as an example, Figure 5A The weight determination model 503 in the figure is represented by a dotted line (ie, the process of training the rendering model 330 for a single source image sample may not have the weight determination model 503), and does not involve a training process related to the weight determination model 503.

[0101] In order to improve the training effect of the rendering model 330, multiple source image samples 511 containing object samples can be used, and the presentation angles of the object samples in each source image sample 511 are different from each other. Figure 5A The weight determination model 364 in introduces the training process of the rendering model 330.

[0102] For a given source image sample in each source image sample 511, the difference determination model 360 processes the target posture action feature representation 504 and the source posture action feature representation 502 to obtain the action difference sample feature representation 562 corresponding to each source image. The target posture action feature representation 504 is determined based on the target image sample 512. The source posture action feature representation 502 is determined based on the source image sample 511. Based on the action difference sample feature representation 562 corresponding to each source image sample 511, the weight determination model 364 is used to determine the weight parameter corresponding to each source image sample 511.

[0103] Figure 5B FIG. 5 shows a schematic diagram 500B of a weight determination model 364 according to some embodiments of the present disclosure. Figure 5A and Figure 5BAs shown, after the weight determination model 364 performs a linear transformation 531 on the source posture action feature representation 502 and the target posture action feature representation 504, the linear transformation result is input into the cross attention calculation model 532 as a query (Q), a key (K), and a value (V) to determine the similarity between the source posture action feature representation 502 and the target posture action feature representation 504, thereby determining the weight of each source image sample 511 (pixel by pixel). The calculation result can be called a key point fusion feature representation 533. The above process fuses the feature information of the source posture action feature representation 502 and the target posture action feature representation 504. For example, the importance of each source image sample 511 in each area can be determined based on the distance and relative position between the corresponding key points. If the key point in a source image sample 511 is very close to the corresponding key point in the target image sample 512, it means that the posture matching degree of the source image sample in the area is high, and the weight of the corresponding pixel position should also be high.

[0104] On the other hand, the weight determination model 364 performs a linear transformation 531 on the action difference sample feature representation 562, and the result of the linear transformation and the key point fusion feature representation 533 are input into the cross attention calculation model 532 as the query (Q), key (K), and value (V) to obtain the visual-key point fusion feature 534. In this process, the action difference sample feature representation 562 interacts with the key point fusion feature representation 533 to further refine the weight parameters corresponding to each source image sample.

[0105] By combining the visual-keypoint fusion feature 534 with the target posture action feature representation 504 to complete the cross calculation 535, a cross attention map can be obtained, which can indicate the parameters of each source image sample 511. Based on each source image sample and the weight parameter corresponding to each source image sample, the visual feature representation of the object sample is determined.

[0106] In order to improve the generalization effect of model training, before determining the weight, each source image sample 511 may be subjected to masking processing. The masking processing of the source image sample is expressed as follows:

[0107] f i =w i (I i )⊙M i (6)

[0108] w i The i-th source image sample (I i ) and the feature representation of the action difference sample given the action sample. i Can represent an occlusion mask.

[0109] For each source image sample 511 after masking, an aggregation process may also be included. The aggregation process is represented as follows:

[0110] S=∑ i f i ⊙W i (7)

[0111] W i It can represent the weight of the i-th source image sample. Through aggregation, the visual feature representation of the (masked) object sample can be obtained.

[0112] By introducing the weight determination model 364, the rendering model 330 can better utilize the information of multiple source image samples during the training process. Compared with the situation where only a single source image can be relied on when the weight determination model 364 is not introduced, after the weight determination model 364 is introduced, the rendering model 330 can perform pixel-by-pixel weighted fusion of posture and action features in different source image samples, thereby selecting the most appropriate features according to the needs of action differences. This pixel-by-pixel weight determination method ensures that the rendering model 330 can comprehensively extract effective visual information from multiple angles and postures when generating predicted images, improves the performance of the generated images in terms of details and overall consistency, and ultimately significantly improves the training effect of the rendering model 330 and the quality and naturalness of the generated images.

[0113] In some embodiments of the present disclosure, the reference gesture action sequence corresponding to the target audio is an action sequence including hand gestures. Compared with whole-body gestures, hand gestures usually express the relevance to the voice content more directly, especially in the communication process, gestures can more effectively convey specific information and emotions. Therefore, the hand-based action sequence can be better synchronized with the voice, enhancing the clarity and expressiveness of communication. In addition, the rendering process of hand gestures is less computationally intensive than that of whole-body gestures, which helps to improve the efficiency of video generation.

[0114] Figure 6 A schematic structural block diagram of an apparatus 600 for video generation according to some embodiments of the present disclosure is shown. The apparatus 600 may be implemented in or included in the electronic device 110, for example. Each module / component in the apparatus 600 may be implemented by hardware, software, firmware, or any combination thereof.

[0115] like Figure 6As shown, the device 600 may include a three-dimensional posture representation determination module 601, which is configured to determine the three-dimensional posture representation of the target object based on the source image of the target object. A reference posture action sequence determination module 602 is configured to determine a reference posture action sequence corresponding to the target audio based on the target audio. A target posture action sequence determination module 603 is configured to adjust the reference posture action sequence based on the three-dimensional posture representation to obtain a target posture action sequence corresponding to the target object. A video generation module 604 is configured to generate a target video of the target object based on the source image of the target object and the target posture action sequence, and the target video represents that the target object performs a posture action corresponding to the target posture action sequence while speaking the target audio.

[0116] In some embodiments of the present disclosure, the reference gesture action sequence determination module 602 may be configured to: determine a feature representation of the target audio and determine a reference gesture action sequence corresponding to the target audio using an action generation model based on the feature representation of the target audio.

[0117] In some embodiments of the present disclosure, the target posture action sequence determination module 603 may be configured to: determine the three-dimensional space coordinates corresponding to each key point in the three-dimensional posture representation of the target object. Based on the three-dimensional space coordinates corresponding to each key point, determine the angle representation and length representation between each key point. Utilize the angle representation and length representation between each key point to adjust each reference posture action included in the reference posture action sequence.

[0118] In some embodiments of the present disclosure, the video generation module 604 may be configured to: determine a visual feature representation of the target object based on a source image of the target object, and render each gesture action in a target gesture action sequence based on the visual feature representation of the target object to obtain a target video of the target object.

[0119] In some embodiments of the present disclosure, the source image of the target object includes multiple source images, and the presentation angles of the target object in each source image are different from each other. In this case, the video generation module 604 can be configured to: for a given gesture action in the target gesture action sequence, based on the action difference and the given gesture action, determine the weight parameters corresponding to each of the multiple source images, and the action difference indicates the difference between the given gesture action and the source gesture actions corresponding to the multiple source images. Based on the multiple source images and the weight parameters corresponding to each of the multiple source images, determine the visual feature representation of the target object. Render the given gesture action based on the visual feature representation of the target object to obtain a video clip corresponding to the given gesture action, and the video clip is a part of the target video of the target object.

[0120] In some embodiments of the present disclosure, the video generation module 604 may be specifically configured to: extract the action difference feature representation corresponding to each source image from the given gesture action and the source gesture action corresponding to the plurality of source images, the action difference feature representation indicating the action difference. Based on the action difference feature representation corresponding to each source image, determine the weight parameter corresponding to each source image.

[0121] In some embodiments of the present disclosure, a baseline posture action sequence is determined using an action generation model, which includes an audio coding model and an action generation model, wherein the audio coding model is trained by pre-training, and the audio coding model and the action generation model are jointly trained by fine-tuning.

[0122] In some embodiments of the present disclosure, a model training module is also included. The model training module can be configured to: extract an audio sample and a benchmark action sequence sample from a first video sample containing a first object sample. Construct a plurality of positive samples, each positive sample comprising a first audio segment and a first benchmark action sequence segment, the first audio segment is obtained by segmenting the audio sample, the first benchmark action sequence segment is obtained by segmenting the benchmark action sequence sample, and the first audio segment and the first benchmark action sequence segment are aligned in time. Construct a plurality of negative samples, each negative sample comprising a second audio segment and a second benchmark action sequence segment, the second audio segment is obtained by segmenting the audio sample, the second benchmark action sequence segment is obtained by segmenting the benchmark action sequence sample, and the second audio segment and the second benchmark action sequence segment are not aligned in time. Utilize a plurality of positive samples and a plurality of negative samples to pre-train an audio coding model through comparative learning.

[0123] In some embodiments of the present disclosure, a target video of a target object is generated by a rendering model, and the model training module can be configured to: select a source image sample and a target image sample containing a second object sample from a second video sample containing a second object sample. Based on the source image sample, a visual feature representation of the second object sample is determined. Based on the second video sample, a gesture action sequence sample related to the second object sample is determined. Based on the visual feature representation, a given action sample in the gesture action sequence sample is rendered using the rendering model to obtain a predicted image of the second object sample, wherein the given action sample is extracted from the target image sample. Based on the difference between the predicted image and the target image sample, the parameters of the rendering model are adjusted.

[0124] In some embodiments of the present disclosure, the source image sample includes multiple source image samples, the presentation angles of the object samples in each source image sample are different from each other, and the model training module can be configured to: determine the weight parameters corresponding to each of the multiple source image samples based on the action difference sample and the given action sample, the action difference is determined based on the given action sample and the source gesture action corresponding to each of the multiple source image samples. Based on each source image sample and the weight parameter corresponding to each source image sample, determine the visual feature representation of the object sample.

[0125] In some embodiments of the present disclosure, the reference gesture action sequence corresponding to the target audio is an action sequence including hand gestures.

[0126] Figure 7 1 shows a block diagram of an electronic device 700 in which one or more embodiments of the present disclosure may be implemented. It should be understood that Figure 7 The electronic device 700 shown is merely exemplary and should not constitute any limitation on the functionality and scope of the embodiments described herein. Figure 7 The electronic device 700 shown may include or be implemented as Figure 1 Electronic device 110 or Figure 6 Device 600.

[0127] like Figure 7 As shown, the electronic device 700 is in the form of a general electronic device. The components of the electronic device 700 may include, but are not limited to, one or more processors or processing units 710, a memory 720, a storage device 730, one or more communication units 740, one or more input devices 750, and one or more output devices 760. The processing unit 710 may be an actual or virtual processor and is capable of performing various processes according to a program stored in the memory 720. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to improve the parallel processing capability of the electronic device 700.

[0128] The electronic device 700 typically includes a plurality of computer storage media. Such media may be any accessible media that is accessible to the electronic device 700, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 720 may be a volatile memory (e.g., registers, caches, random access memory (RAM)), a non-volatile memory (e.g., a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 730 may be a removable or non-removable medium, and may include a machine-readable medium, such as a flash drive, a disk, or any other medium, which may be capable of being used to store information and / or data and may be accessed within the electronic device 700.

[0129] The electronic device 700 may further include additional removable / non-removable, volatile / non-volatile storage media. Figure 7 As shown in , a disk drive for reading or writing from a removable, non-volatile disk (e.g., a "floppy disk") and an optical drive for reading or writing from a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to the bus (not shown) by one or more data media interfaces. The memory 720 may include a computer program product 725 having one or more program modules that are configured to perform various methods or actions of various embodiments of the present disclosure.

[0130] The communication unit 740 implements communication with other electronic devices through a communication medium. Additionally, the functions of the components of the electronic device 700 can be implemented with a single computing cluster or multiple computing machines that can communicate through a communication connection. Therefore, the electronic device 700 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.

[0131] The input device 750 may be one or more input devices, such as a mouse, a keyboard, a tracking ball, etc. The output device 760 may be one or more output devices, such as a display, a speaker, a printer, etc. The electronic device 700 may also communicate with one or more external devices (not shown) through the communication unit 740 as needed, such as a storage device, a display device, etc., communicate with one or more devices that allow a user to interact with the electronic device 700, or communicate with any device that allows the electronic device 700 to communicate with one or more other electronic devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0132] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above.

[0133] According to an exemplary implementation of the present disclosure, a computer program product or a computer program is provided, the computer program product or the computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes Figure 2 The methods provided in the various optional methods are therefore not described in detail here.

[0134] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices, equipment, and computer program products implemented according to the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0135] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0136] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable data processing apparatus, or other device implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0137] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple implementations of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of a module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some implementations as replacements, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0138] The above descriptions of various implementations of the present disclosure are exemplary, non-exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described implementations. The selection of terms used herein is intended to best explain the principles of the implementations, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the various implementations disclosed herein.

Claims

1. A method for video generation, comprising: Determining a three-dimensional pose representation of the target object based on a source image of the target object; Based on the target audio, determining a reference gesture action sequence corresponding to the target audio; adjusting the reference posture action sequence based on the three-dimensional posture representation to obtain a target posture action sequence corresponding to the target object; as well as A target video of the target object is generated based on the source image of the target object and the target gesture action sequence, wherein the target video indicates that the target object performs a gesture action corresponding to the target gesture action sequence while speaking the target audio.

2. The method according to claim 1, wherein determining a target gesture action sequence corresponding to the target audio comprises: Determining a feature representation of the target audio; as well as Based on the feature representation of the target audio, a reference posture action sequence corresponding to the target audio is determined using an action generation model.

3. The method of claim 1 , wherein adjusting the reference pose action sequence based on the three-dimensional pose representation comprises: Determine the three-dimensional space coordinates corresponding to each key point in the three-dimensional posture representation of the target object; Determine the angle representation and the length representation between the key points based on the three-dimensional space coordinates corresponding to the key points; as well as The reference posture actions included in the reference posture action sequence are adjusted by using the angle representation and the length representation between the key points.

4. The method according to claim 1, wherein generating the target video of the target object comprises: Determining a visual feature representation of the target object based on a source image of the target object; as well as Based on the visual feature representation of the target object, each gesture action in the target gesture action sequence is rendered to obtain a target video of the target object.

5. The method according to claim 1, wherein the source image of the target object comprises a plurality of source images, the presentation angles of the target object in each source image are different from each other, and generating the target video of the target object comprises: For a given posture action in the target posture action sequence, Determining weight parameters corresponding to each of the plurality of source images based on an action difference and the given gesture action, wherein the action difference indicates a difference between the given gesture action and each source gesture action corresponding to the plurality of source images; Determining a visual feature representation of the target object based on the multiple source images and weight parameters corresponding to each of the multiple source images; as well as The given gesture action is rendered based on the visual feature representation of the target object to obtain a video clip corresponding to the given gesture action, where the video clip is a part of the target video of the target object.

6. The method according to claim 5, wherein determining the weight parameters corresponding to each of the plurality of source images comprises: Extracting, from the given gesture action and the source gesture actions corresponding to the plurality of source images, a motion difference feature representation corresponding to each source image, wherein the motion difference feature representation indicates the motion difference; and Based on the action difference feature representation corresponding to each source image, a weight parameter corresponding to each source image is determined.

7. The method according to claim 1, wherein the reference posture action sequence is determined using an action generation model, the action generation model comprising an audio coding model and an action generation model, The audio coding model is trained by pre-training, and the audio coding model and the motion generation model are jointly trained by fine-tuning.

8. The method of claim 7, wherein the audio coding model is pre-trained by: extracting an audio sample and a reference motion sequence sample from a first video sample including a first object sample; Constructing a plurality of positive samples, each positive sample comprising a first audio segment and a first benchmark action sequence segment, wherein the first audio segment is obtained by segmenting the audio sample, the first benchmark action sequence segment is obtained by segmenting the benchmark action sequence sample, and the first audio segment and the first benchmark action sequence segment are aligned in time; Constructing multiple negative samples, each negative sample comprising a second audio segment and a second benchmark action sequence segment, the second audio segment is obtained by segmenting the audio sample, the second benchmark action sequence segment is obtained by segmenting the benchmark action sequence sample, and the second audio segment and the second benchmark action sequence segment are not aligned in time; as well as The audio coding model is pre-trained by contrastive learning using the multiple positive samples and the multiple negative samples.

9. The method according to claim 1, wherein the target video of the target object is generated by a rendering model, and the rendering model is trained by: Selecting a source image sample and a target image sample containing a second object sample from a second video sample containing the second object sample; determining a visual feature representation of the second object sample based on the source image sample; Based on the second video sample, determining a gesture action sequence sample related to the second object sample; Based on the visual feature representation, using the rendering model to render a given action sample in the pose action sequence sample to obtain a predicted image of the second object sample, wherein the given action sample is extracted from the target image sample; as well as Based on the difference between the predicted image and the target image sample, the parameters of the rendering model are adjusted.

10. The method according to claim 9, wherein the source image sample comprises a plurality of source image samples, the presentation angles of the object samples in the respective source image samples are different from each other, and determining the visual feature representation of the object sample comprises: Determine, based on an action difference sample and the given action sample, a weight parameter corresponding to each of the plurality of source image samples, wherein the action difference is determined based on the source gesture actions corresponding to each of the given action sample and the plurality of source image samples; as well as Based on the source image samples and the weight parameters corresponding to the source image samples, a visual feature representation of the object sample is determined. The method according to claim 1 , wherein the reference gesture action sequence corresponding to the target audio is an action sequence including hand gestures.

12. A device for video generation, comprising: a three-dimensional posture representation determination module, configured to determine a three-dimensional posture representation of a target object based on a source image of the target object; A reference gesture action sequence determination module is configured to determine a reference gesture action sequence corresponding to the target audio based on the target audio; a target posture action sequence determination module, configured to adjust the reference posture action sequence based on the three-dimensional posture representation to obtain a target posture action sequence corresponding to the target object; as well as The video generation module is configured to generate a target video of the target object based on a source image of the target object and the target gesture action sequence, wherein the target video represents that the target object performs a gesture action corresponding to the target gesture action sequence while speaking the target audio.

13. An electronic device comprising: at least one processing unit; as well as At least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 11 when executed by the at least one processing unit.

14. A computer-readable storage medium having a computer program stored thereon, wherein the computer program can be executed by a processor to implement the method according to any one of claims 1 to 11.

15. A computer program product, comprising computer executable instructions, wherein when the computer executable instructions are executed by a processor, the method according to any one of claims 1 to 11 is implemented.

Images