Image animation generation method and device, electronic equipment and storage medium

Abstract

The invention provides an image animation generation method and device, electronic equipment and a storage medium, and the method comprises the steps: carrying out the feature extraction of a reference image and a driving video based on an encoder of a generative model, obtaining image features and video features, and enabling the image features to comprise three-dimensional texture features; an optical flow transformation module based on the generative model processes the image features and the video features to generate a motion optical flow; based on a decoder of the generative model, decoding the three-dimensional texture features and the moving optical flow to obtain a plurality of target images; and synthesizing the plurality of target images into an image animation based on a generation module of the generative model. Through the combination of the three-dimensional texture features and the motion optical flow, the motion change of the three-dimensional space can be accurately captured, and the stretching distortion problem in the image animation generation is obviously reduced, so that the natural sense and the reality sense of the animation are improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to an image animation generation method, device, electronic equipment and storage medium. Background Art

[0002] Video-driven image animation generation technology is a cutting-edge breakthrough in the field of computer vision and artificial intelligence. It uses deep learning, image synthesis and motion capture technology to enable static photos or images to produce dynamic effects through video drive. This technology has a wide range of applications in entertainment, social media, virtual reality and other fields, meeting users' growing demand for personalized and dynamic content.

[0003] At present, the FOMM (First Order Motion Model for Image Animation) model is widely used in the field of image animation technology. This model is a technology used to apply motion information in videos to static images. Although the FOMM model has achieved certain results in the field of image animation generation, it still has some limitations. The image animation generated by the FOMM model lacks realism in some cases, especially when dealing with large changes in facial expressions or complex limb movements. The generated image animation will be stretched or deformed locally, resulting in distortion of the details of the animation generation and lack of realism. Summary of the invention

[0004] The present invention provides an image animation generation method, device, electronic device and storage medium, which are used to solve the stretching distortion problem of image animation generated by video drive in the prior art, thereby significantly improving the naturalness and detail fidelity of image animation generation.

[0005] The present invention provides an image animation generation method, comprising: Based on the encoder of the generative model, feature extraction is performed on the reference image and the driving video respectively to obtain image features and video features, wherein the image features include three-dimensional texture features; Based on the optical flow transformation module of the generation model, the image features and the video features are processed to generate motion optical flow; Based on the decoder of the generative model, the three-dimensional texture features and the motion optical flow are decoded to obtain a plurality of target images; Based on the generation module of the generation model, multiple target images are synthesized into an image animation.

[0006] According to an image animation generation method provided by the present invention, the encoder based on the generative model extracts features from the reference image and the driving video respectively to obtain image features and video features, including: Based on the image encoding unit of the encoder, feature extraction is performed on the reference image to obtain the image features, wherein the image features also include posture features and expression features; Based on the video encoding unit of the encoder, feature extraction is performed on the driving video to obtain the video features, which include posture features and expression features.

[0007] According to an image animation generation method provided by the present invention, the image features and the video features are processed to generate motion optical flow, including: Converting the posture feature in the image feature into a first posture optical flow, and converting the expression feature in the image feature into a first expression optical flow; Converting the posture feature in the video feature into a second posture optical flow, and converting the expression feature in the video feature into a second expression optical flow; The first posture optical flow, the first expression optical flow, the second posture optical flow and the second expression optical flow are fused to generate the motion optical flow.

[0008] According to an image animation generation method provided by the present invention, the image features further include identity features, and the step of converting the expression features in the image features into a first expression optical flow includes: The identity feature and the expression feature in the image feature are spliced ​​together, and the spliced ​​features are converted into the first expression optical flow.

[0009] According to an image animation generation method provided by the present invention, the decoder based on the generation model decodes the three-dimensional texture features and the motion optical flow to obtain multiple target images, including: Based on the transformation module of the decoder, the motion optical flow is applied to transform the three-dimensional texture features to obtain a plurality of transformed texture features; Based on the three-dimensional module of the decoder, each transformed texture feature is converted into a three-dimensional spatial feature; Based on the two-dimensional module of the decoder, the three-dimensional spatial features are two-dimensionally projected to obtain the target image.

[0010] According to an image animation generation method provided by the present invention, the encoder learns scaling parameters and stretching transformations during training.

[0011] According to an image animation generation method provided by the present invention, the identity characteristics of the image animation are consistent with the identity characteristics of the reference image, and the posture characteristics and expression characteristics of the image animation are consistent with the posture characteristics and expression characteristics of the driving video.

[0012] The present invention also provides an image animation generating device, comprising: An encoding unit, configured to extract features from a reference image and a driving video based on an encoder of a generative model, respectively, to obtain image features and video features, wherein the image features include three-dimensional texture features; A transformation unit, configured to process the image features and the video features based on the optical flow transformation module of the generation model to generate motion optical flow; A decoding unit, configured to decode the three-dimensional texture features and the motion optical flow based on a decoder of the generation model to obtain a plurality of target images; A generation unit is used to synthesize the multiple target images into an image animation based on the generation module of the generation model.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, any of the above-mentioned image animation generation methods is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the image animation generation method described in any one of the above is implemented.

[0015] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the image animation generating method described above is implemented.

[0016] The image animation generation method, device, electronic device and storage medium provided by the present invention can efficiently capture the three-dimensional texture features of the image and the motion features of the video by extracting features of the reference image and the driving video through the encoder of the generation model. This multi-dimensional feature provides a rich information basis for subsequent animation generation. The optical flow transformation module in the generation model is used in combination with image features and video features to accurately generate motion optical flow. The three-dimensional texture features and motion optical flow are combined through a decoder to generate a high-quality target image, thereby flexibly generating image animation based on the target image. The present invention combines three-dimensional texture features with motion optical flow, so that the model can accurately capture the movement changes of objects in three-dimensional space and restore the true form and motion state of the object, thereby significantly reducing the stretching distortion problem in image animation generation and enhancing the naturalness and realism of image animation. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 This is one of the flow charts of the image animation generation method provided by the present invention; Figure 2 It is a schematic diagram of the framework of the image encoding unit provided by the present invention; Figure 3 It is a schematic diagram of the framework of the optical flow transformation module provided by the present invention; Figure 4 It is a schematic diagram of the framework of the decoder provided by the present invention; Figure 5 This is the second flow chart of the image animation generation method provided by the present invention; Figure 6 It is a structural schematic diagram of the image animation generating device provided by the present invention; Figure 7 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] With the breakthrough of deep learning, especially generative models (such as generative adversarial networks GANs, autoencoders, etc.), image synthesis technology has developed rapidly. At the same time, the demand for digital media and virtual content is growing, especially in the fields of entertainment, social media, virtual reality, etc., personalized and dynamic content is becoming more and more important. Users are no longer satisfied with static photos or images, but hope to experience dynamic effects in daily interactions. Therefore, image animation technology came into being.

[0021] Image animation technology is a cutting-edge technology in the field of computer vision and artificial intelligence. It allows static photos or images to be driven by videos to produce dynamic effects. This technology combines deep learning, image synthesis and motion capture technology to create lifelike animations or effects, and is usually used for dynamic simulation of facial expressions, postures or body movements. Specifically, image animation technology has the ability to synthesize and transform images. This technology extracts motion information (such as expressions, head movements, etc.) from videos and applies it to static images, so that the characters or objects in the image can "move". This technology has a wide range of applications in film entertainment, virtual reality, game development, digital identity and other fields.

[0022] At present, the most widely used image animation technology is the FOMM model, which captures motion information in the video and maps it to a static image. Its technical solution includes the following key parts: 1) Motion representation: The core idea of ​​the FOMM model is to separate motion information into two parts. The first is key points: used to represent important geometric features in the image (such as face, limbs, etc.). These key points change over time in the target video and are used to control the changes in the static image. The second is local affine transformations: used to capture local motion associated with key points. It represents deformations in these local areas, such as changes in facial expressions, twisting of limbs, etc. At this stage, the model extracts key points in the target video and captures motion information in the video through motion estimation, which is then applied to the corresponding positions of the static image.

[0023] 2) Driving model based on first-order motion estimation: Different from traditional animation based on skeleton or complex mesh models, this model approximates the movement of the object surface by calculating key points and local affine transformations. It mainly includes key point detector (Keypoint Detector) and motion estimator (Motion Estimator). Among them, the key point detector is responsible for detecting key points in the source image and driving video frames. These key points are used to represent meaningful geometric features in the image. The detector is trained by a convolutional neural network, which can automatically assign a set of key points to different areas in the image. The motion estimator captures how the surface of the object moves by estimating the key point position information in the driving video, which can be used to drive the static image so that its surface changes according to the motion pattern in the video.

[0024] 3) Image Generator: The task of the image generator is to generate new animated images based on the source image, key point information in the driving video, and local affine transformation. This part is usually built based on generative adversarial networks (GANs) or other generative models.

[0025] However, although the FOMM model has achieved certain results in the field of video-driven image animation, it also has some shortcomings and limitations: First, the generated image animation has problems with stretching or deformation, and the driving effect generated by FOMM lacks realism in some cases, especially when dealing with large changes in facial expressions or complex body movements, the image may appear unnaturally distorted or blurred. Moreover, during the video driving process, the details of the generated image are not processed finely enough, resulting in distortion of local features (such as eyes, lips, etc.) when facial expressions change.

[0026] Secondly, FOMM relies on key point detection and uses these key points to drive image animation. If the key point detection is not accurate enough, or the motion in the driven video exceeds the range that the key points can express, the generated animation effect may appear incoherent or inconsistent. Therefore, FOMM is more sensitive to the face area in the image. In practical applications, it is necessary to accurately crop the image to the appropriate area, otherwise the final driving effect will be poor due to key point detection errors.

[0027] In addition, there is no specific mechanism in FOMM to separate facial expressions from other movements. It uses a unified motion estimation model and cannot handle facial expressions, head poses, or other non-rigid movements separately, which results in the generated animation lacking flexibility and control in details when the expressions in the driving video do not match the expression requirements in the source image.

[0028] In this regard, the present invention provides an image animation generation method and proposes the following improvement strategies to overcome the above-mentioned defects: 1) Regarding the stretching or deformation problem: In video-driven image animation, especially when models such as FOMM are processing large-scale motion, the generated image animation is prone to local stretching or deformation, especially in non-rigid motion (such as facial expressions or body movements). The stretching feeling causes the details of the animation generation to be distorted and lack realism. By introducing three-dimensional texture features and motion optical flow, the present invention can more accurately restore the motion changes in three-dimensional space, reduce the stretching distortion problem in image animation generation, and significantly improve the naturalness and detail fidelity of the animation.

[0029] 2) For regional sensitivity issues: When dealing with large-scale motion or complex scenes, the FOMM model tends to be overly sensitive to local features (such as facial details or body movements), resulting in uncoordinated or distorted motion in different regions of the generated image animation. The present invention enables the model to learn potential motion features that are insensitive to regions by learning scaling parameters and stretching transformations, thereby enhancing the robustness to regional motion. This allows the model to maintain the coherence and consistency of each region when dealing with complex movements or large-scale motion, thereby improving the robustness of the image animation generation effect.

[0030] 3) Regarding the problem of expression modification: FOMM lacks an independent expression control mechanism and cannot effectively modify facial expressions. The generated expression changes only rely on the motion information in the driving video, which limits the flexibility of expression generation. The present invention introduces an expression feature mechanism to give the model the ability to modify and generate complex expressions, so that expressions can be adjusted individually without affecting the overall movement or posture. This improvement enables the model to be more flexible and delicate when generating a variety of expression animations, and adapt to different application scenarios. The technical solution provided by the present invention will be introduced in detail below.

[0031] Figure 1 It is one of the flow charts of the image animation generation method provided by the present invention, such as Figure 1 As shown, the method includes: Step 110, based on the encoder of the generative model, feature extraction is performed on the reference image and the driving video respectively to obtain image features and video features, wherein the image features include three-dimensional texture features; It should be noted that the generative model is a machine learning or deep learning model used to generate image animations. It includes four main parts: encoder, optical flow transformation module, decoder and generation module. It can process the input reference image and driving video and output the final image animation. It should be understood that the input of the generative model is a reference image and a driving video, and the output is an image animation, that is, the video corresponding to the reference image. Among them, the identity of the target (object or face, etc.) in the image animation is consistent with the reference image, and the posture and expression of the target in the image animation are consistent with those in the driving video.

[0032] Specifically, the reference image can also be called the source image or the original image, which is the basic image for generating the image animation. The reference image can be a static image, which provides static appearance information for the animation, including the basic shape and texture of the animated character or object. The driving video is a video used to provide motion information for the reference image, which can be a moving video sequence. The motion, action, posture or expression in the driving video will be extracted and applied to the reference image to generate the image animation.

[0033] After obtaining the reference image and the driving video, the reference image and the driving video can be input into the encoder of the generative model to extract features of the reference image and the driving video, respectively, to obtain the corresponding image features and video features. Here, the encoder is an important component of the generative model, which is responsible for converting the input data (such as the reference image and the driving video) into a more abstract and compact representation, namely the feature representation.

[0034] The encoder can be composed of a basic backbone network and multiple multi-head fully connected layers, where the backbone network can be composed of a convolutional neural network (CNN). Through layer-by-layer convolution, pooling and other operations, the input data is mapped to a latent space, that is, multiple latent features are extracted from the input image or video to generate feature vectors or feature maps. After the backbone network extracts the features, these features are processed in parallel through multiple fully connected layers to output different types of latent features, such as posture features, expression features, identity features, texture features, etc.

[0035] It can be understood that image features refer to key information extracted from the reference image, which may include posture features, expression features, identity features and texture features in the reference image. Among them, the pose feature (Pose latent) represents the posture of the object or face in the image (such as head orientation); the expression feature (Expression latent) is used to represent the changing characteristics of facial expressions in the image (such as smiling, frowning, etc.); the identity feature (ID latent) represents the captured information related to the individual identity, so that the generated image animation can retain the identity characteristics of the source image (i.e., the reference image); the texture feature (Volume latent) refers to the potential representation used for three-dimensional information processing in subsequent convolutional layers and decoders to help the model capture spatial structure. The texture feature here can be a three-dimensional texture feature, which describes the texture information of the surface of an object in three-dimensional space, and plays an important role in the subsequent generation of realistic image animations. Video features refer to key information extracted from the driving video, which may include posture features and expression features in the video.

[0036] Step 120, based on the optical flow transformation module of the generation model, the image features and the video features are processed to generate motion optical flow; Specifically, after the image features and video features are extracted, the image features and video features can be input into the optical flow transformation module of the model, which can apply the potential features such as posture, expression and identity in the image features and video features to generate motion optical flow for subsequent transformation of texture features. Here, the optical flow transformation module is an important component of the generation model, which is responsible for processing image features and video features and generating motion optical flow. Motion optical flow refers to a vector field that describes the motion trend of pixels in an image. It is used to capture and represent the posture and expression changes of objects or faces in the image, and is an important basis for subsequent transformation of texture features. In image animation generation, motion optical flow is a very important source of information. By using motion optical flow, the motion state of the object can be accurately captured and applied to the reference image to generate realistic image animation.

[0037] Specifically, the optical flow transformation module converts the posture features and expression features output by the encoder into corresponding posture optical flows (Pose Flow) and expression optical flows (Expression Flow). These optical flows describe the movement trend of the face, among which the posture optical flow reflects the global movement trend, such as the rotation or tilt of the head, and it describes the overall position change of the object or face in the image. The expression optical flow captures the subtle changes in the local area, such as the upward movement of the corners of the mouth or the opening and closing of the eyes, and it describes the subtle changes in the facial expressions of the face in the image. It should be understood that the above-mentioned posture features include posture features extracted from the image and posture features extracted from the video, and the expression features include expression features extracted from the image and expression features extracted from the video. The conversion of posture features and expression features into posture optical flows and expression optical flows can be achieved through a series of calculations and transformations, such as convolutional neural networks or other related algorithms in deep learning.

[0038] Next, the pose optical flow and expression optical flow can be fused to generate a complete motion optical flow, which describes the overall motion changes in the image. This step combines global and local motion to provide a basis for subsequent texture feature transformation. By fusing the pose optical flow and expression optical flow, a complete motion description that includes both global motion and local changes can be obtained.

[0039] Step 130, decoding the three-dimensional texture features and the motion optical flow based on the decoder of the generation model to obtain a plurality of target images; Specifically, after the motion optical flow is fused, the motion optical flow and three-dimensional texture features can be input into the decoder of the generative model. The decoder will gradually decode the input motion optical flow and three-dimensional texture features to generate multiple target images. Here, the decoder is a key component in the generative model, which is responsible for converting the feature representation (such as three-dimensional texture features and motion optical flow) back to the original data space to generate the corresponding target image. The decoder can be a multi-layer neural network, which gradually restores the feature representation to more specific image content through layer-by-layer deconvolution, upsampling and other operations.

[0040] Specifically, the decoder first transforms the three-dimensional texture features generated by the encoder and the motion optical flow generated by the optical flow transformation module, that is, by applying the motion optical flow to the three-dimensional texture features, the model can adjust the shape or position of the three-dimensional texture features according to the generated motion optical flow. This process is similar to stretching, rotating or translating an object in three-dimensional space according to motion changes. After transforming the three-dimensional texture features by motion optical flow, the decoder can process the three-dimensional information in the transformed texture features, further capture the spatial structural features through convolution operations, and convert them into a three-dimensional representation with high details. Next, the decoder can perform a two-dimensional projection of the three-dimensional representation and further optimize the texture details of the image through a convolutional neural network. This stage converts the information of the three-dimensional space into two-dimensional image information, so that the output image maintains a high degree of delicacy and visual consistency on a two-dimensional level, that is, the target image is generated.

[0041] It should be noted that the three-dimensional texture feature is a deep description of the surface details and structure of the object in the reference image. It not only contains two-dimensional information such as the color and brightness of the object, but also contains deep information such as the shape and texture direction of the object in three-dimensional space. In image animation generation, the three-dimensional texture feature can help the model better understand the real shape and surface structure of the object, so as to maintain the coherence and consistency of the object shape during the animation generation process. Motion optical flow is a vector field that describes the direction and speed of the movement of pixels in an image. Through the optical flow transformation module, the model can generate motion optical flow based on video features to accurately simulate the motion state of the object in three-dimensional space. Motion optical flow not only provides dynamic information about the movement of the object, but also helps the model maintain the continuity and smoothness of the object's movement during the animation generation process. When the three-dimensional texture feature is combined with the motion optical flow, the model can capture the static shape and dynamic motion information of the object at the same time. This combination enables the model to more accurately simulate the motion state of the object in three-dimensional space when generating image animation, including the rotation, translation, scaling and other forms of movement of the object.

[0042] During the image animation generation process, the model dynamically adjusts the 3D texture features according to the motion trajectory provided by the motion optical flow. This adjustment ensures that the shape and motion state of the object in the animation are consistent with the object in the driving video. At the same time, because the 3D texture features provide depth information on the surface of the object, the model can more accurately simulate the deformation and stretching of the surface of the object when generating animation, thereby significantly reducing the stretching distortion problem.

[0043] It is understandable that stretching distortion usually occurs when the shape of an object changes dramatically, such as the rapid rotation or scaling of an object in an animation. When the model relies only on two-dimensional information (such as color, brightness, etc.) to generate animation, it is difficult to accurately simulate the motion state of the object in three-dimensional space, resulting in the emergence of stretching distortion problems. In an embodiment of the present invention, the model generates image animation by combining three-dimensional texture features with motion optical flow, which can more accurately simulate the motion state of the object in three-dimensional space. Since the three-dimensional texture features provide depth information on the surface of the object, the model can more accurately calculate the deformation of the surface of the object when generating the animation. At the same time, the motion optical flow provides dynamic information about the movement of the object, so that the model can more accurately simulate the motion trajectory of the object in the animation. This combination enables the model to more accurately restore the true shape and motion state of the object when generating the animation, thereby significantly reducing the stretching distortion problem.

[0044] Step 140: Based on the generation module of the generation model, synthesize the plurality of target images into an image animation.

[0045] Specifically, after the decoding process, the decoder generates a series of continuous target images, which form the basis of the image animation. In order to generate smooth animation effects, the generation module ensures that the content changes between adjacent images are smooth and continuous, and synthesizes the generated target images into a complete image animation, such as a video. Each frame in the image animation retains the identity characteristics of the reference image, but is updated according to the posture and expression in the driving video.

[0046] The method provided by the embodiment of the present invention can efficiently capture the three-dimensional texture features of the image and the motion features of the video by extracting features of the reference image and the driving video through the encoder of the generative model. This multi-dimensional feature provides a rich information basis for subsequent animation generation. By utilizing the optical flow transformation module in the generative model, combined with image features and video features, the motion optical flow can be accurately generated. By combining the three-dimensional texture features and the motion optical flow through the decoder, a high-quality target image can be generated, thereby flexibly generating image animation based on the target image. The present invention combines three-dimensional texture features with motion optical flow, so that the model can accurately capture the movement changes of objects in three-dimensional space and restore the true form and motion state of the object, thereby significantly reducing the stretching distortion problem in image animation generation and enhancing the naturalness and realism of the image animation.

[0047] Based on the above embodiment, step 110 specifically includes: Step 111, based on the image encoding unit of the encoder, extracting features from the reference image to obtain the image features, wherein the image features also include posture features and expression features; Step 112: extracting features from the driving video based on the video encoding unit of the encoder to obtain the video features, where the video features include posture features and expression features.

[0048] Specifically, the encoder may include an image encoding unit and a video encoding unit, wherein the image encoding unit is a component of the generative model encoder, which is specifically used to extract features from the reference image, including posture features, expression features, and three-dimensional texture features, etc. These features together constitute image features, providing a basis for the subsequent generation of motion optical flow and target images. The video encoding unit is also a part of the generative model encoder, but it focuses on extracting features from the driving video. The extracted features mainly include posture features and expression features, which constitute video features and provide key information for generating motion optical flow.

[0049] It can be understood that the expression features capture the changing characteristics of facial expressions, such as smiling, frowning, etc. The embodiment of the present invention introduces expression features so that the model can accurately control and modify the expression alone when generating image animation without affecting the movement or posture of other parts.

[0050] Figure 2 Schematic diagram of the framework of the image encoding unit provided by the present invention, such as Figure 2As shown, the main function of the image coding unit is to extract a variety of potential features from the input reference image for subsequent motion generation and image synthesis. The image coding unit can be composed of a backbone network and multiple multi-head fully connected layers, wherein the backbone network is composed of a convolutional neural network, which is used to extract the basic features of the input image. This part of the network will learn the overall structural information of the image, such as shape, texture, etc.; after the backbone network extracts the features, multiple fully connected layers will process these features in parallel and output different types of potential features, including posture features, expression features, identity features, and three-dimensional texture features. It should be understood that in the image coding unit, a three-dimensional convolution technology can be introduced, through which the three-dimensional texture features in the reference image can be effectively extracted, which can provide richer information for the decoding process, thereby generating more realistic and natural image animations.

[0051] The main function of the video encoding unit is to extract a variety of potential features from the input driving video, including posture features, expression features, etc. The video encoding unit can also be composed of a backbone network and multiple multi-head fully connected layers.

[0052] Based on any of the above embodiments, step 120 specifically includes: Step 121, converting the posture feature in the image feature into a first posture optical flow, and converting the expression feature in the image feature into a first expression optical flow; It should be noted that the posture features and expression features are extracted from the input image or video, representing the posture and expression information of the face. The task of the optical flow transformation module is to convert these features into a more intuitive and easy-to-operate form, namely, posture optical flow and expression optical flow, and fuse them to generate motion optical flow.

[0053] Specifically, the posture features and expression features extracted from the reference image can be converted into the first posture optical flow and the first expression optical flow respectively. The posture features here describe the position, orientation and angle of the object or face in the reference image in space, and the first posture optical flow is a vector field that represents the change of facial posture over time. Each vector points to the position of a point on the face at a certain moment in the future or past, and its length and direction respectively represent the distance and direction of movement. The first posture optical flow is obtained by converting the posture features in the reference image. The optical flow transformation module can use deep learning technology (such as convolutional neural network) to learn the mapping relationship from posture features to posture optical flow. This mapping relationship is learned through a large amount of training data to ensure that the module can accurately predict changes in facial posture.

[0054] Similar to posture features, the above expression features describe the expression changes of the face in the reference image, such as smiling, frowning, etc. These features will be converted into the first expression optical flow. The first expression optical flow is also a vector field, but it focuses more on the subtle expression changes on the face. The conversion process also relies on deep learning technology. The module can accurately predict the changes in facial expressions by learning the mapping relationship from expression features to expression optical flow.

[0055] Pose optical flow and expression optical flow play an important role in animation generation and facial editing. They provide an intuitive way to describe the pose and expression changes of a face. By combining these two types of optical flow information, more natural and realistic animation effects can be generated.

[0056] Furthermore, the image features also include identity features. Accordingly, in step 121, converting the expression features in the image features into the first expression optical flow includes: The identity feature and the expression feature in the image feature are spliced ​​together, and the spliced ​​features are converted into the first expression optical flow.

[0057] Specifically, identity features are key information used to distinguish different individuals. They contain information closely related to individual identity, such as facial contours and facial features. By splicing identity features and expression features and converting the spliced ​​features into expression optical flow, these identity features can be retained in the motion optical flow obtained by subsequent fusion, so that in the process of image animation generation, the identity of the character in the output image animation can be kept consistent with that in the reference image. In this way, even if the posture and expression of the character in the output animation change, its identity features can still be maintained, allowing the audience to clearly identify who the character in the animation is.

[0058] It is understandable that the extraction of identity features helps to improve the realism of image generation. By capturing individual identity features in the reference image, the model can more accurately restore these features when generating new images, making the generated images visually closer to real people. In addition, since posture and expression can be adjusted separately, by extracting identity features, the model can maintain high fidelity of identity features while adjusting posture and expression. This means that the posture and expression of the character can be freely adjusted without changing the identity of the character, thereby generating a variety of expression animations or performing facial editing.

[0059] Step 122, converting the posture feature in the video feature into a second posture optical flow, and converting the expression feature in the video feature into a second expression optical flow; Specifically, the posture features and expression features extracted from the driving video can be converted into the second posture optical flow and the second expression optical flow respectively. The specific process can refer to the description in step 131, which will not be repeated here.

[0060] Step 123: Fusing the first posture optical flow, the first expression optical flow, the second posture optical flow, and the second expression optical flow to generate the motion optical flow.

[0061] Specifically, after extracting each posture optical flow and each expression optical flow, in order to ensure the accuracy of the fusion process, these optical flow features can be preprocessed, such as alignment or normalization. Then, a suitable fusion strategy can be selected to combine these posture optical flows and expression optical flows. For example, a weighted fusion method can be adopted. According to actual needs, different weights are assigned to each posture optical flow and each expression optical flow, and then weighted summation is performed. This method can flexibly adjust the degree of influence of each optical flow on the final motion optical flow. For another example, the main element fusion method can be adopted. In some cases, the corresponding elements of the posture optical flow and the expression optical flow can be directly added or multiplied to generate the fused motion optical flow. For another example, a learning-based fusion strategy can be adopted, using deep learning techniques such as convolutional neural networks or recurrent neural networks to learn the mapping relationship from posture optical flow and expression optical flow to motion optical flow. This method can automatically capture the complex relationship between optical flows and generate high-quality motion optical flow.

[0062] According to the selected fusion strategy, the optical flows of each posture and each expression are fused to generate the final motion optical flow. This motion optical flow will contain the change information of both posture and expression, which can be used for subsequent image rendering or animation generation tasks.

[0063] Figure 3 Schematic diagram of the optical flow transformation module provided by the present invention. Figure 3 As shown in the figure, the task of the optical flow transformation module is to generate motion optical flow through potential features such as posture, expression and identity, which is used for the subsequent transformation of texture features. The input of this module is the posture features and expressions extracted from the encoder. First, the optical flow transformation module converts the posture features and expression features output by the encoder into corresponding posture optical flow and expression optical flow respectively. These optical flows describe the movement trend of the face, among which the posture optical flow reflects the global movement, such as the rotation or tilt of the head; the expression optical flow captures the subtle changes in the local area, such as the upward movement of the corners of the mouth or the opening and closing of the eyes. Next, the posture optical flow and the expression optical flow will be fused to generate a complete motion optical flow, which describes the overall motion changes in the image. This step is crucial because it combines global and local motions to provide a basis for subsequent texture feature transformation.

[0064] Based on any of the above embodiments, step 130 specifically includes: Step 131, based on the transformation module of the decoder, applying the motion optical flow to transform the three-dimensional texture features to obtain a plurality of transformed texture features; Step 132, based on the three-dimensional module of the decoder, convert each transformed texture feature into a three-dimensional space feature; Step 133: Based on the two-dimensional module of the decoder, the three-dimensional spatial feature is two-dimensionally projected to obtain the target image.

[0065] Specifically, Figure 4 Schematic diagram of the decoder provided by the present invention, such as Figure 4 As shown in Figure 1, the decoder's task is to generate the final target image based on the input 3D texture features and motion optical flow. The decoder generates the final output step by step by applying motion optical flow to the 3D texture features and dividing the processing modules into 3D modules and 2D modules.

[0066] The transform module of the decoder will adjust and transform the three-dimensional texture features according to the motion optical flow. This process can be compared to performing corresponding operations on an object according to its motion trend in three-dimensional space. Motion optical flow, as information describing the overall motion changes in the image, includes posture optical flow and expression optical flow, which reflect the global and local motion trends respectively. The transform module first receives the texture features generated from the encoder and the motion optical flow generated by the optical flow transform module. Then, the transform module uses the information in the motion optical flow to transform the three-dimensional texture features, such as stretching, rotating or translating, to simulate changes consistent with the posture and expression of the character in the driving video, thereby obtaining the transformed texture features. Here, the transformed texture features refer to the three-dimensional texture features that have been adjusted and transformed by the transform module according to the motion optical flow. They not only retain the texture information in the original texture features, but also make corresponding adjustments according to the information in the motion optical flow to reflect the new posture and expression.

[0067] Next, the decoder's 3D module receives the transformed texture features and processes them. It further captures the spatial structural features through convolution operations and converts these features into highly detailed 3D representations, i.e., 3D spatial features. The 3D spatial features here refer to a representation that finely depicts and simulates the shape, texture, and lighting of an object in 3D space. This representation can capture subtle changes on the surface of an object, such as subtle differences in texture and subtle changes in lighting, thereby generating a more realistic and delicate image.

[0068] Subsequently, the decoder's 2D module will receive the 3D spatial features and be responsible for projecting them in 2D, further optimizing the texture details of the image through a convolutional neural network to generate the final 2D image, i.e., the target image. This process is similar to projecting an object in 3D space onto a 2D plane. The 2D module will use principles such as perspective projection to convert information such as shape, texture, and lighting in the 3D spatial features into pixel values ​​in a 2D image. Through this process, the decoder is able to generate a target image related to the input posture, expression, and identity, which maintains a high degree of sophistication and visual consistency at the 2D level. The target image here refers to the image information represented on a 2D plane.

[0069] Based on any of the above embodiments, the encoder learns the scaling parameters and the stretching transformation during the training process.

[0070] It should be noted that the existing FOMM model is prone to being overly sensitive to local features (such as facial details or body movements) when processing large-scale motion or complex scenes, resulting in uncoordinated or distorted motion in different regions of the generated image animation. In this regard, in the embodiment of the present invention, when the encoder extracts potential features from the reference image and the driving video, it learns the scaling parameters and stretching transformations, so that the model can learn potential motion features that are insensitive to the region, thereby enhancing the robustness of regional motion.

[0071] Specifically, by introducing a spatial transformer network in the encoder, using deformable convolutions, combining attention mechanisms, and designing appropriate loss functions, the encoder can learn scaling parameters and stretching transformations. Here, the Spatial Transformer Network (STN) is a neural network module that can learn image spatial transformations (such as scaling, rotation, translation, etc.). In the encoder, an STN module can be embedded, which can receive an input image and output a transformed image or feature map. The STN module contains a localization network that is responsible for predicting transformation parameters (including scaling parameters and stretching transformation parameters). These parameters are then used in a sampling grid generator and a sampler to produce a transformed feature map.

[0072] Deformable convolution is a convolution operation that can learn the sampling position of the convolution kernel. Compared with standard convolution, deformable convolution can take into account spatial transformations during the convolution process, thereby achieving dynamic sampling of image features. In the encoder, some standard convolution layers can be replaced with deformable convolution layers so that these layers can learn and adapt to the scaling and stretching transformations of the image. In this way, the encoder can handle the spatial changes of the image more flexibly when extracting latent features.

[0073] The attention mechanism can help the model focus on important areas in the image when extracting features. In the encoder, an attention module can be introduced, which can learn and generate an attention map to indicate which areas in the image are important. This attention map can be used as a weight map to weight each position of the feature map, thereby achieving implicit learning of scaling and stretching transformations. When the image is scaled or stretched, the attention map can adjust its weight distribution accordingly to ensure that the encoder can accurately extract the potential features.

[0074] It is understandable that no matter which of the above methods is used, end-to-end training is required to optimize the model parameters. During the training process, a suitable loss function can be designed to evaluate the model performance and adjust the model parameters accordingly. For learning scaling and stretching transformations, transformation-related regularization terms or constraints can be added to the loss function to guide the model to learn more robust feature representations.

[0075] Based on any of the above embodiments, Figure 5 FIG. 2 is a flow chart of the image animation generation method provided by the present invention. Figure 5 As shown in the figure, the method can be implemented by generating a model, the input of which is a reference image and a driving video, and the output is an image animation (i.e., the video corresponding to the reference image), in which the identity features in the image animation are consistent with those in the reference image, and the posture and expression are consistent with those in the driving video. The entire model architecture consists of four main parts: encoder, Warp Flow module (i.e., optical flow transformation module), decoder, and image generation. First, the encoder extracts posture, expression, identity features, and three-dimensional texture features through the backbone network and multi-head fully connected layers. Subsequently, the Warp Flow module converts these potential features into posture optical flow and expression optical flow, and synthesizes them into a complete motion optical flow to describe the overall motion transformation. The decoder uses three-dimensional texture features and motion optical flow to gradually decode features through three-dimensional modules and two-dimensional modules to generate the final image. The advantage of this architecture is that it can flexibly control posture and expression while maintaining high fidelity of identity features, and achieve high-quality image generation through three-dimensional and two-dimensional decoding processes.

[0076] 1) Encoder: The main function of the encoder is to extract a variety of potential features from the input image and driving video for subsequent motion generation and image synthesis. The encoder consists of a basic backbone network and multiple multi-head fully connected layers. The backbone network of the encoder is composed of a convolutional neural network, which is used to extract the basic features of the input image and driving video. This part of the network will learn the overall structural information of the image, such as shape, texture, etc. After the backbone network extracts the features, the multi-head fully connected layer will process these features in parallel and output different types of potential features, including posture features, expression features, identity features, texture features, etc.

[0077] 2) Warp Flow module: The task of the Warp Flow module is to generate motion optical flow through latent features such as posture, expression and identity, which is used for the subsequent transformation of texture features. The input of this module is the latent features extracted from the encoder. First, the Warp Flow module converts the posture features and expression features output by the encoder into corresponding posture optical flow and expression optical flow respectively. These optical flows describe the movement trend of the face, among which the posture optical flow reflects the global movement, such as the rotation or tilt of the head; the expression optical flow captures the subtle local changes, such as the upward movement of the corners of the mouth or the opening and closing of the eyes. Next, the posture optical flow and the expression optical flow will be fused to generate a complete motion optical flow, which describes the overall motion changes in the image. This step is crucial because it combines global and local motions to provide a basis for subsequent volume transformation.

[0078] 3) Decoder: The task of the decoder is to generate the final image based on the input 3D texture features and motion optical flow. The decoder applies motion optical flow to the 3D texture features and is divided into 3D and 2D processing modules to gradually generate the final output. The decoder first transforms the texture features generated by the encoder and the motion optical flow generated by the Warp Flow module. By applying motion optical flow to the 3D texture features, the model can adjust the shape or position of the 3D texture features according to the generated motion information. This process is similar to stretching, rotating or translating an object in 3D space according to the motion changes. After the 3D texture features are transformed by motion optical flow, the 3D module of the decoder receives the transformed features and is responsible for processing the 3D information in the texture features. The spatial structure features are further captured through convolution operations and converted into a highly detailed 3D representation. Next, the 2D module of the decoder projects the 3D representation information into 2D and further optimizes the texture details of the image through a convolutional neural network. This stage converts the 3D spatial information into 2D image information, so that the output image maintains a high degree of fineness and visual consistency on a 2D level. After processing by the decoder’s 3D and 2D modules, the model can generate image animations related to the input pose, expression, and identity. The image animations retain the main features of the reference image, but are updated based on motion information to present new poses and expressions.

[0079] The advantage of the embodiments of the present invention is that the overall effect of image animation generation is significantly improved. First, through the combination of three-dimensional texture features and motion optical flow, the model can accurately capture the motion changes in three-dimensional space, significantly reduce the stretching distortion problem in image generation, and enhance the naturalness and realism of the animation. At the same time, the model enhances the robustness of motion in different regions by learning scaling parameters and stretching transformations, ensuring the coordination and consistency of various regions in complex actions or large-scale movements. In addition, by extracting expression features, the model is given the ability to modify expressions independently, so that it can flexibly generate a variety of expression animations without affecting the overall motion performance. These improvements significantly improve the adaptability of the model in complex scenes and the robustness of animation generation, which significantly improves the generation quality, detail control and flexibility.

[0080] Based on any of the above embodiments, Figure 6 is a schematic diagram of the structure of the image animation generating device provided by the present invention, such as Figure 6 As shown, the device comprises: The encoding unit 610 is used to extract features from the reference image and the driving video based on the encoder of the generative model to obtain image features and video features, wherein the image features include three-dimensional texture features; A transformation unit 620, configured to process the image features and the video features based on the optical flow transformation module of the generation model to generate motion optical flow; A decoding unit 630, configured to decode the three-dimensional texture features and the motion optical flow based on a decoder of the generation model to obtain a plurality of target images; The generating unit 640 is used to synthesize the multiple target images into an image animation based on the generating module of the generating model.

[0081] The device provided by the embodiment of the present invention can efficiently capture the three-dimensional texture features of the image and the motion features of the video by extracting features of the reference image and the driving video through the encoder of the generative model. This multi-dimensional feature provides a rich information basis for subsequent animation generation. By utilizing the optical flow transformation module in the generative model, combined with image features and video features, the motion optical flow can be accurately generated. By combining the three-dimensional texture features and the motion optical flow through the decoder, a high-quality target image can be generated, thereby flexibly generating image animation based on the target image. The present invention combines three-dimensional texture features with motion optical flow, so that the model can accurately capture the movement changes of objects in three-dimensional space and restore the true form and motion state of the object, thereby significantly reducing the stretching distortion problem in image animation generation and enhancing the naturalness and realism of the image animation.

[0082] Based on any of the above embodiments, the encoding unit 610 is specifically used for: Based on the image encoding unit of the encoder, feature extraction is performed on the reference image to obtain the image features, wherein the image features also include posture features and expression features; Based on the video encoding unit of the encoder, feature extraction is performed on the driving video to obtain the video features, which include posture features and expression features.

[0083] Based on any of the above embodiments, the transforming unit 620 includes: A first conversion subunit, used for converting the posture feature in the image feature into a first posture optical flow, and converting the expression feature in the image feature into a first expression optical flow; A second conversion subunit, used for converting the posture feature in the video feature into a second posture optical flow, and converting the expression feature in the video feature into a second expression optical flow; The fusion subunit is used to fuse the first posture optical flow, the first expression optical flow, the second posture optical flow and the second expression optical flow to generate the motion optical flow.

[0084] Based on any of the foregoing embodiments, the image feature further includes an identity feature, and the first transformation subunit is specifically configured to: The identity feature and the expression feature in the image feature are spliced ​​together, and the spliced ​​features are converted into the first expression optical flow.

[0085] Based on any of the above embodiments, the decoding unit 630 is specifically configured to: Based on the transformation module of the decoder, the motion optical flow is applied to transform the three-dimensional texture features to obtain a plurality of transformed texture features; Based on the three-dimensional module of the decoder, each transformed texture feature is converted into a three-dimensional spatial feature; Based on the two-dimensional module of the decoder, the three-dimensional spatial features are two-dimensionally projected to obtain the target image.

[0086] Based on any of the above embodiments, the encoder learns the scaling parameters and the stretching transformation during the training process.

[0087] Based on any of the above embodiments, the identity features of the image animation are consistent with the identity features of the reference image, and the posture features and expression features of the image animation are consistent with the posture features and expression features of the driving video.

[0088] Figure 7 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 7 As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730 and a communication bus 740, wherein the processor 710, the communication interface 720 and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute the image animation generation method, which includes: based on the encoder of the generation model, respectively extracting features of the reference image and the driving video to obtain image features and video features, wherein the image features include three-dimensional texture features; based on the optical flow transformation module of the generation model, processing the image features and the video features to generate motion optical flow; based on the decoder of the generation model, decoding the three-dimensional texture features and the motion optical flow to obtain multiple target images; based on the generation module of the generation model, synthesizing the multiple target images into an image animation.

[0089] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the relevant technology or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0090] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the image animation generation method provided by the above-mentioned methods, which includes: an encoder based on a generation model, respectively extracting features of a reference image and a driving video to obtain image features and video features, wherein the image features include three-dimensional texture features; an optical flow transformation module based on the generation model, processing the image features and the video features to generate motion optical flow; a decoder based on the generation model, decoding the three-dimensional texture features and the motion optical flow to obtain multiple target images; and a generation module based on the generation model, synthesizing the multiple target images into an image animation.

[0091] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the image animation generation method provided by the above-mentioned methods, the method comprising: an encoder based on a generative model, extracting features of a reference image and a driving video respectively, to obtain image features and video features, wherein the image features include three-dimensional texture features; an optical flow transformation module based on the generative model, processing the image features and the video features, to generate motion optical flow; a decoder based on the generative model, decoding the three-dimensional texture features and the motion optical flow, to obtain multiple target images; and a generation module based on the generative model, synthesizing the multiple target images into an image animation.

[0092] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0093] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment.

[0094] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating an image animation, characterized in that: include: Based on the encoder of the generative model, feature extraction is performed on the reference image and the driving video respectively to obtain image features and video features, wherein the image features include three-dimensional texture features; Based on the optical flow transformation module of the generation model, the image features and the video features are processed to generate motion optical flow; Based on the decoder of the generative model, the three-dimensional texture features and the motion optical flow are decoded to obtain a plurality of target images; Based on the generation module of the generation model, multiple target images are synthesized into an image animation.

2. The image animation generation method according to claim 1, characterized in that: The encoder based on the generative model extracts features from the reference image and the driving video respectively to obtain image features and video features, including: Based on the image encoding unit of the encoder, feature extraction is performed on the reference image to obtain the image features, wherein the image features also include posture features and expression features; Based on the video encoding unit of the encoder, feature extraction is performed on the driving video to obtain the video features, which include posture features and expression features.

3. The image animation generation method according to claim 2, characterized in that: The processing of the image features and the video features to generate motion optical flow includes: Converting the posture feature in the image feature into a first posture optical flow, and converting the expression feature in the image feature into a first expression optical flow; Converting the posture feature in the video feature into a second posture optical flow, and converting the expression feature in the video feature into a second expression optical flow; The first posture optical flow, the first expression optical flow, the second posture optical flow and the second expression optical flow are fused to generate the motion optical flow.

4. The image animation generation method according to claim 3, characterized in that: The image features further include identity features, and the step of converting the expression features in the image features into a first expression optical flow includes: The identity feature and the expression feature in the image feature are spliced ​​together, and the spliced ​​features are converted into the first expression optical flow.

5. The image animation generation method according to claim 1, characterized in that: The decoder based on the generative model decodes the three-dimensional texture features and the motion optical flow to obtain a plurality of target images, including: Based on the transformation module of the decoder, the motion optical flow is applied to transform the three-dimensional texture features to obtain a plurality of transformed texture features; Based on the three-dimensional module of the decoder, each transformed texture feature is converted into a three-dimensional spatial feature; Based on the two-dimensional module of the decoder, the three-dimensional spatial features are two-dimensionally projected to obtain the target image.

6. The image animation generation method according to any one of claims 1 to 5, characterized in that: The encoder learns scaling parameters and stretching transformations during training.

7. The image animation method according to any one of claims 1 to 5, characterized in that: The identity features of the image animation are consistent with the identity features of the reference image, and the posture features and expression features of the image animation are consistent with the posture features and expression features of the driving video.

8. An image animation generating device, characterized in that: include: An encoding unit, configured to extract features from a reference image and a driving video based on an encoder of a generative model, respectively, to obtain image features and video features, wherein the image features include three-dimensional texture features; A transformation unit, configured to process the image features and the video features based on the optical flow transformation module of the generation model to generate motion optical flow; A decoding unit, configured to decode the three-dimensional texture features and the motion optical flow based on a decoder of the generation model to obtain a plurality of target images; A generation unit is used to synthesize the multiple target images into an image animation based on the generation module of the generation model.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the image animation generating method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image animation generating method according to any one of claims 1 to 7 is implemented.

Images