Image video generation method and device based on potential space navigation learning, equipment, medium and program product

Abstract

The invention provides an image video generation method and device based on potential space navigation learning, equipment, a medium and a program product, and relates to the technical field of artificial intelligence. The method comprises the following steps: acquiring an initial image and a driving video; wherein the initial image is represented as a static image containing a first human face, the driving video is represented as a dynamic video containing a second human face, and the second human face has multiple pieces of human face action information; and inputting the initial image and the driving video into a trained image video generation model, and outputting a target image video. The embodiment provided by the invention is used for solving the defects of high cost, weak generalization ability and limited degree of freedom in the prior art, realizes that key points of the face image do not need to be marked, and the trained image video generation model is utilized to directly learn the motion mode of the driving video so as to generate the target image video, and the generalization ability of the model is high. And non-linear motion can be flexibly captured, and traditional freedom degree limitation is broken through.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to an image video generation method and device based on latent space navigation learning, equipment, medium and program product. BACKGROUND

[0002] Due to the significant progress of deep generative models, image animation has become more and more efficient, and the generated results have also become more and more realistic. Current animation methods usually utilize structural representations extracted from driving videos, which help to transfer motion from driving videos to static images. However, these methods often fail when there is a large appearance difference between the source image and the driving video. In addition, extracting structural information requires additional modules, which increases the complexity of the animation model.

[0003] In the related art, traditional image animation generation methods (such as Faceware and 3DMM) rely on explicit structural representations (such as key points and facial landmark points), which require manual annotation or detection of specific feature points. However, these methods have the following limitations: 1) High annotation cost: key point annotation requires a lot of manpower and is easily affected by noise and occlusion.

[0004] 2) Weak generalization ability: the model relies on preset structural assumptions (such as "human faces must be symmetrical"), which makes it difficult to adapt to non-standard morphologies or complex motions.

[0005] 3) Limited degrees of freedom: the number of key points is fixed, and subtle deformations (such as wrinkles and muscle tremors) cannot be flexibly expressed.

[0006] Therefore, how to overcome the technical defects of the prior art and generate image videos is a technical problem that needs to be solved at present. SUMMARY

[0007] The present application provides an image video generation method and device based on latent space navigation learning, equipment, medium and program product, which solves the defects of high cost, weak generalization ability and limited degrees of freedom in the prior art, realizes the generation of target image videos without labeling face image key points, and directly learns the motion patterns of driving videos using a trained image video generation model, which has high generalization ability and can flexibly capture nonlinear motion, breaking through the limitations of traditional degrees of freedom.

[0008] In a first aspect, the present application provides an image video generation method based on latent space navigation learning, comprising the following steps: An initial image and a driving video are obtained; wherein the initial image represents a static image containing a first face, and the driving video represents a dynamic video containing a second face, and the second face has multiple facial action information; inputting the initial image and the driving video into a trained image video generation model, and outputting a target image video; wherein the target image video is a dynamic video containing the first face, and the first face has the plurality of facial motion information of the second face; and the image video generation model is trained based on a driving video sample.

[0009] Preferably, the image video generation method based on latent space navigation learning provided by the present application comprises the following steps: sampling two frames from the driving video sample as a source image sample and a driving image sample, respectively; performing linear motion decomposition processing based on the driving image sample and the source image sample to determine a target latent motion representation; wherein the target latent motion representation represents the motion transformation of the source image sample learned to the driving image sample; performing decoding processing based on the source image sample and the target latent motion representation to determine a latent learning image sample; generating a target driving image sample based on the latent learning image sample; training the image video generation model based on the total loss function between the target driving image sample and the driving image sample to obtain the trained image video generation model.

[0010] Preferably, the image video generation method based on latent space navigation learning provided by the present application comprises the following steps: defining a reference image sample, and performing encoding processing on the driving image sample based on the reference image sample to obtain a driving latent code sample, and performing encoding processing on the source image sample based on the reference image sample to obtain a source latent code sample; wherein the driving latent code sample represents the motion transformation of the reference image sample learned to the driving image sample, and the source latent code sample represents the motion transformation of the source image sample learned to the reference image sample; obtaining a motion dictionary to be trained; wherein the motion dictionary represents all paths in the latent space; performing linear motion decomposition processing on the driving latent code sample and the motion dictionary to be trained to obtain a driving latent path; performing latent space navigation learning processing on the source latent code sample based on the driving latent path to determine the target latent motion representation.

[0011] Preferably, the image video generation method based on latent space navigation learning provided by the present application comprises the following steps: extracting multi-scale source features of the source image sample; inputting the multi-scale source features and the target latent motion representation into a generator to perform decoding processing on the multi-scale source features in the generator by using the target latent motion representation, and outputting the latent learning image sample.

[0012] Preferably, the image video generation method based on latent space navigation learning provided by the present application comprises the following steps: the inputting the multi-scale source features and the target latent motion representation into a generator to perform decoding processing on the multi-scale source features in the generator by using the target latent motion representation, and outputting the latent learning image sample, comprises: performing decoding processing on the target latent motion representation layer by layer in each layer residual network of the flow field generator to generate a flow field pyramid and a repair mask; wherein the flow field pyramid represents a displacement vector field of the source image sample to the driving image sample, and the repair mask represents a feature map for repairing the multi-scale source features after being distorted; performing flow field distortion operation on the multi-scale source features by using the flow field pyramid to obtain a distorted feature map; performing repair processing on the distorted feature map by using the repair mask to obtain a repaired feature map; inputting the repaired feature map into the refinement network to output the latent learning image sample.

[0013] Preferably, the image video generation method based on latent space navigation learning provided by the present application comprises the following steps: in each layer of the refinement network, performing up-sampling processing on the repaired feature map of each layer to obtain an up-sampled image of each layer; performing splicing and fusion processing on the repaired feature map of the current layer and the up-sampled image of the previous layer to obtain an image sample output by the current layer; taking the image sample output by the last layer of the refinement network as the latent learning image sample output by the refinement network.

[0014] In a second aspect, the present application further provides an image video generation device based on latent space navigation learning, comprising the following modules: An acquisition module is configured to acquire an initial image and a driving video; wherein the initial image represents a static image containing a first face, and the driving video represents a dynamic video containing a second face having multiple facial action information; An output target image video module is configured to input the initial image and the driving video into a trained image video generation model, and output a target image video; wherein the target image video represents a dynamic video containing the first face having the multiple facial action information of the second face; and the image video generation model is trained based on a driving video sample.

[0015] In a third aspect, the present application further provides an electronic device, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the image video generation method based on latent space navigation learning according to any one of the above aspects when executing the program.

[0016] In a fourth aspect, the present application further provides a non-transitory computer readable storage medium, which stores a computer program executable on a processor to implement the image video generation method based on latent space navigation learning according to any one of the above aspects.

[0017] In a fifth aspect, the present application further provides a computer program product comprising a computer program executable on a processor to implement the image video generation method based on latent space navigation learning according to any one of the above aspects.

[0018] The image video generation method, device, equipment, medium and program product based on latent space navigation learning provided by the present application acquire an initial image and a driving video; wherein the initial image represents a static image containing a first face, and the driving video represents a dynamic video containing a second face having multiple facial action information; the initial image and the driving video are input into a trained image video generation model, and a target image video is output; wherein the target image video represents a dynamic video containing the first face having the multiple facial action information of the second face; and the image video generation model is trained based on a driving video sample. The present application solves the defects of high cost, weak generalization ability and limited freedom in the prior art, and realizes the direct learning of the motion pattern of the driving video by using the trained image video generation model without labeling the key points of the face image, so as to generate the target image video. The model has high generalization ability and can flexibly capture nonlinear motion, thereby breaking through the limitation of traditional freedom. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to make the technical solutions in the present application or the prior art clearer, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings described below are only some embodiments of the present application, and the ordinary skilled in the art can obtain other drawings according to these drawings without any creative effort.

[0020] Figure 1 is one of the flowcharts of the image video generation method based on latent space navigation learning provided by the present application.

[0021] Figure 2 is the second schematic diagram of the image video generation method based on latent space navigation learning provided by the present application.

[0022] Figure 3 is the structural schematic diagram of the image video generation device based on latent space navigation learning provided by the present application.

[0023] Figure 4 is the structural schematic diagram of the electronic device provided by the present application. DETAILED DESCRIPTION

[0024] In order to make the technical solutions in the present application or the prior art clearer, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings described below are only some embodiments of the present application, and the ordinary skilled in the art can obtain other drawings without any creative effort.

[0025] The following will describe the image video generation method, device, equipment, medium and program product based on latent space navigation learning provided by the present application in combination with Figures 1-4 The present application provides an image video generation method, device, equipment, medium and program product based on latent space navigation learning, which solves the defects of high cost, weak generalization ability and limited freedom in the prior art, realizes the learning of the motion pattern of the driving video directly by using the trained image video generation model without labeling the key points of the face image, and generates the target image video. The model has high generalization ability and can flexibly capture nonlinear motion, breaking through the limitation of traditional freedom.

[0026] Figure 1 is one of the flowcharts of the image video generation method based on latent space navigation learning provided by the present application, as Figure 1 shown, the method can include but is not limited to steps S100 to S200: S100, obtaining an initial image and a driving video; wherein the initial image represents a static image containing a first face, the driving video represents a dynamic video containing a second face, and the second face has multiple face action information; S200, inputting the initial image and the driving video into a trained image video generation model, and outputting a target image video; wherein the target image video is a dynamic video containing the first face, and the first face has the multiple facial action information of the second face; and the image video generation model is trained based on a driving video sample.

[0027] In step S100 of some embodiments, an initial image and a driving video are obtained; wherein the initial image is a static image containing a first face, and the driving video is a dynamic video containing a second face, and the second face has multiple facial action information.

[0028] It can be understood that in the inference stage, the driving video is obtained, and the frame sequence of the driving video is , and the initial image is obtained .

[0029] The initial image is a static image containing a first face, and the driving video is a dynamic video containing a second face, and the second face has multiple facial action information, such as laughing, smiling, crying, opening the mouth, blinking, etc.

[0030] It should be noted that the first face and the second face can be the same face or different faces.

[0031] In step S200 of some embodiments, the initial image and the driving video are input into a trained image video generation model, and a target image video is output; wherein the target image video is a dynamic video containing the first face, and the first face has the multiple facial action information of the second face; and the image video generation model is trained based on a driving video sample.

[0032] It can be understood that the initial image and the driving video are input into a trained image video generation model, and a target image video is output.

[0033] The target image video is a dynamic video containing a first face, and the first face has the same facial action as the multiple facial action information of the second face.

[0034] In this way, only an initial image and a driving video need to be input, and a pre-trained model does not need to be fine-tuned. By learning the motion dictionary in the latent space, the generated motion direction has interpretability. Users can intuitively understand and control the generated motion pattern, enhancing the controllability of the model.

[0035] The image video generation model is trained based on the driving video sample, specifically, is trained based on linear motion decomposition of the driving video sample, to obtain a trained motion dictionary and a trained image video generation model, without labeling the face image key points, the face action of the first face learning the second face can be realized.

[0036] In some embodiments of the present application, the goal is to transfer the face motion in the driving video from to , to generate a new target image video .

[0037] Further, if and come from the same video sequence, that is , the task of the present application includes reconstructing the entire original video sequence.

[0038] Therefore, the absolute transfer strategy is used to construct the potential motion representation of each frame, following the training process, and the formula is as follows: Further, the present application also proposes a relative transfer to eliminate the motion influence of and introduce the motion of in the entire generated driving video sequence. Due to the linear representation of the potential path, the of each frame can be easily represented as: The first term in the formula represents the reconstruction of , and the second term represents the motion from to . The formula shows that the original pose of the first face of the initial image is preserved in , while the face action motion of the second face in the driving video is transferred from the driving video .

[0039] The embodiment of the present application realizes implicit motion modeling through linear motion decomposition (LMD), directly learns the motion pattern from the RGB video (driving video), without manually defining structural rules. The core advantages at least include but are not limited to: Data-driven: the motion pattern is adaptively learned from data, suitable for any shape and motion type.

[0040] Simplified process: omit the key point detection and matching steps, and train the generation model end to end.

[0041] High flexibility: can capture nonlinear motion (such as expression changes, clothing swings), break through the freedom limit of traditional methods.

[0042] In some embodiments of the present application, the step of training the image video generation model based on the driving video sample comprises: Sampling two frames from the driving video sample as a source image sample and a driving image sample, respectively; Based on the driving image sample and the source image sample, linear motion decomposition processing is performed to determine the target latent motion representation; wherein the target latent motion representation represents the learning of the source image sample to the motion transformation of the driving image sample; Based on the source image sample and the target latent motion representation, decoding processing is performed to determine the latent learning image sample; Based on the latent learning image sample, a target driving image sample is generated; Based on the total loss function between the target driving image sample and the driving image sample, the image video generation model is trained to obtain the trained image video generation model.

[0043] It can be understood that when training the image video generation model based on the driving video sample, two frames are sampled from the driving video sample, one frame as a source image sample , and the other frame as a driving image sample .

[0044] It should be noted that in some embodiments, the sampling can be random sampling to increase the flexibility of the image video generation model.

[0045] Further, based on the driving image sample and the source image sample linear motion decomposition processing is performed to determine the target latent motion representation . The complex nonlinear motion (such as expression changes) is simplified to a learnable linear representation, reducing the complexity of the model.

[0046] Further, based on the source image sample and the target latent motion representation decoding processing is performed to determine the latent learning image sample . That is, the source image and the target latent motion representation are input into the decoder (such as GAN generator or VAE decoder) to generate the latent learning image sample.

[0047] This embodiment applies the motion parameters to the source image to generate a preliminary dynamic image (which may not completely match the driving image) that retains the identity features of the source image while injecting the motion information of the driving image.

[0048] Furthermore, based on the potential learning image sample Generate target driving image samples , ensuring the generation of reconstructed images Approximation drive image appearance and sporty details.

[0049] It should be noted that the original driver image Contains complex information such as the motion + background + identity of the second face, and the reconstructed target driving image Contains only the motion information of the second face.

[0050] Furthermore, the image video generation model is trained based on the total loss function between the target driving image sample and the driving image sample to obtain the trained image video generation model.

[0051] It can be understood that the image and video generation model includes at least a generator and a discriminator.

[0052] Initialize the parameters of the generator and discriminator, record the pre-trained VGG19 model, freeze its weights, and use it only for feature extraction.

[0053] The generator is used to combine the source image samples and the target latent motion representation to generate latent learning image samples, so as to create target driving image samples based on the latent learning image samples. The discriminator is used to distinguish the differences between the target driving image samples and the driving image samples.

[0054] In an embodiment of the present invention, the image video generation model is trained in a self-supervised manner to reconstruct the target driving image sample. , three loss functions are used, namely reconstruction loss , Perceptual Loss and combat losses .

[0055] Based on reconstruction loss , Perceptual Loss and combat losses A total loss function between the target driving image samples and the driving image samples may be determined.

[0056] Further, use To minimize the target driving image samples and driving image samples Pixel level between Distance, calculated as: In order to minimize the perceptual distance, in real images (potential learning image samples ) and the generated image (target-driven image sample ) using VGG19-based perceptual loss , which is formulated as: where denotes the i-th layer in the pre-trained VGG19 network.

[0057] In addition, to generate realistic results, an unsaturated adversarial loss is introduced on the latent learning image sample , which is formulated as: where is a discriminator, aiming to distinguish the reconstructed image (target-driven image sample ) and the original image (driven image sample ).

[0058] The total loss function is a combination of the above three losses, as a balance hyperparameter: is the total loss function, is the reconstruction loss function, is the perceptual loss function, is the adversarial loss function.

[0059] Reconstruction loss: ensures that the target-driven image sample and the driven image sample are aligned at the pixel level.

[0060] Perceptual loss: ensures that the real image (latent learning image sample ) and the generated image (target-driven image sample ) are consistent in semantic features (such as edges, textures).

[0061] Adversarial loss: improves the authenticity of the target-driven image sample and reduces synthetic traces.

[0062] Further, in some embodiments, the steps of the training loop include: Generate image: input source image sample and driven image sample, get target-driven image sample through generator.

[0063] Calculate loss: reconstruction loss: compare the pixel difference between the target-driven image sample and the driven image sample . ​

[0064] Perceptual loss: extract target driving image sample by VGG19 Calculate difference with multi-scale features of driving image.

[0065] Adversarial loss: score target driving image sample by discriminator Calculate adversarial loss.

[0066] Backpropagation: fix discriminator parameters, update generator parameters to minimize total loss. Fix generator parameters, update discriminator parameters to enhance discrimination ability.

[0067] Iterative optimization: repeat above steps until total loss converges or reaches preset training rounds.

[0068] This step works: gradually optimize generator and discriminator through gradient descent, making generated images approach real images. Adversarial training encourages generator to generate more realistic images, and discriminator improves discrimination ability.

[0069] Further, the saving and evaluation steps of the image video generation model include: saving the parameters of the generator and discriminator regularly during training (e.g. every 1000 iterations). Evaluate the quality of generated images on the validation set (e.g. PSNR, SSIM, visual inspection). Effect: ensure that the model does not overfit during training, and select the optimal parameters for inference.

[0070] The above embodiments provided by the present application at least achieve the following technical effects: identity fidelity: reconstruction loss and perceptual loss ensure that the generated target driving image sample retains the identity features of the source image sample. Action accuracy: learning of latent motion representation makes the generated target driving image sample reproduce the action of the driving image sample. High fidelity: adversarial loss improves the detail realism of the generated target driving image sample, reducing synthetic artifacts. Robustness: multi-loss joint optimization makes the image video generation model adapt to different identity, lighting and action scenarios.

[0071] Driving image samples are directly sampled from videos and contain real pixel values.

[0072] Reconstructed target driving image samples: generated by image video generation model, rely on latent space motion decomposition and combination.

[0073] The differences between driving image samples and target driving image samples: Different data forms: driving image samples are real data, and target driving image samples are model-generated approximations.

[0074] Different uses: driving image samples are used for supervised training, and target driving image samples are used to verify whether the model has learned the correct motion representation.

[0075] The target driving image sample is created based on the driving image sample, so as to constrain the model by the reconstruction loss, and enable the model to generate an image conforming to the driving motion from an arbitrary source image.

[0076] Embodiments of the present application separate identity (static feature) and motion (dynamic feature) through latent space decomposition, avoid overfitting, and optimize model parameters through reconstruction loss, perception loss and adversarial loss, so as to generate high-fidelity images.

[0077] In some embodiments of the present application, the linear motion decomposition processing based on the driving image sample and the source image sample is performed to determine the target latent motion representation, including: A reference image sample is defined, and the driving image sample is encoded based on the reference image sample to obtain a driving latent code sample, and the source image sample is encoded based on the reference image sample to obtain a source latent code sample; wherein the driving latent code sample represents that the reference image sample learns the motion transformation of the driving image sample, and the source latent code sample represents that the source image sample learns the motion transformation of the reference image sample; A motion dictionary to be trained is obtained; wherein the motion dictionary represents all paths in the latent space; The driving latent code sample and the motion dictionary to be trained are used for linear motion decomposition processing to determine a driving latent path from all paths in the latent space; The source latent code sample is navigated and learned in the latent space based on the driving latent path to determine the target latent motion representation.

[0078] It can be understood that, referring to Figure 2 , it is assumed that there is a reference image sample , and the motion transfer can be modeled as , wherein, is learned in an indirect manner. Wherein, is a source image sample, is a reference image sample, is a driving image sample, is a target latent motion representation.

[0079] The target latent motion representation as a target point in the latent space, can be reached by linear walking from the starting point (source latent code sample) along the linear path (driving latent path), which is given by the following formula: In the formula, is a target latent motion representation, is a source latent code sample, is a driving latent path.

[0080] (source latent code sample) and (driving latent path) respectively represent (the source image sample learns the motion transformation of the reference image sample) and (the reference image sample learns the motion transformation of the driving image sample), and are all independently learnable, (source latent code sample) is obtained by encoding the source image sample through an autoencoder.

[0081] The driving latent code sample and the source latent code sample are used to explicitly express the motion difference between images as a vector offset in the latent space, facilitating subsequent linear decomposition.

[0082] The driving latent path is learned through linear motion decomposition (LMD), the idea of the present application is to learn a set of motion directions representing any path in the latent space.

[0083] where the motion dictionary to be trained is , and is constrained to be an orthogonal basis, which is a set of orthogonal basis vectors in the latent space, wherein the motion dictionary contains multiple motion vectors, each motion vector represents a basic visual transformation (eyebrow, head shaking, etc.), and each motion vector represents a motion direction . Then each vector motion in the orthogonal basis is combined with the driving latent code sample , where represents the size of . Therefore, any linear path (driving latent path) in the latent space can be represented by a linear combination: where is the weighted coefficient of the basis motion vector , which controls the contribution strength of the direction to the final motion, and for all , , . Semantically, each should represent a basic visual transformation, represents the steps required to realize in , that is, the motion path of the latent space is regarded as "walking", represents walking in the direction The number of steps or distance to be moved.

[0084] The motion dictionary to be trained Is implemented as a learnable matrix and applies a Gram-Schmidt process in each forward pass to meet the requirement of orthogonality.

[0085] Is obtained by mapping , The driving image sample After the encoder E, the encoding mapped to the reference state latent space. Therefore, based on the driving latent path, the source latent code sample is navigated in the latent space learning process, and the formula for determining the target latent motion representation is as follows: Wherein, The target latent motion representation, The source latent code sample, The formula for determining the driving latent path.

[0086] Wherein, the Gram-Schmidt process is a method for converting a linearly independent vector group into an orthogonal vector group, which is a conventional technique known to those skilled in the art and will not be described here.

[0087] The above embodiments provided by the present application decompose complex motion into independent basic transformations, which can reduce the complexity of the model. The orthogonality ensures that there is no redundancy in different motion directions, and improves the flexibility of decomposition and combination. The motion of the driving image sample is decomposed into a linear combination of basis vectors, and the motion pattern is explicitly modeled.

[0088] Driving latent path: using the driving latent code sample and the motion dictionary to be trained, generating a path from the source latent code to the target latent code.

[0089] Navigation learning: "walking" in the latent space through linear interpolation, simulating the motion process from the source image sample to the driving image sample.

[0090] The latent representation of the source image is migrated to the target state through basis vector combination, realizing motion transfer.

[0091] The navigation process ensures that the generated motion is coherent and consistent with physical laws (such as gradual change rather than sudden change).

[0092] It should be noted that the embodiment decouples the identity and motion by referring to the frame, decomposes the motion into independent components by using the orthogonal basis motion dictionary, and finally generates the target driving image sample through latent space navigation. The core purpose is to let the model learn the general motion pattern, so as to support the combination of any source image and driving motion, rather than just copying the original driving image. In the training process, the reconstruction loss ensures that the generated image is consistent with the real image, and the perception loss and the adversarial loss improve the visual fidelity.

[0093] In some embodiments of the application, the decoding process based on the source image sample and the target latent motion representation determines a latent learning image sample, comprising: extracting multi-scale source features of the source image sample; inputting the multi-scale source features and the target latent motion representation into a generator to decode the multi-scale source features using the target latent motion representation in the generator, and outputting the latent learning image sample.

[0094] It can be understood that the encoder E in Figure 2 extracts features from the source image sample to obtain feature maps at different levels, i.e. multi-scale source features.

[0095] For example, shallow features (high frequency details) and deep features (low frequency semantic information) are extracted to form multi-scale source features. This step is used to capture local details and global structure of the source image, providing multi-granularity information for subsequent feature warping. Multi-scale source features can enhance the adaptability of the model to complex texture and shape changes.

[0096] In some embodiments of the application, the generator includes at least a flow field generator and a refinement network. The inputting the multi-scale source features and the target latent motion representation into a generator to decode the multi-scale source features using the target latent motion representation in the generator, and outputting the latent learning image sample, comprises: decoding the target latent motion representation layer by layer in the residual network of each layer of the flow field generator to generate a flow field pyramid and a repair mask; wherein the flow field pyramid represents a displacement vector field from the source image sample to the driving image sample, and the repair mask represents a feature map after warping. performing flow field warping operation on the multi-scale source features using the flow field pyramid to obtain a warped feature map; performing repair processing on the warped feature map using the repair mask to obtain a repaired feature map; inputting the repaired feature map into the refinement network to output the latent learning image sample.

[0097] It can be understood that the multi-scale source feature and the target latent motion representation are input into the residual network of the flow field generator to decode layer by layer, and each residual module performs the following operations: The flow field generation step: generate a 3-channel feature map through a convolutional layer, the first two channels are flow fields represented by , and the third channel is a repair mask represented by .

[0098] The motion decoding step: based on the guidance of the target latent motion representation , adjust the direction and amplitude of the flow field, and gradually generate a flow field pyramid (multi-scale displacement vector field).

[0099] The mask prediction step: output the repair mask through the third channel, marking the occluded area that needs to be repaired.

[0100] The role of the flow field pyramid: through multi-layer decoding of the residual network, generate displacement vector fields of different scales to adapt to the multi-scale features of the source image samples.

[0101] The role of the repair mask: explicitly locate the occluded area to provide spatial guidance for subsequent feature repair.

[0102] The role of motion guidance: use the guidance of the target latent motion representation to constrain the generation of the flow field, ensuring that the displacement conforms to the motion pattern of the target driving image sample.

[0103] Further, the flow field pyramid and the multi-scale source feature are matched by level (e.g., shallow flow field corresponds to high-frequency feature, deep flow field corresponds to low-frequency feature).

[0104] Perform the differentiable warping step: for each layer of source feature, perform bilinear interpolation warping based on the corresponding flow field to obtain the warped feature map .

[0105] The role of this embodiment is to migrate the source feature image sample to the target position according to the displacement vector specified by the flow field, simulating the motion effect of the driving image sample. Through layered warping, the details and structure of the source image sample are preserved, avoiding distortion caused by a single scale flow field.

[0106] Further, the repair mask is used on the warped warped feature map , and the area that needs to be repaired is shielded through element-wise multiplication to obtain the feature map output by the repair layer, i.e., the repair feature map.

[0107] The repair layer (such as attention mechanism or context fusion module) predicts the pixel value of the occluded area according to the valid features around the mask area.

[0108] The embodiment at least achieves the following technical effects: missing areas are located by masks, features are repaired in combination with context information, and information loss caused by occlusion is reduced. The repaired feature map is ensured to be smoothly transitioned in space and semantics, and artifacts are avoided.

[0109] Further, in some embodiments of the present application, inputting the repaired feature map into the refinement network and outputting the latent learning image sample include: In each layer of the refinement network, the repaired feature map of each layer is up-sampled to obtain an up-sampled image of each layer; The repaired feature map of the current layer and the up-sampled image of the previous layer are spliced and fused to obtain an image sample output by the current layer; The image sample output by the last layer of the refinement network is taken as the latent learning image sample output by the refinement network.

[0110] It can be understood that for the repaired feature map input to each layer, an up-sampling operation (such as transposed convolution, bilinear interpolation or pixel rearrangement) is used to increase the resolution to the target size. For example, if the size of the current layer feature map is HxW, the up-sampling changes it to 2Hx2W.

[0111] In some embodiments, the parameter settings of up-sampling are as follows: Transposed convolution: a convolution kernel with a step of 2 and a kernel size of 3x3 is used, and the padding mode is "same".

[0112] Interpolation method: directly expand the feature map size by algorithm (such as bilinear interpolation), without introducing learnable parameters.

[0113] Hierarchical progression: starting from the first layer of the refinement network, up-sampling is performed layer by layer, and the up-sampled image output by each layer is taken as one of the inputs of the next layer.

[0114] The function of this embodiment is to gradually increase the resolution of the feature map and finally generate a high-resolution image consistent with the size of the driving image sample.

[0115] Aligning multi-scale features: through hierarchical up-sampling, the flow field pyramid and the source image sample features of different levels are matched to ensure the accuracy of motion transfer.

[0116] Avoiding information loss: up-sampling operation preserves detailed features and avoids blurring or artifacts caused by direct decoding.

[0117] The step of splicing and fusing the repaired feature map of the current layer and the up-sampled image of the previous layer is to splice the repaired feature map of the current layer and the up-sampled image of the previous layer in the channel dimension to form a fused feature map.

[0118] The concatenated fusion feature map is reduced in dimension and features are extracted by a convolutional layer (such as 3x3 convolution + normalization + ReLU) to output the image sample of the current layer after fusion.

[0119] For the first layer, if there is no up-sampling image of the previous layer, the repair feature map of the current layer is directly used as the input.

[0120] The embodiment at least realizes the following technical effects: by concatenating feature maps of different levels, low-level details and high-level semantic information are integrated, and the authenticity of the generated image is enhanced. The up-sampling image of the previous layer is fused to retain the cross-layer context information and avoid feature fragmentation caused by hierarchical processing. The concatenation operation allows the model to adaptively adjust the weights of features at different levels, improving the generalization ability.

[0121] Furthermore, in the last layer (such as the Nth layer) of the refinement network, the fused image sample is finally processed (such as Tanh activation function or Sigmoid normalization) to ensure that the pixel value is within a reasonable range (such as [0, 1]), and the image sample output by the last layer after processing is taken as the potential learning image sample finally output by the refinement network.

[0122] The embodiment at least realizes the following technical effects: the last layer integrates all levels of up-sampling and fused features, and outputs high-resolution and detailed images. The embodiment can also be optimized end-to-end: through layer-by-layer progressive up-sampling and fusion, the final image sample directly reflects the decoding result of the generator on the target potential motion representation. The image sample output by the last layer approximates the driving image in terms of motion patterns and texture details, meeting the task requirements.

[0123] In some embodiments of the present application, as shown in FIG. 1, after obtaining the target potential motion representation Figure 2 , the generator is used to decode the flow field and warp the source image sample . Wherein the generator is composed of two components: a flow field generator and a refinement network .

[0124] In order to learn multi-scale source features, the generator is designed as a residual network containing models, which is used to generate a flow field pyramid in different layers of the flow field generator .

[0125] Further, multi-scale source features are extracted from the encoder , and are input into ​Predicting multi-scale source features The warped feature map is warped to obtain a warped feature map.

[0126] Source image samples Some positions of the source image samples may be occluded, and in order to predict the pixels of the positions, the network needs to repair the warped feature map after warping.

[0127] Therefore, the present application predicts a multi-scale mask and a flow field in the to shield the area that needs to be repaired. In each residual module, the following operations are performed: wherein, represents an element-wise product, represents a mask feature, represents a warping operation, describes the pixel-level motion (such as displacement vector field) from the source image sample to the driving image sample, represents the encoded original feature map, represents a repair mask.

[0128] The dense flow field and the repair mask are generated by letting each residual module output a 3-channel feature map, wherein the first two channels are represented as , and the last channel is output as .

[0129] Based on the repaired repair feature map and the up-sampled image provided by the previous module in the , the RGB image (image sample) of each module is given by the following formula: wherein and respectively represent the repair layer and the up-sampling layer, and the image sample output by the last module constitutes the finally generated latent learning image sample .

[0130] The image video generation method based on latent space navigation learning provided by the present application proposes two strategies of absolute transfer and relative transfer through the above-mentioned embodiments, which are respectively applicable to the scenarios where the source image and the driving video come from the same or different identities. The motion transfer task under different scenarios can be flexibly handled, while the stability and consistency of the generated video are maintained.

[0131] The image video generation method based on latent space navigation learning provided by the application can only input a source image and a driving video in the inference stage, without fine-tuning the pre-trained model. By learning the motion dictionary in the latent space, the generated motion direction has explainability. Users can intuitively understand and control the generated motion mode, enhancing the controllability of the model.

[0132] The embodiments provided by the application significantly improve the performance of the image animation task through latent space navigation, self-supervised learning, linear motion decomposition, and multi-scale flow field generation. Not only does it improve the quality and temporal consistency of the generated video, but it also reduces the dependence on external structure representation, making the model more efficient, flexible, and practical.

[0133] The image video generation method, device, equipment, medium and program product based on latent space navigation learning provided by the application can obtain an initial image and a driving video. The initial image represents a static image containing a first face, and the driving video represents a dynamic video containing a second face. The second face has multiple facial action information. The initial image and the driving video are input into a trained image video generation model to output a target image video. The target image video represents a dynamic video containing the first face, and the first face has multiple facial action information of the second face. The image video generation model is trained based on a driving video sample. The application solves the problems of high cost, weak generalization ability, and limited freedom in the prior art. It does not require labeled facial image key points and directly learns the motion pattern of the driving video using the trained image video generation model to generate a target image video. The model has high generalization ability and can flexibly capture nonlinear motion, breaking through the traditional freedom limit.

[0134] The image video generation device based on latent space navigation learning provided by the application is described below. The image video generation device based on latent space navigation learning described below can be mutually referenced with the image video generation method based on latent space navigation learning described above.

[0135] As Figure 3 The structure diagram of the image video generation device based on latent space navigation learning provided by the application is shown in the figure. The image video generation device based on latent space navigation learning includes the following modules: The acquisition module 310 is configured to acquire an initial image and a driving video. The initial image represents a static image containing a first face, and the driving video represents a dynamic video containing a second face. The second face has multiple facial action information. The output target image video module 320 is configured to input the initial image and the driving video into a trained image video generation model to output a target image video, wherein the target image video is a dynamic video containing the first face, and the first face has the multiple facial motion information of the second face; and the image video generation model is trained based on a driving video sample.

[0136] Preferably, the image video generation device based on latent space navigation learning provided by the application is further configured to sample two frames from the driving video sample as a source image sample and a driving image sample, respectively. The device is further configured to perform linear motion decomposition processing based on the driving image sample and the source image sample to determine a target latent motion representation, wherein the target latent motion representation represents a motion transformation learned by the source image sample to the driving image sample. The device is further configured to perform decoding processing based on the source image sample and the target latent motion representation to determine a latent learning image sample. The device is further configured to generate a target driving image sample based on the latent learning image sample. The device is further configured to train the image video generation model based on a total loss function between the target driving image sample and the driving image sample to obtain the trained image video generation model.

[0137] Preferably, the image video generation device based on latent space navigation learning provided by the application is further configured to define a reference image sample, perform encoding processing on the driving image sample based on the reference image sample to obtain a driving latent code sample, and perform encoding processing on the source image sample based on the reference image sample to obtain a source latent code sample, wherein the driving latent code sample represents a motion transformation learned by the reference image sample to the driving image sample, and the source latent code sample represents a motion transformation learned by the source image sample to the reference image sample. The device is further configured to obtain a motion dictionary to be trained, wherein the motion dictionary represents all paths in the latent space. The device is further configured to perform linear motion decomposition processing on the driving latent code sample and the motion dictionary to be trained to obtain a driving latent path. The device is further configured to perform latent space navigation learning processing on the source latent code sample based on the driving latent path to determine the target latent motion representation.

[0138] Preferably, the image video generation device based on latent space navigation learning provided by the application is further configured to extract a multi-scale source feature of the source image sample. input the multi-scale source feature and the target latent motion representation into a generator, and perform decoding processing on the multi-scale source feature in the generator by using the target latent motion representation, to output the latent learning image sample.

[0139] Preferably, the image and video generation device based on latent space navigation learning provided by the application is specifically further used for the generator comprising at least a flow field generator and a refinement network. The inputting of the multi-scale source feature and the target latent motion representation into the generator and the performing of the decoding processing on the multi-scale source feature in the generator by using the target latent motion representation to output the latent learning image sample comprises: In each layer residual network of the flow field generator, the target latent motion representation is decoded layer by layer to generate a flow field pyramid and a repair mask; wherein the flow field pyramid represents a displacement vector field of the source image sample to the driven image sample, and the repair mask represents a feature map for repairing the distorted multi-scale source feature. The flow field pyramid is used to perform a flow field distortion operation on the multi-scale source feature to obtain a distorted feature map. The repair mask is used to perform a repair operation on the distorted feature map to obtain a repaired feature map. The repaired feature map is input into the refinement network to output the latent learning image sample.

[0140] Preferably, the image and video generation device based on latent space navigation learning provided by the application is specifically further used for performing up-sampling processing on the repaired feature map of each layer in the refinement network to obtain an up-sampled image of each layer. The repaired feature map of the current layer and the up-sampled image of the previous layer are spliced and fused to obtain an image sample output by the current layer. The image sample output by the last layer of the refinement network is taken as the latent learning image sample output by the refinement network.

[0141] The application provides an image video generation method and device based on latent space navigation learning, equipment, medium and program product. The method comprises the following steps: obtaining an initial image and a driving video; wherein the initial image represents a static image containing a first face, and the driving video represents a dynamic video containing a second face, and the second face has multiple face action information; inputting the initial image and the driving video into a trained image video generation model to output a target image video; wherein the target image video represents a dynamic video containing the first face, and the first face has multiple face action information of the second face; and the image video generation model is trained based on a driving video sample. The method can solve the defects of high cost, weak generalization ability and limited freedom in the prior art, realize the learning of the motion mode of the driving video by using the trained image video generation model without labeling face image key points, generate a target image video, the model has high generalization ability, can flexibly capture nonlinear motion, and breaks through the traditional freedom limitation.

[0142] Figure 4 An example of an entity structure diagram of an electronic device is shown in Figure 4 The electronic device can include a processor 410, a communications interface 420, a memory 430, and a communications bus 440, wherein the processor 410, the communications interface 420, and the memory 430 can communicate with each other through the communications bus 440. The processor 410 can call the logic instructions in the memory 430 to execute the image video generation method based on latent space navigation learning, which comprises the following steps: obtaining an initial image and a driving video; wherein the initial image represents a static image containing a first face, and the driving video represents a dynamic video containing a second face, and the second face has multiple face action information; inputting the initial image and the driving video into a trained image video generation model to output a target image video; wherein the target image video represents a dynamic video containing the first face, and the first face has multiple face action information of the second face; and the image video generation model is trained based on a driving video sample.

[0143] Further, the logic instructions in the memory 430 described above can be implemented in the form of software functional units and sold or used as standalone products, which can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the parts that make contributions to the prior art, or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0144] In another aspect, the present application also provides a computer program product, which comprises a computer program, the computer program can be stored on a non-transitory computer readable storage medium, and the computer program can be executed by a processor to enable a computer to execute the image video generation method based on latent space navigation learning provided by the above-mentioned methods. The method comprises: obtaining an initial image and a driving video; wherein the initial image represents a static image containing a first face, and the driving video represents a dynamic video containing a second face, and the second face has multiple facial action information; inputting the initial image and the driving video into a trained image video generation model to output a target image video; wherein the target image video represents a dynamic video containing the first face, and the first face has the multiple facial action information of the second face; and the image video generation model is trained based on a driving video sample.

[0145] In another aspect, the present application also provides a non-transitory computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the image video generation method based on latent space navigation learning provided by the above-mentioned methods. The method comprises: obtaining an initial image and a driving video; wherein the initial image represents a static image containing a first face, and the driving video represents a dynamic video containing a second face, and the second face has multiple facial action information; inputting the initial image and the driving video into a trained image video generation model to output a target image video; wherein the target image video represents a dynamic video containing the first face, and the first face has the multiple facial action information of the second face; and the image video generation model is trained based on a driving video sample.

[0146] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed to multiple network units. Part or all of the modules can be selected to achieve the purposes of the embodiments according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0147] Through the description of the above embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software and the necessary general hardware platform, and of course can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0148] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for image and video generation based on latent space navigation learning, characterized in that: include: Acquire an initial image and a driving video; wherein the initial image is characterized as a static image containing a first face, and the driving video is characterized as a dynamic video containing a second face, wherein the second face has multiple facial motion information; The initial image and the driving video are input into a trained image-video generation model to output a target image-video; wherein the target image-video is represented as a dynamic video containing the first face, and the first face has multiple facial motion information of the second face; the image-video generation model is obtained by training based on the driving video sample.

2. The image and video generation method based on latent space navigation learning according to claim 1, characterized in that: The step of training the image and video generation model based on the driving video sample includes: Sampling two frames from the driving video sample as a source image sample and a driving image sample respectively; Performing linear motion decomposition processing based on the driving image samples and the source image samples to determine a target latent motion representation; wherein the target latent motion representation is characterized by the motion transformation of the source image samples to the driving image samples; Decoding the source image sample and the target latent motion representation to determine a potential learning image sample; generating a target driving image sample based on the potential learning image sample; The image and video generation model is trained based on the total loss function between the target driving image sample and the driving image sample to obtain the trained image and video generation model.

3. The image and video generation method based on latent space navigation learning according to claim 2, characterized in that: The performing linear motion decomposition processing based on the driving image samples and the source image samples to determine the target potential motion representation includes: Defining reference image samples, and encoding the driving image samples based on the reference image samples to obtain driving latent code samples, and encoding the source image samples based on the reference image samples to obtain source latent code samples; wherein the driving latent code samples are characterized by the motion transformation of the reference image samples learned to the driving image samples, and the source latent code samples are characterized by the motion transformation of the source image samples learned to the reference image samples; Obtaining a motion dictionary to be trained; wherein the motion dictionary is represented by all paths in the latent space; Performing linear motion decomposition processing using the driving latent code sample and the motion dictionary to be trained to obtain a driving latent path; The target latent motion representation is determined by learning to navigate the source latent code samples in the latent space based on the driving latent path.

4. The image and video generation method based on latent space navigation learning according to claim 2, characterized in that: The performing decoding processing based on the source image sample and the target potential motion representation to determine the potential learning image sample includes: extracting multi-scale source features of the source image samples; The multi-scale source features and the target potential motion representation are input into a generator, so that the multi-scale source features are decoded by using the target potential motion representation in the generator, and the potential learning image sample is output.

5. The image and video generation method based on latent space navigation learning according to claim 4 is characterized in that: The generator includes at least a flow field generator and a refinement network; The step of inputting the multi-scale source features and the target potential motion representation into a generator, decoding the multi-scale source features using the target potential motion representation in the generator, and outputting the potential learning image sample comprises: In each layer of the residual network of the flow field generator, the target potential motion representation is decoded layer by layer to generate a flow field pyramid and a repair mask; wherein the flow field pyramid is represented as a displacement vector field from the source image sample to the driving image sample, and the repair mask is used to repair the distorted multi-scale source features; Using the flow field pyramid to perform a flow field distortion operation on the multi-scale source feature to obtain a distortion feature map; Repairing the distorted feature map using the repair mask to obtain a repaired feature map; The repair feature map is input into the refinement network, and the potential learning image sample is output.

6. The image and video generation method based on latent space navigation learning according to claim 5, characterized in that: Inputting the repair feature map into the refinement network and outputting the potential learning image sample comprises: In each layer of the refinement network, upsampling is performed on the repair feature map of each layer to obtain an upsampled image of each layer; Performing splicing and fusing processing on the repair feature map of the current layer and the upsampled image of the previous layer to obtain an image sample output by the current layer; The image sample output by the last layer of the refinement network is used as the potential learning image sample output by the refinement network.

7. An image and video generation device based on latent space navigation learning, characterized in that: include: An acquisition module, configured to acquire an initial image and a driving video; wherein the initial image is characterized as a static image containing a first face, and the driving video is characterized as a dynamic video containing a second face, wherein the second face has multiple facial motion information; The target image video output module is used to input the initial image and the driving video into a trained image video generation model to output a target image video; wherein, the target image video is represented as a dynamic video containing the first face, and the first face has multiple facial motion information of the second face; the image video generation model is obtained by training based on the driving video sample.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the program, the image and video generation method based on latent space navigation learning as described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image and video generation method based on latent space navigation learning as described in any one of claims 1 to 6 is implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the image and video generation method based on latent space navigation learning as described in any one of claims 1 to 6 is implemented.

Images