Image generation method and device, electronic equipment and storage medium

Abstract

This invention provides an image generation method, apparatus, device, and medium. The method includes: determining a model image of a target object in a target image, and reference feature data of a reference image, wherein the model image represents the motion state of the target object in three-dimensional space. Motion feature data is extracted from the model image. Based on the reference feature data and the motion feature data, a result image is generated whereby the motion of the target object is controlled by the target image. The motion state of the target object in three-dimensional space is learned from the model image, thereby generating an image consistent with the motion state of the target object. This allows for more natural and realistic images generated even when the target object is moving over a wide range of motion, thus improving performance in a wider range of application scenarios. When applied to portrait editing, motion states such as shape, posture, expression, and camera viewpoint can be learned from the model image of the face, improving the image quality when handling large-scale head movements.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to an image generation method, an image generation apparatus, an electronic device, and a readable storage medium. Background Technology

[0002] In the field of image editing technology, portrait editing is a particularly challenging task, especially when it involves large-scale head movements. Currently, although existing image editing techniques have made some progress in processing static images or images with minor movements, significant technical challenges remain when dealing with large-scale head movements.

[0003] Existing portrait editing technologies face challenges in handling large-scale head movements, limiting their performance in a wider range of applications and preventing the creation of more natural and realistic portrait edits. Summary of the Invention

[0004] The technical problem to be solved by the embodiments of the present invention is to provide an image generation method, apparatus, electronic device, and readable storage medium, so as to solve the problem that existing portrait editing technology faces challenges in handling large-scale head movements, which limits its performance in a wider range of application scenarios and makes it impossible to achieve more natural and realistic portrait editing.

[0005] To address the above problems, the present invention provides an image generation method, the method comprising:

[0006] A model image of the target object in the target image and reference feature data corresponding to the reference image are determined; wherein, the model image represents the motion state of the target object in three-dimensional space;

[0007] Motion feature data are extracted from the model image;

[0008] Based on the reference feature data and the motion feature data, a result image is generated in which the motion of the target object is controlled by the target image.

[0009] Optionally, extracting motion feature data from the model image includes:

[0010] The model image is input into a feature extraction network constructed using n stacked residual blocks to obtain motion feature data output by each residual block.

[0011] Optionally, at least one of the residual blocks has a self-attention layer.

[0012] Optionally, the feature extraction network is a U-shaped network, and the residual blocks in the feature extraction network include a downsampling convolutional layer.

[0013] Optionally, generating a result image of the target object's motion controlled by the target image based on the reference feature data and the motion feature data includes:

[0014] Based on the reference feature data and the motion feature data, the denoising network is controlled to generate denoised feature data;

[0015] The denoised feature data is decoded to obtain the resulting image.

[0016] Optionally, the motion feature data includes motion feature data output by n residual blocks in the feature extraction network, the denoising network includes n denoising spatial attention layers and n denoising cross attention layers, the reference feature data includes spatial reference feature data output by n reference spatial attention layers and cross reference feature data output by n reference cross attention layers in the reference network, where n is a positive integer, the denoising feature data includes n spatial denoising feature data and n cross denoising feature data, and the step of controlling the denoising network to generate denoising feature data based on the reference feature data and the motion feature data includes:

[0017] The spatial reference feature data output from the first reference spatial attention layer is input into the first denoising spatial attention layer to obtain the output first spatial denoising feature data;

[0018] The motion feature data output from the first residual block and the first spatial denoising feature data are fused to obtain the first fused feature data;

[0019] The first fused feature data and the cross-reference feature data output by the first reference cross-attention layer are input into the first denoising cross-attention layer to obtain the output first cross-denoising feature data;

[0020] Starting with x equal to 2, where x is an integer between 2 and n, perform the following steps until x equals n:

[0021] Input the spatial reference feature data output from the xth reference spatial attention layer and the (x-1)th spatial denoising feature data into the xth denoising spatial attention layer to obtain the xth spatial denoising feature data output.

[0022] The motion feature data output from the xth residual block and the xth spatial denoising feature data are fused to obtain the xth fused feature data;

[0023] The xth fused feature data, the cross-reference feature data output by the xth reference cross-attention layer, and the (x-1)th cross-denoising feature data are input into the xth denoising cross-attention layer to obtain the xth output cross-denoising feature data.

[0024] Optionally, the tensor shape of the motion feature data and the tensor shape of the corresponding spatial denoising feature data are corresponding.

[0025] Optionally, determining the model image of the target object in the target image includes:

[0026] Extract model information describing the motion state of the target object in three-dimensional space from the target image;

[0027] The model image is rendered based on the model information.

[0028] The present invention also provides an image generation apparatus, the apparatus comprising:

[0029] The data determination module is used to determine the model image of the target object in the target image, and the reference feature data corresponding to the reference image; wherein, the model image represents the motion state of the target object in three-dimensional space;

[0030] The feature extraction module is used to extract motion feature data from the model image;

[0031] An image generation module is used to generate a result image in which the motion of the target object is controlled by the target image, based on the reference feature data and the motion feature data.

[0032] Optionally, the feature extraction module includes:

[0033] The feature extraction submodule is used to input the model image into a feature extraction network constructed by stacking n residual blocks, and obtain motion feature data output by each residual block.

[0034] Optionally, at least one of the residual blocks has a self-attention layer.

[0035] Optionally, the feature extraction network is a U-shaped network, and the residual blocks in the feature extraction network include a downsampling convolutional layer.

[0036] Optionally, the image generation module includes:

[0037] The feature data generation submodule is used to control the denoising network to generate denoised feature data based on the reference feature data and the motion feature data.

[0038] The decoding submodule is used to decode the denoised feature data to obtain the result image.

[0039] Optionally, the motion feature data includes motion feature data output by n residual blocks in the feature extraction network, the denoising network includes n denoising spatial attention layers and n denoising cross attention layers, the reference feature data includes spatial reference feature data output by n reference spatial attention layers and cross reference feature data output by n reference cross attention layers in the reference network, where n is a positive integer, the denoising feature data includes n spatial denoising feature data and n cross denoising feature data, and the feature data generation submodule includes:

[0040] The first denoising unit is used to input the spatial reference feature data output by the first reference spatial attention layer into the first denoising spatial attention layer to obtain the output first spatial denoising feature data.

[0041] The first fusion unit is used to fuse the motion feature data output by the first residual block and the first spatial denoising feature data to obtain the first fused feature data;

[0042] The second denoising unit is used to input the first fused feature data and the cross-reference feature data output by the first reference cross-attention layer into the first denoising cross-attention layer to obtain the output first cross-denoising feature data.

[0043] Starting with x equal to 2, where x is an integer between 2 and n, perform the following steps until x equals n:

[0044] The third denoising unit is used to input the spatial reference feature data output by the xth reference spatial attention layer and the (x-1)th spatial denoising feature data into the xth denoising spatial attention layer to obtain the output xth spatial denoising feature data.

[0045] The second fusion unit is used to fuse the motion feature data output by the xth residual block and the xth spatial denoising feature data to obtain the xth fused feature data;

[0046] The fourth denoising unit is used to input the xth fused feature data, the cross-reference feature data output by the xth reference cross-attention layer, and the (x-1)th cross-denoising feature data into the xth denoising cross-attention layer to obtain the output xth cross-denoising feature data.

[0047] Optionally, the tensor shape of the motion feature data and the tensor shape of the corresponding spatial denoising feature data are corresponding.

[0048] Optionally, the data determination module includes:

[0049] The information extraction submodule is used to extract model information describing the motion state of the target object in three-dimensional space from the target image;

[0050] The image rendering submodule is used to render the model image based on the model information.

[0051] This invention also discloses an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0052] Memory, used to store computer programs;

[0053] The processor, when executing a program stored in memory, implements the method described above.

[0054] This invention also discloses a readable storage medium that, when the instructions in the storage medium are executed by the processor of an electronic device, enables the electronic device to perform the method described above.

[0055] According to embodiments of the present invention, firstly, a model image of the target object in the target image and reference feature data corresponding to the reference image are determined, wherein the model image represents the motion state of the target object in three-dimensional space. Next, motion feature data is extracted from the model image. Based on the reference feature data and the motion feature data, a result image is generated that controls the motion of the target object from the target image. Through this method, the motion state of the target object in three-dimensional space is learned from the model image, thereby producing a result image consistent with the motion state of the target object. Even when the target object is moving over a wide range, the generated image achieves a more natural and realistic effect, thus improving performance in a wider range of application scenarios. When applied to portrait editing, motion states such as shape, posture, expression, and camera viewpoint can be learned from the model image of the face, improving the image quality when handling large-scale head movements. Attached Figure Description

[0056] Figure 1 A flowchart illustrating the steps of an image generation method according to an embodiment of the present invention is shown.

[0057] Figure 2 An image example of the image generation method is shown;

[0058] Figure 3 A flowchart illustrating the steps of an image generation method according to an embodiment of the present invention is shown.

[0059] Figure 4 A schematic diagram of an extravehicular question-and-answer system based on a multimodal large model is shown.

[0060] Figure 5 This diagram illustrates a structural block diagram of an image generation apparatus embodiment provided by one embodiment of the present invention;

[0061] Figure 6 A structural block diagram of an electronic device for image generation is shown according to an exemplary embodiment. Detailed Implementation

[0062] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0063] The image generation method in this embodiment of the invention can be executed by electronic devices such as smart vehicle systems and mobile terminals. For example, the image generation method can be applied to smart vehicle systems or any other applicable hardware and software. This embodiment of the invention does not limit this application.

[0064] Reference Figure 1 The diagram illustrates a flowchart of an image generation method provided by an embodiment of the present invention, which may specifically include the following steps:

[0065] Step 101: Determine the model image of the target object in the target image, and the reference feature data corresponding to the reference image; wherein, the model image represents the motion state of the target object in three-dimensional space.

[0066] In this embodiment of the invention, the target image refers to an image that guides or controls the movement of a target object during image generation. The target object refers to any object in the target image that is capable of movement, such as a face, a body, or any other applicable object; this embodiment of the invention does not impose any limitations on this.

[0067] In this embodiment of the invention, the model image is an image that reflects a model of the target object.

[0068] In this embodiment of the invention, the specific implementation of determining the model image of the target object in the target image can include various methods, and this embodiment of the invention does not limit this. The core principle is: based on the target image, a three-dimensional model of the target object is constructed, and then based on the three-dimensional target object model, a two-dimensional model image reflecting the model of the target object is generated. This model image can characterize the motion state of the target object in three-dimensional space. For example, a model image of a human face can characterize motion states such as shape, posture, expression, and camera viewpoint.

[0069] For example, a pre-trained model can be used to extract shape, pose, expression, camera, and lighting information from the target image. Then, based on this information, a model image can be rendered.

[0070] In one optional embodiment of the present invention, a specific implementation of determining the model image of the target object in the target image may include: extracting model information describing the motion state of the target object in three-dimensional space from the target image; and rendering the model image based on the model information.

[0071] Model information refers to information describing the motion state of a target object in three-dimensional space, such as shape information, pose information, expression information, camera information, and lighting information.

[0072] First, model information describing the motion state of the target object in three-dimensional space is extracted from the target image. Then, based on this model information, rendering is performed to obtain a model image.

[0073] For example, a DECA (Detailed Expression Capture and Animation, a method for 3D face reconstruction based on a single color image) pre-trained model was used. This model can extract the shape, pose, expression, camera parameters, and lighting information of the face. During rendering, the lighting information is replaced with preset values, while all other extracted information is retained. This allows for the rendering of motion conditions that maintain consistency with the face's motion state; these motion conditions are essentially a model image. The rendering formula is as follows:

[0074]

[0075] in, This represents the motion conditions (i.e., model image) calculated when the input is the i-th reference image and the j-th target image, where M represents motion and P represents the model image. k ∈R H×W×3 These are the 3D motion conditions for rendering. A represents a fixed albedo, such as values ​​(0.7, 0.7, 0.7); β′, θ′, and ψ′ are the facial pose, shape, and expression of the target image; c′∈R 1+2 Indicates the parameters of an orthogonal camera; I K ∈R 3 These are manually set spherical harmonic lighting parameters designed to ensure consistent lighting across all 3D motion-guided applications; the spherical harmonic basis is defined as H... k ∈R 3 .

[0076] The rendered motion conditions are normalized to the range [-1, 1], and their tensor shape is 3×512×512. This represents a three-dimensional array; for example, it can represent image data with three channels, where each channel is a 512×512 matrix.

[0077] In this embodiment of the invention, a reference image refers to an image that guides or controls other aspects of image generation, depending on the specific needs of image generation, and is used to provide additional information to assist in image generation. For example, in some advanced denoising tasks, such as style transfer denoising, a reference image may refer to an image with a specific style or texture, used to guide the restoration of style or texture during the denoising process.

[0078] In this embodiment of the invention, the reference feature data is feature data extracted from a reference image. Specifically, the reference image can be processed by an encoder and a ReferenceNet to extract the reference feature data. The core principle is: encoding the reference image captures the visual appearance of the object (such as a portrait) and its associated background; the reference image is used to control the visual appearance of the generated image.

[0079] For example, an encoder can use a VAE Encoder (Variational Autoencoder Encoder). A reference network is a technique for controlling a generative model; it is designed to guide the generative model to create the desired output. This technique controls the generation process by integrating an additional UNet (U-shaped network) architecture network—ReferenceNet. When generating an image, the reference image is first encoded, and then these encoded features are fed into ReferenceNet. Subsequently, these features are connected to different layers of the main UNet network, thereby precisely guiding and controlling the final generated image. This method is particularly suitable for scenarios that require generation based on a specific image or style. For example, in video generation, it can be used to ensure that the generated video frames are consistent with a specific visual style or content; in clothing-changing applications, it can generate corresponding clothing effects based on clothing styles in reference images; and in face-swapping applications, it can generate new facial images based on facial features in reference images. In this way, reference networks significantly improve the application potential and flexibility of generative models in diverse scenarios.

[0080] Step 102: Extract motion feature data from the model image.

[0081] In this embodiment of the invention, the model image is an image that reflects the model of the target object. The model image ignores other information in the target image but retains information related to motion. Therefore, the feature data extracted from the model image only reflects the motion information of the target object. Thus, the feature data extracted from the model image is denoted as motion feature data.

[0082] In this embodiment of the invention, the specific implementation methods for extracting motion feature data from model images can include various approaches, and this embodiment does not limit these approaches. The core principle is to introduce a dedicated motion module for encoding and processing the motion information of the target object. The purpose of this module is to improve the quality and realism of the generated image by accurately capturing and encoding a wide range of motion of the target object.

[0083] In this embodiment of the invention, since motion feature data needs to be used in conjunction with reference feature data for image generation, the extraction of motion feature data and the design of tensor shape depend on other parts of image generation, mainly on the extraction of feature data and tensor shape that need to be fused with motion feature data.

[0084] In an optional embodiment of the present invention, a specific implementation of extracting motion feature data from the model image may include: inputting the model image into a feature extraction network constructed using n stacked residual blocks to obtain motion feature data output by each residual block.

[0085] The aforementioned motion module can be a feature extraction network. This feature extraction network is constructed using n stacked residual blocks. For example, a feature extraction network with a modified U-Net architecture constructed using 6 stacked residual blocks.

[0086] Residual blocks are used to address the vanishing and exploding gradient problems in deep networks. The core idea of ​​residual blocks is to pass the input through one or more convolutional layers and then directly add it to the input (skip connections), thus forming a residual mapping. This design allows the network to learn the residual function, i.e., the difference between the input and output, rather than directly learning the unprocessed mapping.

[0087] After the model image is input into this feature extraction network, it passes through a residual block, which outputs a motion feature data. n residual blocks will output n motion feature data.

[0088] In an alternative embodiment of the invention, at least one of the residual blocks has a self-attention layer.

[0089] At least one of the n residual blocks contains a self-attention layer. Specifically, it can be designed so that any residual block is followed by a self-attention layer. A self-attention layer is a mechanism in deep learning that allows the model to dynamically adjust its attention to each element when processing sequential data by calculating the correlation scores between elements at different positions in the sequence. This captures complex dependencies within the sequence, leading to better performance from extracted motion feature data. Experimental tuning and design of a suitable self-attention layer are necessary to improve image generation quality and reduce computational complexity.

[0090] For example, the first four blocks consist of residual blocks, and the last two blocks contain a self-attention layer after the residual blocks.

[0091] In an optional embodiment of the present invention, the feature extraction network is a U-shaped network, and the residual blocks in the feature extraction network include a downsampling convolutional layer.

[0092] The feature extraction network is U-Net (U-shaped network), which works based on U-Net, but only includes a downsampling module. That is, each residual block is followed by a downsampling convolutional layer, meaning that a downsampling convolutional layer is needed between two residual blocks.

[0093] Step 103: Based on the reference feature data and the motion feature data, generate a result image in which the motion of the target object is controlled by the target image.

[0094] In this embodiment of the invention, an image generation process is controlled based on reference feature data and motion feature data to generate a result image. This result image is an image in which the motion of the target object is controlled by the target image. Simultaneously, this result image is an image in which the visual appearance is controlled by the reference image.

[0095] In this embodiment of the invention, reference feature data and motion feature data are combined to consider both visual and motion information in the subsequent generation process. This step may involve feature alignment, feature weighting, or the use of attention mechanisms to emphasize important features. Specifically, a preliminary result image is generated by combining the reference feature data and motion feature data. Depending on the needs, multiple iterations or a multi-stage generation strategy may be required to progressively optimize the result image.

[0096] In an optional embodiment of the present invention, a specific implementation of generating a result image of the target object controlled by the target image based on the reference feature data and the motion feature data may include: controlling a denoising network to generate denoised feature data based on the reference feature data and the motion feature data; and decoding the denoised feature data to obtain the result image.

[0097] A denoising network is a specially designed deep learning model for image denoising tasks. This network learns the relationship between noise patterns and image content, effectively removing noise from images while preserving detail and texture information. For example, a denoising network is the Denoising U-Net.

[0098] Denoising networks can use reference feature data, motion feature data, and random noise as inputs. By learning the noise distribution and simulating the inverse process, they can gradually reduce noise and recover the denoising feature data, which is called denoised feature data.

[0099] Then the denoised feature data is decoded to obtain the result image.

[0100] In an optional embodiment of the present invention, the motion feature data includes motion feature data output by n residual blocks in the feature extraction network, the denoising network includes n denoising spatial attention layers and n denoising cross attention layers, the reference feature data includes spatial reference feature data output by n reference spatial attention layers and cross reference feature data output by n reference cross attention layers in the reference network, where n is a positive integer, the denoising feature data includes n spatial denoising feature data and n cross denoising feature data, and a specific implementation of controlling the denoising network to generate denoising feature data based on the reference feature data and the motion feature data may include: inputting the spatial reference feature data output by the first reference spatial attention layer into the first denoising spatial attention layer to obtain the output first spatial denoising feature data; and processing the motion feature data output by the first residual block and the first spatial denoising feature data... The process involves fusing the data to obtain the first fused feature data. The first fused feature data and the cross-reference feature data output from the first reference cross-attention layer are then input into the first denoising cross-attention layer to obtain the first output cross-denoising feature data. Starting from x equal to 2, where x is an integer between 2 and n, the following steps are performed until x equals n: The spatial reference feature data output from the xth reference spatial attention layer and the (x-1)th spatial denoising feature data are input into the xth denoising spatial attention layer to obtain the xth output spatial denoising feature data. The motion feature data output from the xth residual block and the xth spatial denoising feature data are fused to obtain the xth fused feature data. The xth fused feature data, the cross-reference feature data output from the xth reference cross-attention layer, and the (x-1)th cross-denoising feature data are input into the xth denoising cross-attention layer to obtain the xth output cross-denoising feature data.

[0101] In both the reference network and the denoising network, the spatial attention layer and the cross attention layer are two important components that help improve the denoising effect and accuracy.

[0102] The denoising network consists of n spatial attention layers and n cross-attention layers. The output denoised feature data includes spatial denoised feature data and cross-denoised feature data. The spatial denoised feature data is output by the spatial attention layers, and the cross-denoised feature data is output by the cross-attention layers.

[0103] The reference network consists of n spatial reference attention layers and n cross-reference attention layers. The output reference feature data includes spatial reference feature data and cross-reference feature data. Spatial reference feature data is output by the spatial reference attention layers, and cross-reference feature data is output by the cross-reference attention layers.

[0104] The spatial attention layer allows the network to focus on the most important regions of an image, emphasizing detailed features and improving segmentation performance. It analyzes image content and assigns different attention weights to different regions, enabling the network to pay closer attention to key parts of the image, such as edges and textures. This mechanism helps the network focus more on the details of the image during denoising, thereby improving the denoising effect.

[0105] Cross-attention layers are typically added between the encoder and decoder of U-Net, enabling the network to control and adjust the output based on provided textual descriptions. Cross-attention layers can capture long-distance dependencies between features, addressing the semantic gap between encoder and decoder features.

[0106] The spatial reference feature data output from the first reference spatial attention layer is input into the first denoising spatial attention layer. The first denoising spatial attention layer processes the input spatial reference feature data and outputs the first denoised spatial feature data.

[0107] In the denoising network, the motion feature data output by each residual block is multiplied element-wise with the spatial denoised feature data output by the corresponding denoised spatial attention layer of the denoising network. This is a feature fusion method that can enhance the network's ability to capture motion information. Other applicable fusion methods can also be used, and this embodiment of the invention does not limit this.

[0108] By fusing the first motion feature data and the first spatial denoising feature data, the first fused feature data can be obtained.

[0109] In this way, the network can more effectively utilize motion features to guide the denoising process, thereby generating more accurate and realistic denoised images. This element-wise multiplication operation is actually performed at the feature level, allowing the network to independently adjust the feature weights at each spatial location, thus better integrating information from different sources.

[0110] Then the first fused feature data is input into the first denoising cross-attention layer, and the first denoising cross-attention layer can output the first cross-denoising feature data based on the first fused feature data.

[0111] Starting from x=2, process the subsequent feature data step by step until x=n. The processing procedure for each step is as follows:

[0112] The spatial reference feature data output from the x-th reference spatial attention layer and the (x-1)-th spatial denoising feature data are input into the x-th denoising spatial attention layer.

[0113] The x-th spatial attention layer processes the input feature data and outputs the x-th spatial denoised feature data.

[0114] The motion feature data output from the xth residual block is fused with the xth spatial denoising feature data to obtain the xth fused feature data.

[0115] The xth fused feature data, the cross-reference feature data output by the xth reference cross-attention layer, and the (x-1)th cross-denoising feature data are input into the xth denoising cross-attention layer.

[0116] The x-th denoising cross-attention layer processes the input feature data and outputs the x-th cross-denoised feature data.

[0117] The final spatial denoising feature data and cross-denoising feature data guide the decoding process, thereby generating a denoised image, which is the result image.

[0118] For example, this invention proposes integrating motion feature data with spatially denoised feature data output from the spatial attention layer of a denoising network. The integration mechanism can be represented as follows:

[0119] s′ i =s i ⊙tanh(p i ),

[0120] The motion feature data output before each downsampling convolution in the U-Net for extracting motion feature data is called p. i In Denoising U-Net, the output (spatial denoising feature data) of each spatial attention layer is called s. iIn Denoising U-Net, this invention proposes using s′ i Replace the outputs of each spatial attention layer i and s′ i The input is fed into a cross-attention layer. This method achieves pixel-level control, improving motion generation and maintaining stability even during large-scale motion.

[0121] In an optional embodiment of the present invention, the tensor shape of the motion feature data and the tensor shape of the corresponding spatial denoising feature data are corresponding.

[0122] In order for motion feature data and spatial denoising feature data to be fused, it is necessary to have appropriate tensor shapes for the motion feature data and the corresponding tensor shapes for the spatial denoising feature data. That is, the two are corresponding, and the tensor shape of the motion feature data depends on the tensor shape of the corresponding spatial denoising feature data.

[0123] For example, the motion representations (i.e., motion feature data) output from 6 blocks (4 residual blocks and 2 residual blocks with self-attention layers) have the same tensor shape for every two blocks. The tensor shapes for every two blocks are 4096×320, 1024×640, and 256×1280, respectively.

[0124] According to embodiments of the present invention, firstly, a model image of the target object in the target image and reference feature data corresponding to the reference image are determined, wherein the model image represents the motion state of the target object in three-dimensional space. Next, motion feature data is extracted from the model image. Based on the reference feature data and the motion feature data, a result image is generated that controls the motion of the target object from the target image. Through this method, the motion state of the target object in three-dimensional space is learned from the model image, thereby producing a result image consistent with the motion state of the target object. Even when the target object is moving over a wide range, the generated image achieves a more natural and realistic effect, thus improving performance in a wider range of application scenarios. When applied to portrait editing, motion states such as shape, posture, expression, and camera viewpoint can be learned from the model image of the face, improving the image quality when handling large-scale head movements.

[0125] Specifically, the accuracy of pose estimation is a key challenge. Existing pose estimation techniques perform reasonably well when dealing with small head movements, but their accuracy drops significantly when dealing with large-scale movements. This is mainly because head movements involve complex joint and muscle movements, and existing models struggle to capture these subtle changes, resulting in unnatural or distorted poses in the generated images.

[0126] This invention learns the motion state of a target object in three-dimensional space through model images, and can generate large-scale motion consistent with the target image, especially maintaining good detail in areas such as hair and ears. Figure 2 The image shown is an example of Head Motion.

[0127] Secondly, the consistency issue of texture mapping becomes particularly prominent during large-scale head movements. Although current texture mapping techniques perform well in smooth areas, texture mapping may become distorted or inconsistent during complex head movements, such as changes in facial expressions or head rotation, further affecting the realism of the image.

[0128] This invention learns detailed motion states such as facial expressions from model images, enabling it to maintain facial texture without distortion and control facial expressions while performing large-scale movements. Figure 2 The image shown is an example of Facial Animation.

[0129] The challenges of depth estimation should not be ignored. In static scenes, depth estimation techniques generally perform well, but in dynamic scenes, especially with head movement, the accuracy of depth estimation is affected, which may lead to incorrect relative positions of people and backgrounds in the generated image.

[0130] This invention learns a wide range of motion states, such as camera viewpoint, from model images, effectively preserving facial features and accurately representing the relative positional relationship between the person and the background. Figure 2 The image shown is an example of Camera Motion.

[0131] Reference Figure 3 The diagram illustrates a flowchart of an image generation method provided by an embodiment of the present invention, which may specifically include the following steps:

[0132] Step 201: Extract model information describing the motion state of the target object in three-dimensional space from the target image.

[0133] Step 202: Render the model image based on the model information.

[0134] Step 203: Determine the reference feature data corresponding to the reference image.

[0135] Step 204: Input the model image into a feature extraction network constructed using n stacked residual blocks to obtain motion feature data output by each residual block.

[0136] Step 205: Based on the reference feature data and the motion feature data, control the denoising network to generate denoised feature data.

[0137] Step 206: Decode the denoised feature data to obtain the result image.

[0138] For example, such as Figure 4 The diagram illustrates the image generation principle. The Reference Frame obtains spatial reference feature data and cross-reference feature data through VAE Encoder and ReferenceNet. The Target Image is processed by Motion-aware Render to obtain 3D Motion Guidance, i.e., the model image. The 3D Motion Guidance is then input into the Motion Module for feature extraction, yielding motion feature data. This motion feature data is then input into the Denoising U-Net. The motion feature data is fused with the spatial denoising feature data output from the spatial attention layer in the Denoising U-Net, and the fused feature data is input into the cross-attention layer to obtain cross-denoising feature data. The spatial denoising feature data and cross-denoising feature data are then decoded by the VAE Decoder to obtain the Denoised Image, i.e., the result image. During training, the Spatial Loss between the Target Image and the Denoised Image is calculated to measure the spatial similarity between the denoised image and the target image. Spatial loss is commonly used to guide the training process of denoising networks, ensuring that the generated image maintains spatial consistency with the target image. This loss function helps the network learn how to better preserve the structural information of the image, such as edges and textures, thereby improving the quality of the denoised image. In image denoising tasks, spatial loss can be combined with other loss functions (such as mean squared error, MSE) to optimize model weights and improve denoising performance.

[0139] According to embodiments of the present invention, model information describing the motion state of the target object in three-dimensional space is extracted from the target image. Based on the model information, a model image is rendered. Referring to the reference feature data corresponding to the image, the model image is input into a feature extraction network constructed using n stacked residual blocks to obtain motion feature data output by each residual block. Based on the reference feature data and the motion feature data, a denoising network is controlled to generate denoised feature data. The denoised feature data is then decoded to obtain the result image. This method learns the motion state of the target object in three-dimensional space from the model image, thereby producing a result image consistent with the motion state of the target object. Images generated when the target object is moving over a wide range of motion can achieve a more natural and realistic effect, thus improving performance in a wider range of application scenarios. When applied to portrait editing, motion states such as shape, posture, expression, and camera viewpoint can be learned from the model image of the face, improving the image quality when handling large-scale head movements.

[0140] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0141] Reference Figure 5 The diagram illustrates a structural block diagram of an image generation apparatus embodiment provided by another embodiment of the present invention, which may specifically include the following modules:

[0142] The data determination module 301 is used to determine the model image of the target object in the target image, and the reference feature data corresponding to the reference image; wherein, the model image represents the motion state of the target object in three-dimensional space;

[0143] The feature extraction module 302 is used to extract motion feature data from the model image;

[0144] The image generation module 303 is used to generate a result image in which the motion of the target object is controlled by the target image, based on the reference feature data and the motion feature data.

[0145] Optionally, the feature extraction module includes:

[0146] The feature extraction submodule is used to input the model image into a feature extraction network constructed by stacking n residual blocks, and obtain motion feature data output by each residual block.

[0147] Optionally, at least one of the residual blocks has a self-attention layer.

[0148] Optionally, the feature extraction network is a U-shaped network, and the residual blocks in the feature extraction network include a downsampling convolutional layer.

[0149] Optionally, the image generation module includes:

[0150] The feature data generation submodule is used to control the denoising network to generate denoised feature data based on the reference feature data and the motion feature data.

[0151] The decoding submodule is used to decode the denoised feature data to obtain the result image.

[0152] Optionally, the motion feature data includes motion feature data output by n residual blocks in the feature extraction network, the denoising network includes n denoising spatial attention layers and n denoising cross attention layers, the reference feature data includes spatial reference feature data output by n reference spatial attention layers and cross reference feature data output by n reference cross attention layers in the reference network, where n is a positive integer, the denoising feature data includes n spatial denoising feature data and n cross denoising feature data, and the feature data generation submodule includes:

[0153] The first denoising unit is used to input the spatial reference feature data output by the first reference spatial attention layer into the first denoising spatial attention layer to obtain the output first spatial denoising feature data.

[0154] The first fusion unit is used to fuse the motion feature data output by the first residual block and the first spatial denoising feature data to obtain the first fused feature data;

[0155] The second denoising unit is used to input the first fused feature data and the cross-reference feature data output by the first reference cross-attention layer into the first denoising cross-attention layer to obtain the output first cross-denoising feature data.

[0156] Starting with x equal to 2, where x is an integer between 2 and n, perform the following steps until x equals n:

[0157] The third denoising unit is used to input the spatial reference feature data output by the xth reference spatial attention layer and the (x-1)th spatial denoising feature data into the xth denoising spatial attention layer to obtain the output xth spatial denoising feature data.

[0158] The second fusion unit is used to fuse the motion feature data output by the xth residual block and the xth spatial denoising feature data to obtain the xth fused feature data;

[0159] The fourth denoising unit is used to input the xth fused feature data, the cross-reference feature data output by the xth reference cross-attention layer, and the (x-1)th cross-denoising feature data into the xth denoising cross-attention layer to obtain the output xth cross-denoising feature data.

[0160] Optionally, the tensor shape of the motion feature data and the tensor shape of the corresponding spatial denoising feature data are corresponding.

[0161] Optionally, the data determination module includes:

[0162] The information extraction submodule is used to extract model information describing the motion state of the target object in three-dimensional space from the target image;

[0163] The image rendering submodule is used to render the model image based on the model information.

[0164] According to embodiments of the present invention, firstly, a model image of the target object in the target image and reference feature data corresponding to the reference image are determined, wherein the model image represents the motion state of the target object in three-dimensional space. Next, motion feature data is extracted from the model image. Based on the reference feature data and the motion feature data, a result image is generated that controls the motion of the target object from the target image. Through this method, the motion state of the target object in three-dimensional space is learned from the model image, thereby producing a result image consistent with the motion state of the target object. Even when the target object is moving over a wide range, the generated image achieves a more natural and realistic effect, thus improving performance in a wider range of application scenarios. When applied to portrait editing, motion states such as shape, posture, expression, and camera viewpoint can be learned from the model image of the face, improving the image quality when handling large-scale head movements.

[0165] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0166] Figure 6 This is a structural block diagram illustrating an electronic device 700 for image generation according to an exemplary embodiment. For example, the electronic device 700 may be an in-vehicle computer, a computer, a digital broadcasting terminal, a messaging device, a game console, a medical device, a fitness device, etc.

[0167] Reference Figure 6The electronic device 700 may include one or more of the following components: a processing component 702, a memory 704, a power supply component 706, a multimedia component 708, an audio component 710, an input / output (I / O) interface 712, a sensor component 714, and a communication component 716.

[0168] Processing component 702 typically controls the overall operation of electronic device 700, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 702 may include one or more processors 720 to execute instructions to complete all or part of the steps of the methods described above. Furthermore, processing component 702 may include one or more modules to facilitate interaction between processing component 702 and other components. For example, processing component 702 may include a multimedia module to facilitate interaction between multimedia component 708 and processing component 702.

[0169] Memory 704 is configured to store various types of data to support the operation of device 700. Examples of this data include instructions for any application or method operating on electronic device 700, contact data, phonebook data, messages, pictures, videos, etc. Memory 704 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0170] Power supply component 706 provides power to various components of electronic device 700. Power supply component 706 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 700.

[0171] Multimedia component 708 includes a screen that provides an output interface between the electronic device 700 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 708 includes a front-facing camera and / or a rear-facing camera. When the electronic device 700 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0172] Audio component 710 is configured to output and / or input audio signals. For example, audio component 710 includes a microphone (MIC) configured to receive external audio signals when electronic device 700 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 704 or transmitted via communication component 716. In some embodiments, audio component 710 also includes a speaker for outputting audio signals.

[0173] I / O interface 712 provides an interface between processing component 702 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0174] Sensor assembly 714 includes one or more sensors for providing state assessments of various aspects of electronic device 700. For example, sensor assembly 714 may detect the on / off state of device 700, the relative positioning of components such as the display and keypad of electronic device 700, changes in position of electronic device 700 or a component of electronic device 700, the presence or absence of user contact with electronic device 700, orientation or acceleration / deceleration of electronic device 700, and temperature changes of electronic device 700. Sensor assembly 714 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 714 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 714 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0175] Communication component 716 is configured to facilitate wired or wireless communication between electronic device 700 and other devices. Electronic device 700 can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 716 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 716 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0176] In an exemplary embodiment, the electronic device 700 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0177] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 704 including instructions, which can be executed by a processor 720 of an electronic device 700 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0178] A non-transitory computer-readable storage medium, when instructions in the storage medium are executed by a terminal's processor, enables the terminal to perform an image generation method, the method comprising:

[0179] A model image of the target object in the target image and reference feature data corresponding to the reference image are determined; wherein, the model image represents the motion state of the target object in three-dimensional space;

[0180] Motion feature data are extracted from the model image;

[0181] Based on the reference feature data and the motion feature data, a result image is generated in which the motion of the target object is controlled by the target image.

[0182] Optionally, extracting motion feature data from the model image includes:

[0183] The model image is input into a feature extraction network constructed using n stacked residual blocks to obtain motion feature data output by each residual block.

[0184] Optionally, at least one of the residual blocks has a self-attention layer.

[0185] Optionally, the feature extraction network is a U-shaped network, and the residual blocks in the feature extraction network include a downsampling convolutional layer.

[0186] Optionally, generating a result image of the target object's motion controlled by the target image based on the reference feature data and the motion feature data includes:

[0187] Based on the reference feature data and the motion feature data, the denoising network is controlled to generate denoised feature data;

[0188] The denoised feature data is decoded to obtain the resulting image.

[0189] Optionally, the motion feature data includes motion feature data output by n residual blocks in the feature extraction network, the denoising network includes n denoising spatial attention layers and n denoising cross attention layers, the reference feature data includes spatial reference feature data output by n reference spatial attention layers and cross reference feature data output by n reference cross attention layers in the reference network, where n is a positive integer, the denoising feature data includes n spatial denoising feature data and n cross denoising feature data, and the step of controlling the denoising network to generate denoising feature data based on the reference feature data and the motion feature data includes:

[0190] The spatial reference feature data output from the first reference spatial attention layer is input into the first denoising spatial attention layer to obtain the output first spatial denoising feature data;

[0191] The motion feature data output from the first residual block and the first spatial denoising feature data are fused to obtain the first fused feature data;

[0192] The first fused feature data and the cross-reference feature data output by the first reference cross-attention layer are input into the first denoising cross-attention layer to obtain the output first cross-denoising feature data;

[0193] Starting with x equal to 2, where x is an integer between 2 and n, perform the following steps until x equals n:

[0194] Input the spatial reference feature data output from the xth reference spatial attention layer and the (x-1)th spatial denoising feature data into the xth denoising spatial attention layer to obtain the xth spatial denoising feature data output.

[0195] The motion feature data output from the xth residual block and the xth spatial denoising feature data are fused to obtain the xth fused feature data;

[0196] The xth fused feature data, the cross-reference feature data output by the xth reference cross-attention layer, and the (x-1)th cross-denoising feature data are input into the xth denoising cross-attention layer to obtain the xth output cross-denoising feature data.

[0197] Optionally, the tensor shape of the motion feature data and the tensor shape of the corresponding spatial denoising feature data are corresponding.

[0198] Optionally, determining the model image of the target object in the target image includes:

[0199] Extract model information describing the motion state of the target object in three-dimensional space from the target image;

[0200] The model image is rendered based on the model information.

[0201] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0202] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, embodiments of the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0203] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0204] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0205] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0206] Although preferred embodiments of the present invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the present invention.

[0207] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0208] The foregoing has provided a detailed description of an image generation method, an image generation device, an electronic device, and a readable storage medium provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. An image generation method, characterized in that, The method includes: A model image of the target object in the target image and reference feature data corresponding to the reference image are determined; wherein, the model image represents the motion state of the target object in three-dimensional space; Motion feature data are extracted from the model image; Based on the reference feature data and the motion feature data, a result image is generated in which the motion of the target object is controlled by the target image.

2. The method according to claim 1, characterized in that, The step of extracting motion feature data from the model image includes: The model image is input into a feature extraction network constructed using n stacked residual blocks to obtain motion feature data output by each residual block.

3. The method according to claim 2, characterized in that, At least one of the residual blocks has a self-attention layer.

4. The method according to claim 2 or 3, characterized in that, The feature extraction network is a U-shaped network, and the residual blocks in the feature extraction network include a downsampling convolutional layer.

5. The method according to any one of claims 1-4, characterized in that, The step of generating a result image in which the motion of the target object is controlled by the target image based on the reference feature data and the motion feature data includes: Based on the reference feature data and the motion feature data, the denoising network is controlled to generate denoised feature data; The denoised feature data is decoded to obtain the resulting image.

6. The method according to claim 5, characterized in that, The motion feature data includes motion feature data output by n residual blocks in the feature extraction network; the denoising network includes n denoising spatial attention layers and n denoising cross attention layers; the reference feature data includes spatial reference feature data output by n reference spatial attention layers and cross reference feature data output by n reference cross attention layers in the reference network, where n is a positive integer; the denoising feature data includes n spatial denoising feature data and n cross denoising feature data; and the step of controlling the denoising network to generate denoising feature data based on the reference feature data and the motion feature data includes: The spatial reference feature data output from the first reference spatial attention layer is input into the first denoising spatial attention layer to obtain the output first spatial denoising feature data; The motion feature data output from the first residual block and the first spatial denoising feature data are fused to obtain the first fused feature data; The first fused feature data and the cross-reference feature data output by the first reference cross-attention layer are input into the first denoising cross-attention layer to obtain the output first cross-denoising feature data; Starting with x equal to 2, where x is an integer between 2 and n, perform the following steps until x equals n: Input the spatial reference feature data output from the x-th reference spatial attention layer and the (x-1)-th spatial denoising feature data into the x-th denoising spatial attention layer to obtain the output x-th spatial denoising feature data; The motion feature data output from the xth residual block and the xth spatial denoising feature data are fused to obtain the xth fused feature data; The xth fused feature data, the cross-reference feature data output by the xth reference cross-attention layer, and the (x-1)th cross-denoising feature data are input into the xth denoising cross-attention layer to obtain the xth output cross-denoising feature data.

7. The method according to claim 6, characterized in that, The tensor shape of the motion feature data corresponds to the tensor shape of the corresponding spatial denoising feature data.

8. The method according to any one of claims 1-7, characterized in that, The method for determining the model image of the target object in the target image includes: Extract model information describing the motion state of the target object in three-dimensional space from the target image; The model image is rendered based on the model information.

9. An image generation apparatus, characterized in that, The device includes: The data determination module is used to determine the model image of the target object in the target image, and the reference feature data corresponding to the reference image; wherein, the model image represents the motion state of the target object in three-dimensional space; The feature extraction module is used to extract motion feature data from the model image; An image generation module is used to generate a result image in which the motion of the target object is controlled by the target image, based on the reference feature data and the motion feature data.

10. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method described in any one of claims 1-8.

11. A readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor of the electronic device, the electronic device is able to perform the method as described in any one of claims 1-8.

Images