A multi-view character image reconstruction method and device based on twin diffusion model

Abstract

The present invention discloses a multi-view character image reconstruction method and device based on a twin diffusion model. The method obtains a two-dimensional RGB reference character image and a two-dimensional RGB target character image to form a training data pair. Based on the two-dimensional RGB target character image, a corresponding posture image is obtained. An overall human semantic segmentation mask map and a local region segmentation map are extracted. The training data pair is encoded as latent features and random noise is added. Feature fusion and denoising processes are performed based on the twin diffusion model to perform image restoration. Finally, a final target character image is obtained based on a decoder. The present invention uses the twin diffusion model framework to overcome the inherent shortcomings of previous traditional methods, which have a weak ability to retain fine-grained information. This method effectively improves the appearance consistency of characters with complex patterned clothing and overcomes the problem of previous traditional methods being unable to reasonably generate images when the reference character pose and the target pose vary greatly. This significantly improves the quality of the generated images.

Description

Technical Field

[0001] The present invention belongs to the field of image data generation and artificial intelligence technology, and in particular relates to a multi-view character image reconstruction method and device based on a twin diffusion model. Background Art

[0002] AI-generated content (AIGC) technology has flourished in recent years, possessing broad social value in e-commerce, short video effects, and fashion. The task of generating multi-view portrait images from a single RGB image using pose guidance primarily involves using a generative model to process a single RGB portrait image and target pose features, such as 2D portrait key points, to generate a portrait image that matches the target pose. High appearance consistency across different poses is the most important evaluation metric for this task, determining its upper limit of application value.

[0003] To achieve high-quality image generation, most existing technologies employ popular diffusion model-based approaches. For example, CFLD (Coarse-to-fine latent diffusion for pose-guided person images synthesis) utilizes mixed force attention and perceptual decoders for coarse-to-fine image generation, while PCDM (Advancing pose-guided image synthesis with progressive conditional diffusion models) utilizes multi-stage diffusion models to gradually fuse and repair features to generate high-fidelity images. However, these two approaches lack the quality of generating fine-grained, complex clothing patterns and are unable to effectively restore the clothing pattern of a single input RGB person image 1:1. Furthermore, when the pose of the input single RGB person image differs significantly from the target pose, reasonable person images cannot be generated. Therefore, designing a generation method that maximizes the consistency of multi-view person images remains an unresolved issue. Summary of the Invention

[0004] The present invention aims to address the shortcomings of the existing technology and provide a multi-view character image reconstruction method and device based on the twin diffusion model. This method can significantly improve the appearance consistency and image quality of characters from multiple perspectives. Compared with previous methods, this method can effectively solve the problem of being unable to reasonably generate and generate complex patterns when the pose of the input RGB character image is significantly different from the target pose.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: a multi-view character image reconstruction method based on the twin diffusion model, which includes two parts: model training and model inference:

[0006] (1) Model training

[0007] Step 1.1: Obtain a 2D RGB reference person image, a 2D RGB target person image, and a 2D RGB target pose image;

[0008] Step 1.2: Obtain the overall human semantic segmentation mask map and local region segmentation map based on the 2D RGB reference person image and the 2D RGB target pose image;

[0009] Step 1.3: Input the 2D RGB reference person image and the 2D RGB target person image into the encoder to obtain the latent reference person image features and the target person image features, and add noise to the target person image features to obtain the latent variables; input the 2D RGB target pose image into the pose multi-layer convolutional network to obtain the pose features;

[0010] Step 1.4: Based on the twin diffusion model, the results of steps 2 and 3 are fused, and the noise is removed through the reverse diffusion process to restore the target person's image features;

[0011] (2) Model reasoning

[0012] Step 2.1: Obtain a 2D RGB reference person image and a 2D RGB target pose image to be reconstructed. Based on the model training process, obtain an overall human semantic segmentation mask and a local region segmentation map. Obtain latent reference person image features and pose features based on the trained encoder and pose multi-layer convolutional network. Add random noise to the latent reference person image features.

[0013] Step 2.2: Perform feature fusion based on the trained twin diffusion model, remove noise through the reverse diffusion process, restore and generate the complete latent image features, and input them into the decoder to obtain the final target person image.

[0014] Furthermore, in step 1.1, the 2D RGB reference person image should have clear clothing texture and outline as well as complete portrait features, and the person should occupy more than 70% of the entire image. The 2D RGB reference person image and the 2D RGB target person image of the same person in different postures or perspectives are used as a data pair for training.

[0015] Furthermore, in step 1.2, the human body parsing information of the two-dimensional RGB reference person image and the RGB image of the reference person's human body posture key points are extracted; the human body parsing information, the RGB image of the human body posture key points and the two-dimensional RGB target posture image are combined to obtain the overall human body semantic segmentation mask map and the local area segmentation map of the overlapping area.

[0016] Furthermore, the two-dimensional RGB reference person image is estimated and inferred to obtain masks of various body parts to obtain human body parsing information, and the two-dimensional RGB reference person image is inferred through the posture estimation model to obtain the reference person's human body posture key point RGB image; based on key point matching, the key points of the overlapping area of ​​the reference person's human body posture key point RGB image and the two-dimensional RGB target posture image are obtained, and then the human body parsing information is screened to obtain local masks of each overlapping area, which are combined into a mask image to obtain the overall human body semantic segmentation map; the two-dimensional RGB reference person image is segmented using the local mask to obtain each local area segmentation map.

[0017] Furthermore, in step 1.4, the twin diffusion model uses two UNet networks, namely the feature extraction network and the generation network, to achieve fine-grained feature fusion in two stages based on the attention fusion mechanism guided by the character parsing mask and the local alignment mechanism guided by CLIP.

[0018] Furthermore, in the first stage, the attention fusion mechanism guided by the person parsing mask uses the interpolation method to reshape the overall human semantic segmentation map into the feature variable shape of each layer in the generative network. In the second stage, CLIP is used to extract the features of overlapping local areas and fuse them with the latent variables to achieve feature alignment enhancement of local areas.

[0019] Furthermore, in step 2.1, the random noise is random Gaussian noise, which is obtained by random initialization, and its shape and size are consistent with the characteristics of the potential reference person image.

[0020] In the second aspect, the present invention also provides a multi-perspective character image reconstruction device based on the twin diffusion model, comprising a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it implements the multi-perspective character image reconstruction method based on the twin diffusion model.

[0021] In a third aspect, the present invention further provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the multi-perspective character image reconstruction method based on the twin diffusion model.

[0022] In a fourth aspect, the present invention further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the multi-perspective character image reconstruction method based on the twin diffusion model.

[0023] The advantages of the present invention over the existing technology are: the present invention adopts the twin diffusion model framework to overcome the inherent shortcomings of traditional methods in retaining fine-grained information, effectively improves the appearance consistency of people with complex patterned clothes, overcomes the problem that traditional methods cannot reasonably generate when the reference character posture and the target posture change too much, and significantly improves the quality of the generated images. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a flow chart for implementing the method of the present invention.

[0025] Figure 2 It is the overall framework diagram of the method framework of the present invention.

[0026] Figure 3 It is a schematic diagram of segmenting and estimating the overlapping area between the reference person image posture and the target person image posture provided by the present invention.

[0027] Figure 4 This is a schematic diagram of the mask-guided attention fusion mechanism proposed in this invention.

[0028] Figure 5 This is a schematic diagram comparing the application effect of the method provided by the example of the present invention with the effect of the current method.

[0029] Figure 6 It is a structural schematic diagram of a multi-view character image reconstruction device based on a twin diffusion model provided by the present invention. DETAILED DESCRIPTION

[0030] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0031] Variational autoencoder, a generative model, consists of two main parts: encoder and decoder The encoder encodes the input high-dimensional variables into a latent space, obtaining low-dimensional latent variables. This allows the neural network to learn features with the highest information density, reducing computational overhead. The decoder then projects the processed low-dimensional latent variables back into the original high-dimensional space.

[0032] The diffusion model is a generative model whose training process includes two processes: forward denoising and reverse denoising. The forward denoising process is to gradually add random noise to the input variable according to the time step t so that it gradually becomes random Gaussian noise. The reverse denoising process is to train a denoising model. , input random Gaussian noise to predict the noise added at each step and then gradually remove it to generate target data from noise.

[0033] The present invention provides a multi-view character image reconstruction method based on the twin diffusion model, such as Figure 1 As shown, the following steps are included:

[0034] Model training part:

[0035] Step 1: Obtain the 2D RGB reference person image input by the user , two-dimensional RGB target person image ;

[0036] Reference person images can be sourced from online shopping websites or real-life photographs, and can be from various perspectives, including the front, side, and back. To achieve optimal generation quality, the person should occupy at least 70% of the entire image, and the image should have clear clothing textures and outlines, as well as complete portrait features. This paper uses the In-Shop Clothes Retrieval dataset from the open-source multi-viewpoint person dataset DeepFashion for training, using two images of the same person in different poses or perspectives (reference image - target image) as a data pair for training.

[0037] Step 2: Extract the human body analysis information of the reference person image above , reference human body posture key point RGB image ,use , 2D RGB target pose image and To obtain the overall human semantic segmentation mask map of the overlapping area and local region segmentation map ;

[0038] Reference human body analysis information of human images Through the open source parsing segmentation model Perform estimated reasoning to obtain the masks of each body part, and Through the open source openpose pose estimation model and The present invention classifies the obtained mask into four regions: head, torso, legs and hands. ,Right now Then the present invention utilizes and conduct Figure 3 The key points shown in the figure are matched to obtain the key points of the overlapping area. Filter to obtain the local masks of the overlapping regions , combine them into a mask image, that is, the overall human semantic segmentation map , using local mask pairs Segmentation is performed to obtain images of each local area .

[0039] Step 3: Place the above and Input into the encoder of the pre-trained variational autoencoder to obtain the potential reference person image features and target person image features , input the target posture image into the posture multi-layer convolutional network to obtain posture features . For the target person image features According to the preset time step t, random Gaussian noise is gradually added to obtain the noisy latent variable ;

[0040] Combined with attachment Figure 2 , using encoder Will Encoding into the latent space yields , which can be expressed as The posture multi-layer convolutional network consists of four layers, where the input channels of each layer are (16, 32, 96, 256) respectively. Each layer contains two standard convolutional layers, where the stride lengths are 1 and 2 respectively, and the convolution kernel size is 3×3. According to the preset time step t, random Gaussian noise is gradually added to obtain the noisy latent variable .

[0041] Step 4: Place the , , , and Input into the twin diffusion model framework for feature fusion, gradually remove noise through the reverse diffusion process, and restore the complete potential image features ;

[0042] Combined with attachment Figure 2 The twin diffusion model proposed in this paper adopts two UNet networks, namely the feature extraction network and the generation network; the feature extraction network is responsible for extracting To extract fine-grained features from the dataset, the weights in both UNet networks are initially loaded with pre-trained UNet weights from Stable Diffusion v1.5. The CLIP model loads the pre-trained model to extract local features. In the entire framework, only the parameters in the feature extraction network, the generation network, and the pose encoder are trained; the remaining model parameters are fixed. The model training objective function is:

[0043]

[0044] in For each step Random noise added through training To predict the noise added in each step to achieve denoising. Figure 4 , this paper proposes a two-stage strategy to achieve fine-grained feature fusion, including the attention fusion mechanism guided by the character parsing mask and the local alignment mechanism guided by CLIP. The attention fusion mechanism guided by the character parsing mask first uses the interpolation method to transform the overall human semantic segmentation map into Reshaped into the shape of the feature variables of each layer in the generative network, the calculation formula for one stage in the generative network is:

[0045]

[0046] in This is to generate the query, key, and value matrices in the attention modules of each layer of the network, where d is the dimension of the key matrix. This step allows the model to focus on feature extraction in overlapping areas, thereby avoiding the impact of irrelevant areas on the generation quality, and can effectively improve the reconstruction effect of complex patterned clothing. The calculation formula for the second stage in the generation network is:

[0047]

[0048] In the second stage, CLIP is used to extract the features of the overlapping local areas and fuse them with the latent variables to achieve feature alignment enhancement of the local areas, which can effectively retain the fine-grained features of the local clothes and improve the generation quality. Figure 4 The present invention visualizes the attention distribution map, and it can be found that the fusion mechanism proposed in the present invention can make the generation process pay more attention to the features of the overlapping areas and thus better retain fine-grained information.

[0049] Based on the above loss function, backpropagation calculations are performed on the trainable parameters. The optimizer and stochastic gradient descent are used to optimize and update the trainable parameters until the model converges. After the model is trained, it can be used for inference to achieve image synthesis.

[0050] Model reasoning part:

[0051] Step 1: Obtain the 2D RGB reference person image input by the user and a 2D RGB target pose image of the target human pose key points ;

[0052] Reference person images can be sourced from online shopping websites or real-life photos, and can be from various angles, such as the front, side, and back. To achieve better generation quality, the person should occupy more than 70% of the entire image, and the image should have clear clothing textures and contours, as well as complete portrait features. Should be consistent with the openpose data format, such as the number of key points.

[0053] Step 2: Extract the above Human body analysis information , 2D RGB target pose image of the reference human body pose key points ,use , and Or the user directly inputs the overall human semantic segmentation mask of the overlapping area and local region segmentation map ;

[0054] There are two ways to obtain and The first is to use the open source parsing segmentation model to Perform estimation reasoning to obtain the masks of each body part, and use the open source openpose pose estimation model to estimate the pose of the body part. Inference , and then use and Perform key point matching to obtain the key points of the overlapping area and then Filter to obtain the local masks of the overlapping regions , combine them into a mask image, that is, the overall human semantic segmentation map , using local mask pairs Segmentation is performed to obtain images of each local area The second method is for users to directly input the overall human body semantic segmentation map and .

[0055] Step 3: Place the above Input into the encoder of the pre-trained variational autoencoder to obtain the potential reference person image features , input the target posture image into the posture multi-layer convolutional network to obtain posture features Random initialization to get random Gaussian noise , its shape and size are consistent;

[0056] Combined with attachment Figure 2 , using encoder Will Encoding into the latent space yields , which can be expressed as The posture multi-layer convolutional network consists of four layers, where the input channels of each layer are (16, 32, 96, 256), and each layer contains two standard convolutional layers with strides of 1 and 2, and the convolution kernel size is 3×3. Gaussian noise of the same shape .

[0057] Step 4: Place the , , , and Input into the twin diffusion model framework for feature fusion, gradually remove noise through the reverse diffusion process, and generate target potential image features ;

[0058] Combined with attachment Figure 2 The reasoning process uses the trained twin diffusion model and extracts the feature extraction network Fine-grained appearance features. The noise of each time step is gradually predicted through the proposed two-stage fusion mechanism. , and then gradually remove The noise in the target image is obtained by The present invention adopts the Classifier-Free (Classifier-free diffusion guidance) reasoning strategy to achieve controllable generation, the formula is as follows

[0059]

[0060] in Represents a vector with the same shape as the input but with a value of 0. This strategy can improve the diversity and accuracy of controllable generation. is the control condition coefficient (generally set to 3.5).

[0061] Step 5: Place the Input to The final target person image is obtained by mapping it into a high-dimensional image space.

[0062] Combined with attachment Figure 5 A comparison chart showing the generation effect of the method of the present invention and the previous method is shown in Table 1.

[0063] Table 1

[0064]

[0065] Compared with previous methods, this paper adopts the twin diffusion model framework to overcome the inherent shortcomings of previous methods in retaining fine-grained information. It effectively improves the appearance consistency of people with complex patterned clothes, overcomes the problem of previous methods that cannot generate reasonably when the reference character posture and the target posture change too much, and significantly improves the quality of generated images.

[0066] Corresponding to the aforementioned embodiment of a multi-perspective character image reconstruction method based on a twin diffusion model, the present invention also provides an embodiment of a multi-perspective character image reconstruction device based on a twin diffusion model.

[0067] See also Figure 6 An embodiment of the present invention provides a multi-perspective character image reconstruction device based on a twin diffusion model, comprising a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it is used to implement a multi-perspective character image reconstruction method based on a twin diffusion model in the above embodiment.

[0068] The embodiment of the multi-view character image reconstruction device based on the twin diffusion model provided by the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located reading the corresponding computer program instructions in the non-volatile memory into the memory for execution. From the hardware level, if Figure 6 As shown in the figure, a hardware structure diagram of a multi-view character image reconstruction device based on a twin diffusion model provided by the present invention is provided in any device with data processing capabilities, except Figure 6 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0069] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0070] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present invention. A person of ordinary skill in the art can understand and implement the present invention without inventive work.

[0071] An embodiment of the present invention further provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, a multi-view character image reconstruction method based on a twin diffusion model in the above embodiment is implemented.

[0072] The computer-readable storage medium may be an internal storage unit of any device with data processing capabilities described in any of the aforementioned embodiments, such as a hard disk or memory. The computer-readable storage medium may also be an external storage device of any device with data processing capabilities, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capabilities. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and may also be used to temporarily store data that has been output or is to be output.

[0073] The present invention also provides a computer program product, including a computer program. When the computer program is executed by a processor, it implements the multi-view character image reconstruction method based on the twin diffusion model.

[0074] The above embodiments are used to illustrate the present invention rather than to limit the present invention. Any modifications and changes made to the present invention within the spirit of the present invention and the protection scope of the claims shall fall within the protection scope of the present invention.

Claims

1. A multi-view character image reconstruction method based on a twin diffusion model, characterized in that: This method includes two parts: model training and model inference: (1) Model training Step 1.1: Obtain a 2D RGB reference person image and a 2D RGB target person image, and infer a 2D RGB target pose image; Step 1.2: Obtain the overall human semantic segmentation mask map and local region segmentation map based on the 2D RGB reference person image and the 2D RGB target pose image; Step 1.3: Input the 2D RGB reference person image and the 2D RGB target person image into the encoder to obtain the latent reference person image features and the target person image features, and add noise to the target person image features to obtain the latent variables; input the 2D RGB target pose image into the pose multi-layer convolutional network to obtain the pose features; Step 1.4: Based on the twin diffusion model, the results of step 1.2 are fused with the potential reference person image features, latent variables, and posture features obtained in step 1.

3. The noise is removed through the reverse diffusion process to restore the target person image features. The twin diffusion model uses two UNet networks: a feature extraction network and a generative network. It implements fine-grained feature fusion in two stages: a person parsing mask-guided attention fusion mechanism and a CLIP-guided local alignment mechanism. In the first stage, the person parsing mask-guided attention fusion mechanism uses interpolation to reshape the overall human semantic segmentation map into the shape of feature variables at each layer in the generative network. In the second stage, CLIP is used to extract features of overlapping local regions and fuse them with latent variables to achieve feature alignment enhancement in local regions. (2) Model reasoning Step 2.1: Obtain the 2D RGB reference person image and 2D RGB target pose image to be reconstructed, as well as the overall human semantic segmentation mask and local region segmentation map. Based on the trained encoder and pose multi-layer convolutional network, obtain the potential reference person image features and pose features. And random noise is obtained by random initialization; Step 2.2: Perform feature fusion based on the trained twin diffusion model, remove noise through the reverse diffusion process, restore and generate the complete latent image features, and input them into the decoder to obtain the final target person image.

2. The multi-view character image reconstruction method based on the twin diffusion model according to claim 1, characterized in that: In step 1.1, the 2D RGB reference person image should have clear clothing texture and outlines and complete portrait features. The person should occupy more than 70% of the entire image. The 2D RGB reference person image and the 2D RGB target person image of the same person in different poses or perspectives are used as a data pair for training.

3. The multi-view character image reconstruction method based on the twin diffusion model according to claim 1, characterized in that: In step 1.2, the human body parsing information of the two-dimensional RGB reference person image and the RGB image of the reference person's human body posture key points are extracted; the human body parsing information, the RGB image of the human body posture key points and the two-dimensional RGB target posture image are combined to obtain the overall human body semantic segmentation mask map and the local area segmentation map of the overlapping area.

4. The multi-view character image reconstruction method based on the twin diffusion model according to claim 3, characterized in that: The two-dimensional RGB reference person image is estimated and inferred to obtain the mask of each body part to obtain the human body parsing information. The two-dimensional RGB reference person image is inferred through the posture estimation model to obtain the RGB image of the reference person's human body posture key points; Based on key point matching, the key points of the overlapping areas of the reference person's human posture key point RGB image and the two-dimensional RGB target posture image are obtained. Then, the human body analysis information is filtered to obtain the local masks of the overlapping areas, which are combined into a mask image to obtain the overall human body semantic segmentation map; the two-dimensional RGB reference person image is segmented using the local mask to obtain the segmentation maps of each local area.

5. The multi-view character image reconstruction method based on the twin diffusion model according to claim 1, characterized in that: In step 2.1, the random noise is random Gaussian noise, which is obtained by random initialization and has the same shape and size as the characteristics of the potential reference person image.

6. A multi-view character image reconstruction device based on a twin diffusion model, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the processor executes the executable code, it implements a multi-view character image reconstruction method based on a twin diffusion model according to any one of claims 1 to 5.

7. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, a multi-view character image reconstruction method based on a twin diffusion model according to any one of claims 1 to 5 is implemented.

8. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the multi-view character image reconstruction method based on the twin diffusion model as described in any one of claims 1 to 5 is implemented.

Images