Posture correction method and computer device for controllable human image generation

By pooling, cropping, and flattening the text features of the pre-trained model, and by optimizing pose control using self-attention and cross-attention mechanisms, the problem of poor pose control accuracy in existing models is solved, and a higher precision pose control effect is achieved.

CN118675233BActive Publication Date: 2024-11-12NANJING UNIV OF POSTS & TELECOMM
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411170926.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-26
Publication Date
2024-11-12
Estimated Expiration
2044-08-26

AI Technical Summary

Technical Problem

Existing controllable human image generation models have poor pose control accuracy when faced with complex or rare action conditions, and are prone to generation failures, especially in crowded scenes.

Method used

By performing max pooling, cropping, and flattening on the text features of a pre-trained controllable human image generation model, and combining self-attention and cross-attention mechanisms, the pose control process is optimized to generate images with higher pose control accuracy.

Benefits of technology

It improves the accuracy of pose control, enhances the pose consistency and text consistency of generated images, and is suitable for pre-trained pose-controlled human image generation models with diffusion models, demonstrating excellent generation results.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118675233B_ABST
    Figure CN118675233B_ABST
Patent Text Reader

Abstract

The present invention discloses a posture correction method and computer device for controllable human image generation, wherein the method comprises: S1: extracting a text feature tensor from a controllable human image generation model; S2: cropping and flattening the text feature tensor after pooling to obtain a one-dimensional text tensor; S3: adding the one-dimensional text tensor to the diffusion step tensor to obtain a conditional tensor; S4: slicing the predicted noise tensor to obtain a noise tensor; S5: adding the position number tensor to the noise tensor to obtain a new noise tensor; S6: inputting the new noise tensor into a layer normalization network and a multi-head self-attention network in sequence to obtain an output tensor; S7: inputting the output tensor and the conditional tensor into a multi-head cross-attention network at the same time to obtain a final predicted noise tensor; S8: generating a final image with higher posture control accuracy. The present invention can improve the accuracy of posture control.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of generative artificial intelligence technology, and in particular to a posture correction method and computer equipment for controllable human image generation. Background Art

[0002] Controllable human image generation is essentially a process of generating images from text. Existing large-scale image generation models can achieve high-quality human-centric image generation based on natural language text prompts containing human descriptions. However, these general large models cannot control the pose of the human in the generated image. Controllable human image generation aims to generate human-centric images given conditions such as human pose and text. It has many applications, such as animation, game making, and virtual try-on, and has attracted great attention from academia and industry.

[0003] For the problem of controllable human image generation, existing research is usually based on a pre-trained large-scale image generation model Stable Diffusion (SD) to fully utilize the image generation capabilities of the large model. One solution is to introduce an additional learnable posture conditional branch while freezing the weights of the SD model. The additional conditional branch enables the model to consider posture conditions during image generation, improving the posture controllability of the original SD model, which has gained widespread attention and recognition. Another solution is to analyze the drawbacks of frozen branches and use a heat map-guided denoising loss to fine-tune the original SD model instead of using dual-branch diffusion, which is superior to the method of introducing additional branches in terms of precise posture control and image quality. However, when faced with very crowded scenes and complex and rare action conditions, the above two solutions still fail to generate. Summary of the invention

[0004] The present invention provides a posture correction method and computer device for controllable human image generation that effectively improves posture control accuracy, which can solve at least one of the above technical problems.

[0005] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0006] A posture correction method for controllable human image generation comprises the following steps:

[0007] S1: Let the predicted noise tensor output by the noise prediction network UNet in the pre-trained controllable human image generation model be N output , the extracted text feature tensor in the pre-trained controllable human image generation model is denoted as T CLIP ;

[0008] S2: Text feature tensor T CLIPInput the one-dimensional maximum pooling network for maximum pooling, and then crop and flatten it after pooling to obtain a one-dimensional text tensor T;

[0009] S3: Add the one-dimensional text tensor T to the diffusion step tensor t in the pre-trained controllable human image generation model to obtain the conditional tensor C;

[0010] S4: predict the noise tensor N output Do slicing to get the noise tensor N' output ;

[0011] S5: Combine the position number tensor P with the noise tensor N' output Add them together to get the new noise tensor N;

[0012] S6: Input the new noise tensor N into the layer normalization network and the multi-head self-attention network SA in sequence, and then input it into the layer normalization network to obtain the output tensor N4;

[0013] S7: Input the output tensor N4 and the condition tensor C into the multi-head cross attention network CA at the same time, and input the results into the layer normalization network and the position element-by-element feedforward network in sequence to obtain the final prediction noise tensor N';

[0014] S8: The final predicted noise tensor N' is input back into the subsequent process of the pre-trained controllable human image generation model to generate a final image with higher posture control accuracy.

[0015] Furthermore, in S1, the noise tensor N is predicted output The size is 64×64×4, the text feature tensor T CLIP The size is 77×1024.

[0016] Furthermore, in S2, the specific method of pooling is: the text feature tensor T CLIP Input into the one-dimensional maximum pooling network for maximum pooling, reducing the size of the feature dimension from 1024 to 16, and the resulting pooled text feature tensor T maxpooling The size is 77×16.

[0017] Furthermore, in S2, the specific method of flattening is: the pooled text feature tensor T maxpooling It is cropped along the length dimension into a tensor of size 72×16, and then flattened to obtain a one-dimensional text tensor T of size 1×1152.

[0018] Furthermore, in S3, the size of the conditional tensor C is 1×1152.

[0019] Furthermore, in S4, the specific method of slicing is: slicing the prediction noise tensor N with a length and width of 64×64 with a 2×2 slicing scaleoutput Cut into 1024 2×2 small blocks, stack the 2×2 small blocks along the channel dimension, then the size of each small block is 1×1×16, and then map the channel dimension of size 16 to 1152.

[0020] Furthermore, in S6, the method for obtaining the output tensor N4 is: inputting the new noise tensor N into the layer normalization network LN1 to obtain the output tensor N1, inputting the output tensor N1 into the multi-head self-attention network SA to obtain the output tensor N2, adding the new noise tensor N to the output tensor N2 to obtain the output tensor N3, and inputting the output tensor N3 into the layer normalization network LN2 to obtain the output tensor N4.

[0021] Furthermore, in S7, the method for obtaining the final predicted noise tensor N' is: input the output tensor N4 and the conditional tensor C into the multi-head cross-attention network CA at the same time to obtain the output tensor N5, add the output tensor N5 and the output tensor N3 to obtain the output tensor N6, input the output tensor N6 into the layer normalization network LN3 to obtain the output tensor N7, input the output tensor N7 into the position element-by-element feedforward network PF to obtain the output tensor N8, add the output tensor N8 and the output tensor N6 to obtain the final predicted noise tensor N'.

[0022] Furthermore, in S6, the multi-head self-attention network SA has 16 attention heads.

[0023] Furthermore, in S7, the multi-head cross attention network CA has 16 attention heads.

[0024] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the posture correction method for controllable human image generation.

[0025] The beneficial effects of the present invention are embodied in:

[0026] 1. The present invention uses pooling, cropping and flattening methods to further process the text features in the original pre-trained posture-controllable human image generation model so that it can be reasonably input into the posture control process.

[0027] 2. The present invention utilizes the self-attention mechanism and the cross-attention mechanism to enable the model to better learn the contextual information of the posture image block, thereby achieving the purpose of posture correction, and enabling the model to generate human images with higher posture control accuracy.

[0028] 3. The present invention can be reasonably used for different pre-trained posture-controllable human image generation models based on diffusion models. It is lightweight and convenient to use and can effectively improve the performance of the model.

[0029] 4. The present invention fully learns the posture control information and can achieve excellent results on the current advanced pre-trained posture-controllable human image generation model and public data of human images. It has good application prospects and practical value in scenarios with high requirements for posture control accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The drawings described herein are used to provide further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute improper limitations on the present application.

[0031] Figure 1 It is a flow chart of a posture correction method according to an embodiment of the present invention.

[0032] Figure 2 It is a schematic diagram of a model of an embodiment of the present invention.

[0033] Figure 3 is the original image.

[0034] Figure 4 yes Figure 3 An image generated after being processed by the posture correction method of an embodiment of the present invention.

[0035] Figure 5 yes Figure 3 The image is generated after being processed by HumanSD, a pose-controllable human image generation model.

[0036] Figure 6 It is a structural block diagram of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. In the absence of conflict, the embodiments in this application and the features in the embodiments can be combined with each other. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0038] See also Figure 1-Figure 2The embodiment of the present invention provides a posture correction method for controllable human image generation, which can be reasonably used for different pre-trained posture controllable human image generation models based on diffusion models. This embodiment selects the current advanced pre-trained model HumanSD, which is a controllable human image generation model with posture and text as control conditions. By inputting a posture condition image and a sentence of description text, an image that conforms to the text description can be generated, and the posture of the person in the image is consistent with the input posture. Further, the following steps S1-S8 are included:

[0039] S1: Let the predicted noise tensor output by the noise prediction network UNet in HumanSD be N HumanSD , the text feature tensor extracted from HumanSD is T CLIP ;

[0040] Prediction noise tensor N HumanSD The size of the text feature tensor T is 64×64×4. CLIP The size is 77×1024.

[0041] S2: Convert the text feature tensor T in S1 CLIP Input the one-dimensional maximum pooling network for maximum pooling, reducing the size of the feature dimension from 1024 to 16, that is, the pooled text feature tensor T maxpooling The size is 77×16;

[0042] The pooled text feature tensor T maxpooling It is cropped along the length dimension into a tensor of size 72×16, and then flattened to obtain a one-dimensional text tensor T of size 1×1152.

[0043] S3: Add the one-dimensional text tensor T obtained in S2 to the diffusion step tensor t in HumanSD to obtain the conditional tensor C, the size of which is 1×1152.

[0044] S4: transform the predicted noise tensor N in S1 HumanSD Do slicing and slice the predicted noise tensor N with a length and width of 64×64 with a 2×2 slice scale HumanSD Cut into 1024 2×2 small blocks, stack the 2×2 small blocks along the channel dimension, then the size of each small block is 1×1×16, and then map the channel dimension of size 16 to 1152, that is, 64×64×4 prediction noise tensor N HumanSD After this step, we get a noise tensor N' with a size of 1×1024×1152. HumanSD .

[0045] S5: Number the 1024 small blocks in S4, that is, 1024 numbers from 1 to 1024, and map each number into a tensor of size 1×1152. The size of the tensor P composed of 1024 numbers is 1×1024×1152.

[0046] Compare the position number tensor P with the noise tensor N' obtained in S4 HumanSD Add them together to get the new noise tensor N.

[0047] S6: Input the new noise tensor N obtained in S5 into the layer normalization network LN1 to obtain the output tensor N1, input the output tensor N1 into the multi-head self-attention network SA to obtain the output tensor N2, add the new noise tensor N and the output tensor N2 to obtain the output tensor N3, input the output tensor N3 into the layer normalization network LN2 to obtain the output tensor N4;

[0048] The multi-head self-attention network SA has 16 attention heads.

[0049] S7: Input the output tensor N4 and the condition tensor C obtained in S6 into the multi-head cross attention network CA at the same time to obtain the output tensor N5, add the output tensor N5 and the output tensor N3 to obtain the output tensor N6, input the output tensor N6 into the layer normalization network LN3 to obtain the output tensor N7, input the output tensor N7 into the position element-by-element feedforward network PF to obtain the output tensor N8, add the output tensor N8 and the output tensor N6 to obtain the final prediction noise tensor N';

[0050] The multi-head cross attention network CA has 16 attention heads.

[0051] S8: Input the final predicted noise tensor N' obtained in S7 back into the subsequent process of HumanSD to generate a final image with higher posture control accuracy.

[0052] In this embodiment, the HumanSD model of the above-mentioned posture correction method and the original HumanSD model are trained on the public human image dataset LAION-Human dataset, and tested on the test set of the public human image dataset Human-Art. LAION-Human constructs a dataset containing large-scale Internet images. Specifically, the dataset collects approximately 1 million image-text pairs, which are filtered by rules of high image quality and high human estimation confidence scores. Human-Art collects 50k high-quality images from 5 real scenes and 15 virtual scenes, with clear posture and text annotations, which can provide accurate postures and multiple scenes for quantitative evaluation. This embodiment follows the settings of Human-Art to divide the test set. The overall performance comparison results are shown in Table 1.

[0053] Among them: FID is Fréchet Inception Distance, an important indicator for measuring the quality of generated images in the field of image generation; KID is Kernel Inception Distance, an important indicator for measuring the diversity of generated images; AP is the average precision based on distance, AP (m) is the AP for medium-sized humans, CAP is the average precision based on pose cosine similarity, PCE is the headcount error, these metrics measure the difference between a given pose condition and the pose extracted from the generated image; CLIPSIM is an important indicator for measuring the degree of consistency between the generated image and the text.

[0054] Table 1

[0055]

[0056] As can be seen from Table 1, the present invention is comparable to HumanSD in terms of quality indicator FID, and comparable to HumanSD in terms of diversity indicator KID. Compared with HumanSD, the present invention shows different degrees of improvement in posture accuracy: AP is improved by 41.6%, AP(m) is improved by 6.7%, CAP is improved by 12%, and PCE is improved by 16.1%. At the same time, the present invention can also maintain CLIPSIM comparable to HumanSD. The better condition injection and posture correction methods of the present invention lead to such performance enhancement.

[0057] See also Figure 3-Figure 5 , the generation results of the method of the present invention and the current mainstream posture controllable human image generation model HumanSD for the same sentence and one posture condition are compared. Among them, Figure 3 The image in is the true value, that is, the original image, Figure 4 The human image in is an image obtained by the present invention, Figure 5 The human image in is an image generated by HumanSD, a current high-performance posture-controllable human image generation model. It can be seen that compared with HumanSD, the posture control effect of the present invention is more obvious.

[0058] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor executes the steps of the above method.

[0059] See also Figure 6 An embodiment of the present invention further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the above method.

[0060] It is understandable that the computer-readable storage medium, computer device and computer program product provided by the embodiments of the present invention correspond to the methods provided by the embodiments of the present invention, and the explanation, examples and beneficial effects of the relevant contents can refer to the corresponding parts of the above methods.

[0061] It should be noted that those skilled in the art can understand that all or part of the steps implemented in the embodiments of the present invention can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a DVD), or a semiconductor medium (eg, a solid state disk (SSD)).

[0062] In summary, the present invention provides a posture correction method for controllable human image generation. First, the text feature tensor of the pre-trained posture controllable human image generation model is subjected to maximum pooling, cropped after pooling, and flattened to obtain a one-dimensional text tensor, and the one-dimensional text tensor is added to the diffusion step in the pre-trained controllable human image generation model to obtain a conditional tensor; secondly, the predicted noise of the pre-trained posture controllable human image generation model is sliced ​​and added to the position number, and the result is input into a layer normalization network and a multi-head self-attention network, and then into a layer normalization network; thirdly, the result and the conditional tensor are simultaneously input into a multi-head cross-attention network, and the result is sequentially input into a layer normalization network and a position element-by-element feedforward network to obtain a final predicted noise tensor; finally, the final predicted noise tensor is input back into the subsequent process of the pre-trained controllable human image generation model to generate a final image with higher posture control accuracy;

[0063] The present invention provides a posture correction method for controllable human image generation, which can overcome the problem of poor posture control accuracy in the existing controllable human image generation technology. The method can be used for a posture controllable human image generation model pre-trained based on a diffusion model to further improve the accuracy of posture control, and can achieve excellent results on public data of posture controllable human image generation. The method has good application prospects and practical value in human image generation scenarios with high requirements for posture controllability.

[0064] It should be understood that the examples and implementation modes described herein are for illustrative purposes only and are not intended to limit the present invention. Those skilled in the art may make various modifications or changes based on the examples and implementation modes. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A posture correction method for controllable human image generation, characterized in that: The following steps are involved: S1: Let the predicted noise tensor output by the noise prediction network UNet in the pre-trained controllable human image generation model be N output , the extracted text feature tensor in the pre-trained controllable human image generation model is denoted as T CLIP ; S2: Text feature tensor T CLIP Input the one-dimensional maximum pooling network for maximum pooling, and then crop and flatten it after pooling to obtain a one-dimensional text tensor T; S3: Add the one-dimensional text tensor T to the diffusion step tensor t in the pre-trained controllable human image generation model to obtain the conditional tensor C; S4: predict the noise tensor N output Do slicing to get the noise tensor N' output ; S5: Combine the position number tensor P with the noise tensor N' output Add them together to get the new noise tensor N; S6: Input the new noise tensor N into the layer normalization network and the multi-head self-attention network SA in sequence, and then input it into the layer normalization network to obtain the output tensor N4; S7: Input the output tensor N4 and the condition tensor C into the multi-head cross attention network CA at the same time, and input the results into the layer normalization network and the position element-by-element feedforward network in sequence to obtain the final prediction noise tensor N'; S8: The final predicted noise tensor N' is input back into the subsequent process of the pre-trained controllable human image generation model to generate a final image with higher posture control accuracy.

2. The posture correction method for controllable human image generation according to claim 1, characterized in that: In S1, the predicted noise tensor N output The size is 64×64×4, the text feature tensor T CLIP The size is 77×1024.

3. The posture correction method for controllable human image generation according to claim 1, characterized in that: In S2, the specific method of pooling is: CLIP Input into the one-dimensional maximum pooling network for maximum pooling, reducing the size of the feature dimension from 1024 to 16, and the resulting pooled text feature tensor T maxpooling The size is 77×16.

4. The posture correction method for controllable human image generation according to claim 1, characterized in that: In S2, the specific method of flattening is: maxpooling It is cropped along the length dimension into a tensor of size 72×16, and then flattened to obtain a one-dimensional text tensor T of size 1×1152.

5. The posture correction method for controllable human image generation according to claim 1, characterized in that: In S3, the size of the condition tensor C is 1×1152.

6. The posture correction method for controllable human image generation according to claim 1, characterized in that: In S4, the specific method of slicing is: slice the predicted noise tensor N with a length and width of 64×64 with a 2×2 slice scale. output Cut into 1024 2×2 small blocks, stack the 2×2 small blocks along the channel dimension, then the size of each small block is 1×1×16, and then map the channel dimension of size 16 to 1152.

7. The posture correction method for controllable human image generation according to claim 1, characterized in that: In S6, the method for obtaining the output tensor N4 is: input the new noise tensor N into the layer normalization network LN1 to obtain the output tensor N1, input the output tensor N1 into the multi-head self-attention network SA to obtain the output tensor N2, add the new noise tensor N and the output tensor N2 to obtain the output tensor N3, input the output tensor N3 into the layer normalization network LN2 to obtain the output tensor N4.

8. The posture correction method for controllable human image generation according to claim 1, characterized in that: In S7, the method for obtaining the final predicted noise tensor N' is: input the output tensor N4 and the conditional tensor C into the multi-head cross-attention network CA at the same time to obtain the output tensor N5, add the output tensor N5 and the output tensor N3 to obtain the output tensor N6, input the output tensor N6 into the layer normalization network LN3 to obtain the output tensor N7, input the output tensor N7 into the position element-by-element feedforward network PF to obtain the output tensor N8, add the output tensor N8 and the output tensor N6 to obtain the final predicted noise tensor N'.

9. The posture correction method for controllable human image generation according to claim 1, characterized in that: In S6, the multi-head self-attention network SA has 16 attention heads, and in S7, the multi-head cross-attention network CA has 16 attention heads.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 9.