Character photo hand and foot repairing method and device and storage medium

By training the human hand and foot shape grid reconstruction module and the Controlnet depth control model, combined with the DIT diffusion model, the problem of hand and foot deformity in the generated image is solved, real-life hand and foot repair is achieved, and the beauty and reality of the character photo image is improved.

CN119048663AInactive Publication Date: 2024-11-29XIAMEN SHEQU INFORMATION TECH CO LTD

Patent Information

Application Number
CN202411558587.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-11-04
Publication Date
2024-11-29
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

In image generation technology based on diffusion model, when generating full-body photo images of a person, the number of fingers and toes is incorrect or the hand shape is distorted, seriously affecting the beauty and reality of the image.

Method used

By obtaining the correct hand and foot details and small diagrams as standard data, the human hand and foot shape grid reconstruction module and Controlnet depth control model are trained, and the DIT structure is used as the diffusion model to detect and repair hands and feet incomplete details to ensure that the hand and foot shape meets ergonomic standards.

Benefits of technology

Effectively repair the defects and deformities of the generated hands and feet, enhance the reality and beauty of the character photo images, ensure the correct number of fingers and toes, and keep other parts of the original image unchanged.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119048663A_ABST
    Figure CN119048663A_ABST
Patent Text Reader

Abstract

According to the figure pictorial hand and foot repairing method and device based on the image generation model and the storage medium provided by the invention, the generated figure pictorial image with incomplete and deformed hands and feet can be repaired through the diffusion model, the Controlnet depth control model and the human body hand and foot shape grid reconstruction module, and the sense of reality and aesthetic feeling of the figure pictorial image are improved. The whole algorithm adopts a conditional method for repairing and correcting deformed hands and feet, and the rest parts of the original figure image are kept unchanged. Wherein the human body hand and foot shape grid reconstruction module always keeps correct finger and toe numbers and shapes, and can fit required depth grid images of hand and foot postures in generated hand and foot postures of a person.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a method, device and storage medium for repairing hands and feet in a portrait, in particular to a method, device and storage medium for repairing hands and feet in a portrait based on an image generation model. Background Art

[0002] The rapid development of AIGC (artificial intelligence generated content) image generation technology has led to the widespread application of text-generated images and image-generated images related technologies based on diffusion models, especially the use of text-generated images and image-generated images to generate portrait images of people, which has high commercial entertainment value. However, when generating full-body portrait images of people including hands and feet, problems often occur, such as the wrong number of fingers and toes or strange hand shape distortion (such as Figure 1 As shown in the figure, the beauty and realism of the generated character images are seriously affected, and the monetization value of the character portrait images is greatly reduced. Therefore, it is necessary to design a character portrait hand and foot repair method based on the image generation model, which generates the shape of hands and feet that meet ergonomic standards through hand and foot defect detection and deep image control repair, so as to repair the character's hands and feet and improve the realism and beauty of the character portrait images. Summary of the invention

[0003] The present invention provides a method, device and storage medium for repairing hands and feet of a portrait, which can effectively solve the above problems.

[0004] The present invention is achieved in that: A method for restoring human portrait hands and feet based on an image generation model comprises the following steps: S1, obtaining a plurality of correct hand and foot detail images and correct hand and foot thumbnails as standard data, wherein the correct hand and foot thumbnails are half-body or full-body images of a person including hands and / or feet, and the correct hand and foot detail images are torso images including only hands or feet; S2, training a human hand and foot shape mesh reconstruction module through the correct hand and foot detail map and the hand and foot defect map generated by processing the correct hand and foot detail map in and out based on a diffusion model; S3, using the correct hand and foot detail image and the corresponding hand and foot repair images generated by processing the hand and foot defect image based on the human hand and foot shape mesh reconstruction module, to form a training data set 1 in pairs; S4, masking the correct hand and foot thumbnails, randomly masking the correct hand and foot to obtain a full-body image of the corresponding person with incomplete hands and feet, and using the correct hand and foot thumbnails and the full-body image of the corresponding person with incomplete hands and feet in pairs to form a training data set 2; S5, organizing the training data set 1 and the training data set 2 into a complete Controlnet deep control model training data set for training; S6, uses the DIT structure as the diffusion model for hand and foot redrawing, and uses the previously pre-trained Controlnet deep control model as an additional conditional constraint module to perform hand and foot image repair.

[0005] The present invention further provides a device for repairing human portrait hands and feet based on an image generation model, comprising: processor; A memory for storing processor executable instructions; wherein the processor is configured to call the instructions stored in the memory to execute the above method.

[0006] The present invention further provides a computer-readable storage medium storing a computer program, wherein the computer program can be executed by a processor of a device where the computer-readable storage medium is located to implement the above method. The beneficial effects of the present invention are as follows: the method, device and storage medium for repairing the hands and feet of a person portrait provided by the present invention can repair the generated person portrait images with incomplete and deformed hands and feet through the diffusion model, the Controlnet deep control model and the human hand and foot shape grid reconstruction module, thereby improving the realism and beauty of the person portrait images. The entire algorithm adopts a conditional method for repairing and correcting deformed hands and feet, leaving the rest of the original person image unchanged. Among them, the human hand and foot shape grid reconstruction module always adheres to the correct number and shape of fingers and toes, and can fit the required deep grid image of the hand and foot posture in the already generated person hand posture and foot posture. Through the pre-trained Controlnet deep control model, the correct hand and foot information is re-injected into the diffusion model to complete the hand and foot repair of the person image, achieving a realistic and ergonomic effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0008] Figure 1 In the prior art, the full-body portrait images of people show an incorrect number of fingers and toes or a strange and distorted hand shape.

[0009] Figure 2It is the correct hand and foot detail map used in the method for repairing hands and feet of a portrait based on an image generation model provided in an embodiment of the present invention.

[0010] Figure 3 These are the correct thumbnails of hands and feet used in the method for repairing hands and feet in portraits based on an image generation model provided in an embodiment of the present invention.

[0011] Figure 4 In the method for repairing the hands and feet of a portrait based on an image generation model provided by an embodiment of the present invention, a corresponding hand and foot defect map is generated by using a conventional diffusion generation model.

[0012] Figure 5 In the method for repairing the hands and feet of a portrait based on an image generation model provided by an embodiment of the present invention, the correct hand and foot detail map is used to generate a corresponding hand and foot mesh map through an existing mesh reconstruction model.

[0013] Figure 6 In the method for repairing the hands and feet of human portraits based on the image generation model provided by an embodiment of the present invention, the hand and foot defect images and the corresponding hand and foot mesh images are paired to form a training data set image.

[0014] Figure 7 In the method for restoring hands and feet of portraits based on an image generation model provided in an embodiment of the present invention, the corresponding hand and foot restoration images obtained by using the correct hand and foot detail images and the human hand and foot shape mesh reconstruction module constitute a training data set one in pairs.

[0015] Figure 8 In the method for restoring the hands and feet of a person's portrait based on an image generation model provided in an embodiment of the present invention, a pair of correct small images of hands and feet and corresponding full-body images of a person with incomplete hands and feet are used to form a training data set 2.

[0016] Fig. 9 It is a comparison image before and after image restoration in the method for restoring hands and feet of a portrait based on an image generation model provided by an embodiment of the present invention.

[0017] Fig.10 It is a logic block diagram for performing hand and foot image restoration in a method for restoring hand and feet of a portrait based on an image generation model provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the invention claimed for protection, but merely represents the selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0019] In the description of the present invention, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0020] Reference Figure 2-9 As shown, an embodiment of the present invention provides a method for repairing hands and feet of a portrait based on an image generation model, comprising the following steps: S1, obtain multiple correct hand and foot detail images and correct hand and foot small images as standard data. The present invention only takes hand repair as an example for specific description. Please refer to Figure 2-3 As shown; among them, Figure 2 The correct hand and foot thumbnail is shown, which is a half-body or full-body picture of a person including hands and / or feet; Figure 3 The correct hand and foot detail map is shown, which is a torso map that only includes hands or feet. The correct hand and foot detail map and the correct hand and foot thumbnail can be obtained through online photos or other means, and there is no restriction here. The correct hand and foot detail map is a torso map that only includes hands or feet, and the torso map of the hands or feet includes close-up images of various commonly used gestures or close-up images of various commonly used foot postures, and the number is not limited. The resolution of the correct hand and foot detail map and the correct hand and foot thumbnail is 512*512 pixels to 7680x4320 pixels. If the resolution is too small, the model training repair process will not be able to obtain too many hand and foot detail features.

[0021] S2, training a human hand and foot shape mesh reconstruction module using the correct hand and foot detail image. In step S2, the step of training a human hand and foot shape mesh reconstruction module using the correct hand and foot detail image specifically includes the following steps: S21, see Figure 4 As shown, the correct hand and foot detail image is masked with a random mask algorithm, and then the corresponding hand and foot defect image is generated through a conventional diffusion generation model; S22, see Figure 5 As shown, the correct hand and foot detail map is then used to generate corresponding hand and foot mesh maps through the existing mesh reconstruction model; S23, see Figure 6 As shown, the hand and foot defect images and the corresponding hand and foot mesh images are then paired to form a training data set for training the human hand and foot shape mesh reconstruction module. The trained human hand and foot shape mesh reconstruction module is mainly used to automatically detect hand and foot defects and reconstruct hand and foot shape meshes for the input human hand and foot defect generation image. Through this module, the defective hands and feet of the input human image can be meshed and have a correct hand and foot shape and posture. Figure 4-6 It can be seen that the present invention is illustrated with three common gestures, but the types and numbers of gestures are not limited.

[0022] S3, see Figure 7 As shown, through the Controlnet deep control model, the correct hand and foot detail map and the corresponding hand and foot repair map obtained by the human hand and foot shape mesh reconstruction module are used to form a training data set 1 in pairs. Figure 7 As can be seen from the figure, five pairs of training data sets are included for illustration, but the type and number of training data sets are not limited.

[0023] S4, see Figure 8 As shown, the correct hand and foot images are masked, and the correct hands and feet are randomly masked to obtain the corresponding full-body images of the person with incomplete hands and feet. The correct hand and foot images and the corresponding full-body images of the person with incomplete hands and feet are used in pairs to form a training data set 2. Figure 8 As can be seen from the figure, the training data set 2 includes 8 pairs of groups for illustration, but the type and number of the training data set 2 are not limited.

[0024] S5, organizing the training data set 1 and the training data set 2 into a complete Controlnet deep control model training data set for training.

[0025] S6, using the DIT structure as the diffusion model for hand and foot redrawing, and using the previously pre-trained Controlnet deep control model as an additional conditional constraint module to perform hand and foot image repair. Fig.10 Specifically, according to the previously trained human hand and foot shape mesh reconstruction module, the depth map of the reconstructed hand and foot parts can be obtained (such as Fig.10The repaired hand and foot mesh in ) and the mask of the character's hand and foot area (such as Fig.10 Through mask inversion, the white mask of the character's hands and feet can be inverted into a black mask to cover the corresponding area in the input original image to generate a mask image (such as Fig.10 Afterwards, the hand and foot redrawing inversion mask and the character's hand and foot redrawing mask are used as the input of the DIT diffusion model (such as Fig.10 The depth map of the repaired hand and foot areas is used as the Controlnet depth control signal. The control signal transmits the control information to the attention layer of the DIT diffusion model through the cross-attention mechanism, thereby guiding the model to repair and generate the correct hand and foot areas. The advantage of this is that such a masking strategy can ensure that only the hand and foot areas covered by the mask are repaired, ensuring the consistency of the non-hand and foot areas before and after the repair process and the hand and foot quality.

[0026] See also Fig. 9 As shown, it is a comparison diagram before and after the repair of the incomplete hands and feet portrait image by the method of the present invention. It can be seen from the figure that the method for repairing the hands and feet of a person portrait provided by the present invention can repair the generated incomplete and deformed hands and feet portrait image through the diffusion model, the Controlnet depth control model and the human hand and foot shape grid reconstruction module, thereby improving the realism and beauty of the person portrait image.

[0027] The embodiment of the present invention further provides a device for repairing human portrait hands and feet based on an image generation model, comprising: processor; A memory for storing processor executable instructions; wherein the processor is configured to call the instructions stored in the memory to execute the above-mentioned method for repairing hands and feet in human portraits.

[0028] The embodiment of the present invention further provides a computer-readable storage medium storing a computer program, wherein the computer program can be executed by a processor of a device where the computer-readable storage medium is located to implement the above-mentioned method for restoring hands and feet in portraits.

[0029] It should be understood by those skilled in the art that embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0030] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0031] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0032] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0033] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for restoring the hands and feet of a person's portrait based on an image generation model, characterized in that: The following steps are involved: S1, obtaining a plurality of correct hand and foot detail images and correct hand and foot thumbnails as standard data, wherein the correct hand and foot thumbnails are half-body or full-body images of a person including hands and / or feet, and the correct hand and foot detail images are torso images including only hands or feet; S2, training a human hand and foot shape mesh reconstruction module through the correct hand and foot detail image and the hand and foot defect image generated by processing the correct hand and foot detail image in and out based on a diffusion model; S3, using the correct hand and foot detail image and the corresponding hand and foot repair images generated by processing the hand and foot defect image based on the human hand and foot shape mesh reconstruction module, to form a training data set 1 in pairs; S4, masking the correct hand and foot thumbnails, randomly masking the correct hand and foot to obtain a full-body image of the corresponding person with incomplete hands and feet, and using the correct hand and foot thumbnails and the full-body image of the corresponding person with incomplete hands and feet in pairs to form a training data set 2; S5, organizing the training data set 1 and the training data set 2 into a complete Controlnet deep control model training data set for training; S6, uses the DIT structure as the diffusion model for hand and foot redrawing, and uses the previously pre-trained Controlnet deep control model as an additional conditional constraint module to perform hand and foot image repair.

2. The method for restoring the hands and feet of a portrait based on an image generation model as claimed in claim 1, characterized in that: The correct hand and foot detail image is a torso image that only includes hands or feet, and the torso image of hands or feet includes close-up images of various commonly used hand gestures or close-up images of various commonly used foot postures.

3. The method for restoring the hands and feet of a portrait based on an image generation model as claimed in claim 1, characterized in that: The resolution of the correct hand and foot detail image and the correct hand and foot thumbnail image is 512*512 pixels to 7680x4320 pixels.

4. The method for restoring the hands and feet of a portrait based on an image generation model as claimed in claim 1, characterized in that: In step S2, the step of training the human hand and foot shape mesh reconstruction module by using the correct hand and foot detail image specifically includes: The correct hand and foot detail image is masked with a random mask algorithm, and the corresponding hand and foot defect image is generated through a conventional diffusion generation model; Then, the correct hand and foot detail images are used to generate corresponding hand and foot mesh images through the existing mesh reconstruction model; Then, the hand and foot defect images and the corresponding hand and foot mesh images are paired to form a training data set for training a human hand and foot shape mesh reconstruction module.

5. The method for restoring the hands and feet of a portrait based on an image generation model as claimed in claim 1, characterized in that: In step S6, the DIT structure is used as the diffusion model for hand and foot redrawing, and the previously pre-trained Controlnet deep control model is used as an additional conditional constraint module to perform the steps of hand and foot image repair, which specifically include: According to the trained human hand and foot shape mesh reconstruction module, the depth map of the reconstructed hand and foot parts and the white mask of the human hand and foot area are obtained; Through mask inversion, the white mask of the human hand and foot area is inverted into a black mask to cover the corresponding area in the input original image to generate a mask image; The hand and foot redrawing inversion mask and the character's hand and foot redrawing mask are used as the input of the DIT diffusion model, and the repaired hand and foot area depth map is used as the control signal of the Controlnet deep control model; the control signal transmits the control information to the attention layer of the DIT structure through the cross-attention mechanism, thereby guiding the model to repair and generate the correct hand and foot areas.

6. A device for repairing human portrait hands and feet based on an image generation model, characterized in that: include: processor; A memory for storing processor-executable instructions; wherein the processor is configured to call the instructions stored in the memory to execute the method according to any one of claims 1 to 5.

7. A computer-readable storage medium, characterized in that: A computer program is stored, and the computer program can be executed by a processor of the device where the computer-readable storage medium is located to implement the method according to any one of claims 1 to 5.

Citation Information

Patent Citations

  • Digital image restoration method, apparatus and device, and readable storage medium

    CN114511463A

  • Image processing method and device and computer readable storage medium

    CN117649335A

  • Image restoration method, figure image restoration method, electronic equipment and storage medium

    CN118840295A

  • Techniques for generating images of object interactions

    US20240161468A1

Cited By

  • Image inpainting method and apparatus, and image inpainting model training method

    WO2026144301A1