Method for modifying portrait part based on image detection and diffusion technology and electronic equipment

Abstract

The invention discloses a method for modifying a portrait part based on an image detection and diffusion technology and electronic equipment. According to the method, mask recognition detection and mask cutting are carried out by adopting YoloWorld and Effcient SAM, meanwhile, the obtained mask result is input into a potential diffusion model, resampling and noise reduction are carried out, the size of a human body part better meeting the requirement is obtained, and the whole process is automatically operated through one key. The method solves the problem that a set of automatic figure repairing method does not exist in the field of portrait picture repairing at present, and the set of method achieves the purpose that a perfect figure scheme can be automatically selected through one key in the picture beautifying direction, the use experience feeling of a user is improved, and a previous method that a plurality of platform software PS is needed for figure modification is replaced.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method and electronic equipment for modifying a portrait part based on image detection and diffusion technology. Background Art

[0002] When it is necessary to retouch a certain part of the portrait in the image, such as the chest, the existing portrait retouching software technologies, such as Photoshop and Meitu XiuXiu, etc., require manual selection of the body shape, manual editing of the chest position, and the use of liquefaction tools, push-pull, twist and other methods to push the pixels of the picture to form a visual enlargement and beautification effect. This effect will result in obvious traces of retouching and is difficult to operate manually. Without the basic use of Photoshop, Meitu XiuXiu and other retouching software, it is difficult to achieve the desired effect.

[0003] Therefore, the prior art has defects and needs to be improved. Summary of the invention

[0004] The technical problem to be solved by the present invention is: to provide a method and electronic device for modifying portrait parts based on image detection and diffusion technology, to realize automated photo editing, to increase the user experience, and to replace the existing method of photo editing that requires multiple platform software.

[0005] The technical solution of the present invention is as follows: a method for modifying a portrait part based on image detection and diffusion technology is provided, comprising the following steps.

[0006] S1: Use Yolo World technology to detect the body mask of the portrait in the image.

[0007] S2: For the parts of the portrait that need to be modified, the Yolo World technology is used to once again detect the masks of the parts of the portrait in the image that do not need to be modified.

[0008] S3: Use the open source technology EfficientSAM to segment and subtract the two detected masks to obtain a mask of the part that needs to be modified.

[0009] S4: Use the diffusion model to apply the obtained mask of the part that needs to be modified, and obtain a first text prompt word, where the first text prompt word is: the part described by the descriptive word. The diffusion model resamples and reduces the noise of the mask area of ​​the part that needs to be modified, and generates a mask that conforms to the part described by the descriptive word.

[0010] S5: Replace the original unmodified part with the mask of the part that meets the description of the descriptor, and output the modified image.

[0011] Furthermore, in step S1, the second text prompt word is input to allow the Yolo World technology to detect the body mask of the portrait; in step S2, the third text prompt word is input to allow the Yolo World technology to detect the masks of the positions of the various parts of the portrait that do not need to be modified.

[0012] Furthermore, the first text prompt word, the second text prompt word, and the third text prompt word are all implemented using the prompt method in the javascript language.

[0013] Furthermore, in step S4, the diffusion model uses the latent space to resample and denoise the mask area of ​​the position of the part to be modified to generate a part that meets the description of the descriptor.

[0014] Furthermore, the method is implemented in the open source framework ComfyUI.

[0015] Furthermore, in step S5, the mask of the part that meets the description of the descriptor is combined with the mask of each part that does not need to be modified obtained in step S2 to obtain a modified image.

[0016] Furthermore, the present invention also provides an electronic device, comprising: a processor; a GPU graphics card connected to the processor for communication; a memory connected to the processor for communication; an application, wherein the application is stored in the memory and configured to be executed by the processor, and the program is configured to execute the aforementioned method for modifying portrait parts based on image detection and diffusion technology; the GPU graphics card is used to process images.

[0017] Furthermore, the GPU graphics card adopts CUDA architecture and the memory adopts at least 16G VRAM.

[0018] By adopting the above scheme, the present invention provides a method and electronic device for modifying the parts of a portrait based on image detection and diffusion technology, which solves the gap in the field of automatic body modification for portrait beautification in the existing image retouching field, and solves the automatic processing of partial beautification of a portrait. It can accurately realize the recognition of the parts of the portrait, and regenerate a more perfect body through the ability of text generation of the diffusion model. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0020] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0021] See also Figure 1 The present invention provides a method for modifying a portrait part based on image detection and diffusion technology, taking the modification of the chest of a lady's portrait as an example, comprising the following steps.

[0022] S1: Use Yolo World technology to detect the body mask of the portrait in the image.

[0023] S2: For the parts of the portrait that need to be modified, the Yolo World technology is used to once again detect the masks of the parts of the portrait in the image that do not need to be modified (such as the hair, face, arms, legs, etc. of the lady's portrait).

[0024] S3: Use the open source technology EfficientSAM to segment and subtract the two detected masks to obtain a mask of the part that needs to be modified, that is, the mask of the chest of the lady portrait is retained.

[0025] S4: Using the diffusion model (latent diffusion model) on the mask of the part to be modified (i.e., the mask of the chest of the lady's portrait), and obtaining a first text prompt word, wherein the first text prompt word is: the part described by the descriptive word (e.g., a plump chest), the diffusion model resamples and reduces the noise of the mask area of ​​the part to be modified (i.e., the mask of the chest of the lady's portrait), and generates a mask of the part that meets the description of the descriptive word.

[0026] S5: Replace the original unmodified part with the mask of the part that meets the description of the descriptor, and output the modified image.

[0027] In this embodiment, in step S1, the second text prompt word is input to allow the Yolo World technology to detect the body mask of the portrait; in step S2, the third text prompt word is input to allow the Yolo World technology to detect the masks of the positions of the various parts of the portrait that do not need to be modified. In this embodiment, the second text prompt word is: detect the body mask of the portrait, and the third text prompt word is: detect the portrait mask except the chest UV.

[0028] In this embodiment, the first text prompt word, the second text prompt word, and the third text prompt word are all implemented using the prompt method in the javascript language.

[0029] In this embodiment, in step S4, the diffusion model uses the latent space to resample and reduce the noise of the mask area at the location of the part to be modified to generate a part that meets the description of the descriptor.

[0030] In this embodiment, the method is implemented in the open source framework ComfyUI.

[0031] In this embodiment, in step S5, the mask of the part that meets the description of the descriptor is combined with the mask of each part that does not need to be modified obtained in step S2 to obtain a modified image.

[0032] Specifically, the Yolo World technology is used to detect the entire body mask of the lady, and then the masks of the hair, face, arms, legs and other positions of the lady's portrait are detected again. The open source technology EfficientSAM is used to segment and subtract the two detected masks to obtain a mask that only retains the chest position. The obtained chest position mask uses a diffusion model (latent diffusion model) to resample and reduce the noise of the mask area (the detected and retained chest position of the lady) by inputting text prompt words to generate a chest that meets the description of the word, such as inputting a plump chest. The entire set of operation processes is implemented in an open source architecture ComfyUI. You only need to upload a photo and enter the prompt word to get the final effect.

[0033] The present invention has the following technical effects: (1) The operation method is simple. For the entire workflow, you only need to modify the uploaded image, input the accurate prompt word text, and click to confirm to start. (2) It is fast because it uses YoloWorld and EffcientSAM's mask recognition detection and mask cropping, and the mask results obtained are input into the potential diffusion model for resampling and noise reduction to obtain a chest size that better meets the requirements. The entire process is automatically run with one click, and the overall time is less than 1 minute (hardware: RTX 4090 24G VRAM). (3) It is highly accurate and authentic. Thanks to the maturity of the mask detection technology and the potential diffusion model, the technology of the entire workflow method is more stable and accurate. Except for the chest position, which will become larger (or smaller, depending on the prompt word input control), other parts of the entire image will not be deformed.

[0034] The present invention also provides an electronic device, comprising: a processor; a GPU graphics card connected to the processor in communication; a memory connected to the processor in communication; an application program, wherein the application program is stored in the memory and configured to be executed by the processor, and the program is configured to execute the aforementioned method for modifying the portrait part based on image detection and diffusion technology; the GPU graphics card is used to process the image. The GPU graphics card adopts the CUDA architecture, and the memory adopts at least 16G VRAM.

[0035] In summary, the present invention provides a method and electronic device for modifying the parts of a portrait based on image detection and diffusion technology, which solves the gap in the field of automatic body modification for portrait beautification in the existing image retouching field, and solves the automatic processing of partial beautification of portraits. It can accurately recognize the parts of the portrait, and regenerate a more perfect body through the ability of text generation of the diffusion model.

[0036] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for modifying a portrait part based on image detection and diffusion technology, characterized in that: The following steps are involved: S1: Use Yolo World technology to detect the body mask of the portrait in the image; S2: For the parts of the portrait that need to be modified, the Yolo World technology is used to detect the masks of the parts of the portrait in the image that do not need to be modified again; S3: Use the open source technology EfficientSAM to segment and subtract the two detected masks to obtain a mask of the part that needs to be modified; S4: using a diffusion model to apply the obtained mask of the part to be modified, and obtaining a first text prompt word, wherein the first text prompt word is: the part described by the descriptive word, and the diffusion model resamples and reduces noise on the mask area of ​​the part to be modified, and generates a mask that meets the description of the part; S5: Replace the original unmodified part with the mask of the part that meets the description of the descriptor, and output the modified image.

2. The method for modifying a portrait part based on image detection and diffusion technology according to claim 1, characterized in that: In step S1, the second text prompt word is input to allow the Yolo World technology to detect the body mask of the portrait; in step S2, the third text prompt word is input to allow the Yolo World technology to detect the masks of the positions of the various parts of the portrait that do not need to be modified.

3. The method for modifying a portrait part based on image detection and diffusion technology according to claim 2, characterized in that: The first text prompt word, the second text prompt word and the third text prompt word are all implemented using the prompt method in the javascript language.

4. The method for modifying a portrait part based on image detection and diffusion technology according to claim 1, characterized in that: In step S4, the diffusion model uses the latent space to resample and denoise the mask area of ​​the position of the part to be modified to generate a part that meets the description of the descriptor.

5. The method for modifying a portrait part based on image detection and diffusion technology according to claim 1, characterized in that: This method is implemented in the open source framework ComfyUI.

6. The method for modifying a portrait part based on image detection and diffusion technology according to claim 1, characterized in that: In step S5, the mask of the part that meets the description of the descriptor is combined with the masks of the parts that do not need to be modified obtained in step S2 to obtain a modified image.

7. An electronic device, characterized in that: include: processor; A GPU graphics card communicatively connected to the processor; a memory communicatively coupled to the processor; An application program, wherein the application program is stored in the memory and configured to be executed by the processor, and the program is configured to execute the method for modifying a portrait part based on image detection and diffusion technology according to any one of claims 1 to 6; The GPU graphics card is used to process images.

8. An electronic device according to claim 7, characterized in that: The GPU graphics card adopts CUDA architecture and the memory adopts at least 16G VRAM.

Images