Hair drawing method and system based on diffusion model
By using diffusion and neural network models in hair rendering, combined with user interaction and hair segmentation technology, the process of hair rendering is simplified and the results are naturally consistent. This solves the problems of complex operation and unsatisfactory generation in existing technologies and supports user customization needs.
Patent Information
- Application Number
- CN202511083629.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-08-04
- Publication Date
- 2025-11-14
AI Technical Summary
Existing technologies are complex to operate when drawing hair, and the generated results are not ideal. They are difficult to support the flexibility and customization needs of unconventional hairstyles and hair colors. Furthermore, they perform poorly under special lighting conditions, and the generated results are too random and have obvious color differences.
By obtaining the user's smearing instructions on the original image, the target area for drawing hair is determined, a mask image is generated, the hair mask image of the original image is extracted using a hair segmentation model, and hair color filling and control injection are performed in combination with a diffusion model. A multi-layer neural network model is used for training and correction to achieve precise control over the hair shape and color of specific areas.
It simplifies the hair drawing process, generates results with good consistency with the original image, and blends colors naturally. It supports customized hairstyles and hair colors according to user needs, avoiding randomness and color difference issues in the generated results.
Smart Images
Figure CN120953418A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of image processing technology, and in particular to a hair rendering method and system based on a diffusion model. Background Technology
[0002] With the development of modern digital image processing and mobile devices, users have higher requirements for the hairstyle effects in portrait photos, making hair drawing a hot demand in image processing. When the hairstyle of the person being photographed does not meet aesthetic requirements, it is necessary to draw hair in the portrait image in post-production to optimize the hairstyle or hair volume; or in character simulation (cosplay), it is desirable to show a gravity-defying flowing hairstyle, but this is difficult to achieve with shooting props, so it relies on hair drawing.
[0003] However, hair drawing is often limited by technical means, making it difficult and time-consuming. Manual drawing using traditional image processing software like Photoshop is complex and requires high-level drawing skills, especially when drawing large areas of hair, where the time cost is particularly significant. Generating hair using Generative Adversarial Networks (GANs) often faces training instability issues, such as pattern collapse and difficulty in convergence; furthermore, it struggles to support the flexibility and customization needs of unconventional hairstyles and colors, and performs poorly in special lighting conditions. While hair based on diffusion models can generally meet the realism requirements, the generated results are too random, unable to stably generate hair in specific areas according to user needs, and often exhibits color differences between the generated hair and the original hair color, resulting in an unnatural and unattractive effect. Summary of the Invention
[0004] To overcome, to some extent, the problems of complex operation and unsatisfactory drawing results when drawing hair in portrait images in related technologies, this application provides a hair drawing method and system based on a diffusion model.
[0005] The proposed solution is as follows:
[0006] According to a first aspect of the embodiments of this application, a hair rendering method based on a diffusion model is provided, comprising:
[0007] Get the original image;
[0008] The target area for drawing hair is determined based on the smearing instructions entered by the user on the original image, and a mask image of the target area for drawing hair is generated.
[0009] The original image is input into the hair segmentation model to obtain the original hair mask image;
[0010] The target area for hair in the original image is filled to obtain the hair color filling result image of the mask area;
[0011] Based on the target area mask image and the hair color filling result image of the mask area, the diffusion model hair drawing control injection conditions are obtained;
[0012] The original image and the hair drawing control injection conditions of the diffusion model are input into the diffusion model, so that the diffusion model redraws the area to be drawn in the original image as hair according to the hair drawing control injection conditions of the diffusion model, and the hair drawing result image of the diffusion model is obtained.
[0013] Using the hair area indicated by the original hair mask image in the original image as a hair color reference, the hair color indicated by the hair target area mask image in the hair drawing result image of the diffusion model is corrected;
[0014] Output the resulting image of the hair after color correction.
[0015] Preferably, filling the target area in the original image includes:
[0016] For the pixel in the target area of the hair in the original image that is currently to be colored, mark the original hair color sampling point in the hair mask image of the nearby original image;
[0017] Calculate the weight values of the original hair color sampling points marked;
[0018] The hair color of all marked original hair color sampling points is weighted and averaged according to the weights, and the weighted average result is used as the fill color value of the current pixel to be colored.
[0019] Preferably, the closer the original hair color sampling point is to the pixel to be colored, the higher its color weight.
[0020] Preferably, the method further includes:
[0021] Obtain first training data with matching correspondences; the first training data includes: a set of real portrait images with multiple hairstyles, a set of hair mask images, a set of portrait images with filled hair color, and a set of portrait image description text, as well as a first matching correspondence table;
[0022] Construct the first neural network model;
[0023] The first neural network model is trained using the first training data until the index converges, thus obtaining the conditional injection model for drawing.
[0024] Preferably, the method further includes:
[0025] Input the target area mask image and the hair color fill result image of the mask area into the hair drawing condition injection model;
[0026] The hair drawing condition injection model outputs a diffusion model to control the hair drawing injection conditions.
[0027] Preferably, the method further includes:
[0028] Obtain second training data with matching correspondences; the second training data includes: a set of hair images and a set of text descriptions of hair images, as well as a second matching correspondence table;
[0029] Construct a second neural network model;
[0030] The second neural network model is trained using the second training data until the index converges, thus obtaining the diffusion model.
[0031] Preferably, the method further includes:
[0032] Obtain third training data with matching correspondences; the third training data includes: a set of original hair images, a set of original hair mask images, a set of reference hair images, a set of reference hair mask images, and a set of hair color reference dyeing result images, as well as a third matching correspondence table;
[0033] Construct a third neural network model;
[0034] The third neural network model is trained using the third training data until the index converges, thus obtaining a hair color reference dyeing model.
[0035] Preferably, the method further includes:
[0036] Input the original image, the original hair mask image, the hair drawing result image of the diffusion model, and the hair drawing target area mask image into the hair color reference dyeing model;
[0037] The hair color reference hair dyeing model uses the hair area indicated by the original hair mask image in the original image as the hair color reference to correct the hair color of the target area indicated by the mask image in the hair drawing result image of the diffusion model.
[0038] Preferably, the first neural network model, the second neural network model, and the third neural network model are constructed in different ways.
[0039] According to a second aspect of the embodiments of this application, a hair rendering system based on a diffusion model is provided, comprising:
[0040] Processor and memory;
[0041] The processor and memory are connected via a communication bus:
[0042] The processor is used to call and execute the program stored in the memory;
[0043] The memory is used to store a program, which is at least used to execute a hair drawing method based on a diffusion model as described in any of the above.
[0044] The technical solution provided in this application may include the following beneficial effects: The hair drawing method based on the diffusion model in this application includes: acquiring an original image; determining the target area for drawing hair based on the smearing instructions entered by the user on the original image, and generating a mask image of the target area for drawing hair; inputting the original image into a hair segmentation model to obtain the original image hair mask image; filling the target area for drawing hair in the original image to obtain a hair color filling result image of the mask area; obtaining the diffusion model hair drawing control injection conditions based on the target area mask image and the mask area hair color filling result image; inputting the original image and the diffusion model hair drawing control injection conditions into the diffusion model, so that the diffusion model redraws the area to be drawn in the original image as hair according to the diffusion model hair drawing control injection conditions, and obtains the diffusion model hair drawing result image; using the hair area indicated by the original image hair mask image as a hair color reference, correcting the hair color indicated by the target area mask image in the diffusion model hair drawing result image; and outputting the hair drawing result image after correcting the hair color.
[0045] In this technical solution, users manually select the desired hair drawing location (e.g., a hair replacement area or a special hairstyle area) through interactive means such as clicking and smearing, generating a corresponding hair drawing target area mask image. A pre-trained hair segmentation model performs semantic segmentation on the original image, automatically extracting existing hair areas to form the original image's hair mask image. Based on the original image's hair color, the user-specified hair drawing target area mask image is color-filled to improve the color consistency of the generated hair. By combining the hair drawing target area mask image and the mask area hair color-filled result image, a hair drawing control injection condition for the diffusion model is constructed, achieving precise control over the hair shape and color of specific areas, effectively avoiding the problem of excessive randomness in the diffusion model's generated results. Simultaneously, using the hair area indicated by the original image's hair mask image as a hair color reference, the hair color indicated by the hair drawing target area mask image in the diffusion model's hair drawing result image is corrected, ensuring that the newly added hair blends naturally with the original image in terms of texture detail, lighting consistency, and tone harmony.
[0046] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description
[0047] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0048] Figure 1 This is a schematic flowchart of a hair rendering method based on a diffusion model provided in one embodiment of this application;
[0049] Figure 2 This is a schematic diagram of the structure of a hair rendering system based on a diffusion model provided in one embodiment of this application.
[0050] Reference numerals: Processor-21; Memory-22. Detailed Implementation
[0051] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.
[0052] Figure 1 This is a flowchart illustrating a hair rendering method based on a diffusion model, provided in one embodiment of this application. (Refer to...) Figure 1 A hair rendering method based on a diffusion model includes:
[0053] S1: Get the original image;
[0054] The user provides an original image of a person as the input for drawing the hair. The original image is a raw image without any processing.
[0055] S2: Determine the target area for drawing hair based on the smearing instructions entered by the user on the original image, and generate a mask image of the target area for drawing hair;
[0056] mask Figure 1 Generally refers to a mask image.
[0057] Users can manually select the location where they want to add hair (such as a hair replacement area or a special hairstyle area) through interactive methods such as clicking and smearing, and generate a corresponding hair target area mask image.
[0058] S3: Input the original image into the hair segmentation model to obtain the original hair mask image;
[0059] A pre-trained hair segmentation model is used to perform semantic segmentation on the original image, automatically extracting existing hair regions to form a mask image. Automated segmentation using this hair segmentation model reduces manual annotation work for users and provides a structural basis for subsequent hair color extraction, tone matching, and style unification.
[0060] S4: Fill the target area of the hair in the original image to obtain the hair color filling result image of the mask area;
[0061] The newly added hair area specified by the user is filled with color. In this embodiment, the filling is based on the hair area of the adjacent original image to improve the color consistency of the generated hair.
[0062] S5: Based on the mask image of the target area and the hair color filling result image of the mask area, obtain the injection conditions for hair control in the diffusion model;
[0063] S6: Input the original image and the hair drawing control injection conditions of the diffusion model into the diffusion model, so that the diffusion model can redraw the area to be drawn in the original image as hair according to the hair drawing control injection conditions of the diffusion model, and obtain the hair drawing result image of the diffusion model.
[0064] The diffusion model hair drawing control injection condition is used to effectively guide the diffusion model to redraw and color the hairstyle within a specified area.
[0065] By combining the target area mask map and the result map of hair color filling in the mask area, the hair drawing control injection conditions for the diffusion model are constructed, which realizes precise control over the hair shape and color in specific areas and effectively avoids the problem of excessive randomness in the generated results of the diffusion model.
[0066] S7: Using the hair area indicated by the original hair mask image in the original image as a hair color reference, correct the hair color indicated by the hair target area mask image in the hair drawing result image of the diffusion model;
[0067] The hair rendering output by the diffusion model is corrected to ensure that the hair color is similar to the original image or that the style is consistent.
[0068] S8: Output the result image of the hair color after correction.
[0069] It should be noted that filling the target area in the original image includes:
[0070] For the pixel in the target area of the hair in the original image that is currently to be colored, mark the sampling point of the original hair color in the hair mask image of the nearby original image;
[0071] Calculate the weight values of the original hair color sampling points marked;
[0072] The hair color of all marked original hair color sampling points is weighted and averaged according to the weights, and the weighted average result is used as the fill color value of the current pixel to be colored.
[0073] The fill color value of each pixel in the target hair region mask image comes from the original hair color sampling point marked by its neighboring original hair mask image. The closer the original hair sampling point is to the pixel to be colored, the higher its weight, which is calculated by a Gaussian function. Finally, the hair colors of all sampling points are weighted and averaged according to their weights to fill the target hair region with a color that matches the original image's hair color, resulting in the mask region hair color filling result image. In this result image, the mask portion corresponding to the target hair region mask image is filled with the matching hair color.
[0074] Example 2
[0075] It should be noted that the method also includes:
[0076] Obtain first training data with matching correspondences; the first training data includes: a set of real portrait images with multiple hairstyles, a set of hair mask images, a set of portrait images with filled hair color, and a set of portrait image description text, as well as a first matching correspondence table;
[0077] Construct the first neural network model;
[0078] The first neural network model is trained using the first training data until the index converges, thus obtaining the conditional injection model for drawing.
[0079] Based on this, the method also includes:
[0080] Input the target area mask image and the mask area hair color fill result image into the hair drawing conditions and inject them into the model;
[0081] The injection model outputs a diffusion model that controls the injection conditions by drawing hair.
[0082] It should be noted that the four sets of data in the first training data have the same number of data points. The first matching correspondence table is used to specify the correspondence between the four sets of data in the first training data. For example, if the real portrait image A1, the hair mask image A2, the portrait image A3 with filled hair color, and the portrait image text description A4 are matched and corresponded, then part or all of the hair area of image A1 (reference image A01) is indicated pixel by pixel by image A2 (reference image A02); the hair area in image A3 (reference image B01) indicated by image A2 is filled with hair color; and the portrait image text description A4 describes the content of image A1.
[0083] Example 3
[0084] It should be noted that the method also includes:
[0085] Obtain second training data with matching correspondences; the second training data includes: a set of hair images and a set of text descriptions of hair images, as well as a second matching correspondence table;
[0086] Construct a second neural network model;
[0087] The second neural network model is trained using the second training data until the index converges, thus obtaining the diffusion model.
[0088] It should be noted that the two sets of data in the second training data have the same number of data points. The second matching correspondence table is used to specify the correspondence between the two sets of data in the second training data. For example, if image A1 in the image group and text A2 in the text description group are matched, then text A2 describes the content of image A1.
[0089] Example 4
[0090] It should be noted that the method also includes:
[0091] Obtain third training data with matching correspondences; the third training data includes: a set of original hair images, a set of original hair mask images, a set of reference hair images, a set of reference hair mask images and a set of hair color reference dyeing result images, as well as a third matching correspondence table;
[0092] Construct a third neural network model;
[0093] The third neural network model is trained using the third training data until the index converges, thus obtaining the hair color reference dyeing model.
[0094] Based on this, the method also includes:
[0095] Input the original image, the original hair mask image, the hair drawing result image from the diffusion model, and the hair drawing target area mask image into the hair color reference dyeing model;
[0096] The hair color reference hair dyeing model uses the hair area indicated by the original hair mask map in the original image as the hair color reference to correct the hair color of the target area indicated by the mask map in the hair drawing result image of the diffusion model.
[0097] It should be noted that the five sets of data in the third training data have the same number of data points. The third matching correspondence table is used to specify the correspondence between the five sets of data in the third training data. For example, if the original hair image group A1, the original hair mask image group A2, the reference hair image group A3, the reference hair mask image group A4, and the hair color reference dyeing result image group A5 are matched and corresponded, then the hair area of image A1 is pointed out pixel by pixel by image A2, the hair area of image A3 is pointed out pixel by pixel by image A4, and image A5 is the dyed hair result image obtained by dyeing the hair in the area of image A2 of image A1 with the hair color of the area shown in image A4 of image A3 as the reference hair color.
[0098] It should be noted that the first, second, and third neural network models are constructed in different ways.
[0099] Preferably, the first neural network model is a neural network model structure implemented by reconstructing and modifying ControlNet technology. In this embodiment, the specific implementation details of the first neural network model are not limited. The trained hair-drawing condition injection model can apply control conditions to the specific position and color of the hair drawn by the diffusion model, controlling the process of the diffusion model generating and drawing the hair.
[0100] Preferably, the second neural network model is a neural network model structure reconstructed and modified based on the diffusion model. This embodiment does not limit the specific implementation details of the second neural network model. The trained diffusion model is used to generate corresponding hairstyles based on input mask images of arbitrary shapes. Even if the user inputs a mask image of flowing hair in an anime style that defies gravity, it can stably redraw natural and beautiful hair without redrawing the corresponding areas as clothing, background, or other elements.
[0101] Preferably, the third neural network model is a convolutional neural network (CNN) model structure. This embodiment does not limit the specific implementation details of the third neural network model. The hair color reference dyeing model is used to extract the hair color as a reference color based on the input hair color reference image and its reference area, and then dye the area to be dyed using the input primary color.
[0102] Example 5
[0103] Figure 2 This is a schematic diagram of the structure of a hair rendering system based on a diffusion model provided in one embodiment of this application, with reference to... Figure 2 A hair rendering system based on a diffusion model, comprising:
[0104] Processor 21 and memory 22;
[0105] Processor 21 and memory 22 are connected via a communication bus:
[0106] The processor 21 is used to call and execute the program stored in the memory 22;
[0107] The memory 22 is used to store a program, which is used to execute at least one of the hair drawing methods based on a diffusion model in the above embodiments.
[0108] It is understood that the same or similar parts in the above embodiments can be referred to each other, and the contents not described in detail in some embodiments can be referred to the same or similar contents in other embodiments.
[0109] It should be noted that in the description of this application, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this application, unless otherwise stated, "a plurality of" means at least two.
[0110] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the function involved, as will be understood by those skilled in the art to which embodiments of this application pertain.
[0111] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.
[0112] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.
[0113] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.
[0114] The storage media mentioned above can be read-only memory, disk, or optical disk, etc.
[0115] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.
[0116] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.
Claims
1. A hair rendering method based on a diffusion model, characterized in that, include: Get the original image; The target area for drawing hair is determined based on the smearing instructions entered by the user on the original image, and a mask image of the target area for drawing hair is generated. The original image is input into the hair segmentation model to obtain the original hair mask image; The target area for hair in the original image is filled to obtain the hair color filling result image of the mask area; Based on the target area mask image and the hair color filling result image of the mask area, the diffusion model hair drawing control injection conditions are obtained; The original image and the hair drawing control injection conditions of the diffusion model are input into the diffusion model, so that the diffusion model redraws the area to be drawn in the original image as hair according to the hair drawing control injection conditions of the diffusion model, and the hair drawing result image of the diffusion model is obtained. Using the hair area indicated by the original hair mask image in the original image as a hair color reference, the hair color indicated by the hair target area mask image in the hair drawing result image of the diffusion model is corrected; Output the resulting image of the hair after color correction.
2. The method according to claim 1, characterized in that, Filling the target area in the original image includes: For the pixel in the target area of the hair in the original image that is currently to be colored, mark the original hair color sampling point in the hair mask image of the nearby original image; Calculate the weight values of the original hair color sampling points marked; The hair color of all marked original hair color sampling points is weighted and averaged according to the weights, and the weighted average result is used as the fill color value of the current pixel to be colored.
3. The method according to claim 2, characterized in that, The closer the original hair color sampling point is to the pixel to be colored, the higher its color weight.
4. The method according to claim 1, characterized in that, The method further includes: Obtain first training data with matching correspondences; the first training data includes: a set of real portrait images with multiple hairstyles, a set of hair mask images, a set of portrait images with filled hair color, and a set of portrait image description text, as well as a first matching correspondence table; Construct the first neural network model; The first neural network model is trained using the first training data until the index converges, thus obtaining the conditional injection model for drawing.
5. The method according to claim 4, characterized in that, The method further includes: Input the target area mask image and the hair color fill result image of the mask area into the hair drawing condition injection model; The hair drawing condition injection model outputs a diffusion model to control the hair drawing injection conditions.
6. The method according to claim 4, characterized in that, The method further includes: Obtain second training data with matching correspondences; the second training data includes: a set of hair images and a set of text descriptions of hair images, as well as a second matching correspondence table; Construct a second neural network model; The second neural network model is trained using the second training data until the index converges, thus obtaining the diffusion model.
7. The method according to claim 6, characterized in that, The method further includes: Obtain third training data with matching correspondences; the third training data includes: a set of original hair images, a set of original hair mask images, a set of reference hair images, a set of reference hair mask images, and a set of hair color reference dyeing result images, as well as a third matching correspondence table; Construct a third neural network model; The third neural network model is trained using the third training data until the index converges, thus obtaining a hair color reference dyeing model.
8. The method according to claim 7, characterized in that, The method further includes: Input the original image, the original hair mask image, the hair drawing result image of the diffusion model, and the hair drawing target area mask image into the hair color reference dyeing model; The hair color reference hair dyeing model uses the hair area indicated by the original hair mask map in the original image as the hair color reference to correct the hair color of the target area indicated by the mask map in the hair drawing result image of the diffusion model.
9. The method according to claim 7, characterized in that, The first, second, and third neural network models are constructed in different ways.
10. A hair rendering system based on a diffusion model, characterized in that, include: Processor and memory; The processor and memory are connected via a communication bus: The processor is used to call and execute the program stored in the memory; The memory is used to store a program, which is at least used to execute a hair drawing method based on a diffusion model as described in any one of claims 1-9.