Generation and optimization method for hair slice tangential chartlet based on diffusion model
Through the tangential map generation and optimization method of hairpin based on diffusion model, combined with the guidance of Gabor Filter, the limitations of map recognition and completion in the existing technology are solved, the quality and efficiency of the texture are improved, and it is suitable for different types of hairpin. Stickers.
Patent Information
- Application Number
- CN202411143432.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-20
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2044-08-20
AI Technical Summary
The prior art has limitations in the generation and optimization of tangential maps of hairpin models, especially in the case of changes in hair thickness and overlapping, Gabor Filter is difficult to accurately identify and complete information, resulting in missing maps and poor quality.
Using a diffusion model-based method, the model is trained through a custom data set and combined with the guidance of Gabor Filter, the tangential map of the truncated sheet is generated and optimized, and the noise prediction module and the sampling module are used for reverse inference and gradient descent optimization to ensure that the texture quality is in line with the art. Pipeline requirements.
It realizes the completion of missing art resources in the hairpin model, improves the quality and utilization efficiency of the texture, meets the high-level needs of the hair model, and is suitable for different types of hairpin maps.
Smart Images

Figure CN119048625A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer technology, and in particular to a method for generating and optimizing a hair tangent map based on a diffusion model. Background Art
[0002] The hair model is modeled and expressed by using a hair piece model in combination with a hair piece map, and is applied to the three-dimensional character models of common game engines (Unreal Engine, Unity Engine). This is one of the mainstream solutions for building character hair models in the industry.
[0003] Since different hair models are adapted to different art pipelines and uses, some textures are often not saved or missing when using resources, especially the tangent map of the hair. However, in many hair model usage methods, this type of map is necessary, so a solution is needed to generate or optimize the tangent map of the existing hair.
[0004] So far, some solutions based on image filters have been able to initially solve such problems, such as GaborFilter. Gabor transform is essentially a special case of short-time Fourier transform, in which the window function used is a Gaussian function. Fourier transform is performed by applying a sliding time window to different time periods of the signal to analyze the local spectral characteristics of the signal. Gabor Filter can encode features in images at different scales and directions, and has shown good performance in texture analysis, pattern recognition, and computer vision tasks, and can capture and distinguish certain key visual features.
[0005] However, Gabor Filter has great limitations in tangential recognition of hair patch maps, because each hair in the hair patch map occupies multiple pixels, and the number of pixels occupied varies greatly with the thickness of the hair, which brings great difficulties to the parameter adjustment of Gabor Filter. Secondly, since the hair patch map only contains the information of the hair after single-view orthogonal projection, when the hair is thick, a large number of hairs completely overlap, and the information loss of the overlapping part is serious. The Gabor Filter cannot get the correct solution, so a model with data prior is needed to correct the results of Gabor Filter and complete the information.
[0006] The diffusion model is currently the mainstream model in the field of 2D image generation. It can learn the distribution of a type of data and generate it randomly. It can be considered for this task. However, as a generation model, the diffusion model must solve the problem of how to control the generation results when using it. Summary of the invention
[0007] The purpose of the present invention is to provide a method for generating and optimizing hair tangent maps based on a diffusion model on the basis of the traditional Gabor Filter technology in view of the shortcomings of the prior art. The method uses a customized data set to train the diffusion model so that the model learns a relatively correct hair tangent map data distribution, and uses the results of the Gabor Filter to guide the generation process of the diffusion model, and finally generates a relatively high-quality hair tangent map to meet the data quality requirements of the subsequent art pipeline.
[0008] The objective of the present invention is achieved through the following technical solutions:
[0009] A method for generating and optimizing a hair tangent map based on a diffusion model comprises the following steps:
[0010] Generate hair-piece tangent map guidance data corresponding to the tangent map to be generated or to be optimized; wherein, for the tangent map to be generated, if there is only one corresponding other type of map, then the corresponding hair-piece tangent map guidance data is obtained based on the Gabor Filter; if there are multiple corresponding other types of maps, then a corresponding hair-piece tangent map is generated for each other type of map based on the GaborFilter, and then the direction with the largest value in all directions of the corresponding pixel points of each hair-piece tangent map is weighted averaged to obtain the hair-piece tangent map guidance data of the tangent map to be generated; for the tangent map to be optimized, if there is only the tangent map to be optimized, then the tangent map to be optimized is directly used as the hair-piece tangent map guidance data; if there are other corresponding types of maps, then the tangent map to be optimized is directly used as the hair-piece tangent map guidance data or each other type of map is weighted averaged based on the GaborFilter. The filter generates a corresponding hair-piece tangent map, and then performs weighted average of the direction with the largest value among all directions of the corresponding pixel points of the tangent map to be optimized and each hair-piece tangent map, to obtain the hair-piece tangent map guide data of the tangent map to be optimized;
[0011] A randomly generated Gaussian noise image is input into a trained diffusion model, and reverse inference denoising is performed step by step through the diffusion model, wherein, in each reverse inference process of the diffusion model, the input image of the current step is predicted to have noise and reversely upsampled to obtain a first denoised image of the previous time step, and at the same time, an estimated value of the denoised image of time step 0 is reversely inferred based on the predicted noise, and the derivative of the estimated value and the first-order loss function of the hair tangent map guide data for the denoised image at the current time step is calculated, and the first denoised image of the previous time step obtained by sampling is updated twice in combination with the gradient descent method to obtain a second denoised image of the previous time step; the second denoised image is used as the input image of the diffusion model of the previous time step until the time step is 0, and the final tangent map is obtained.
[0012] The present invention adopts a method of adding tangential guidance and mask guidance in the diffusion model sampling process, and utilizes model priors to regenerate a high-quality tangential map based on existing mask maps and other maps.
[0013] Furthermore, the diffusion model includes a noise prediction module and a sampling module, wherein the noise prediction module is used to predict the noise of the input image of the current step, and the sampling module is used to reversely upsample based on the predicted noise to obtain the first denoised image of the previous time step and reversely infer the estimated value of the denoised image to time step 0.
[0014] Furthermore, the first denoised image of the previous time step is obtained by reverse upsampling based on the predicted noise and the estimated value of the denoised image of the time step 0 is reversely inferred, specifically:
[0015]
[0016] In the formula, represents the noise obtained by the noise prediction module at the current time step, t represents the time step, β T is the diffusion coefficient at the current time step, It is calculated by the diffusion coefficient at different time steps, x t represents the input image at the current time step, is the estimated value of the denoised image at time step 0 estimated by the data at the current time step.
[0017] Furthermore, the main network structure of the noise prediction module is U-net, which includes ResnetBlock and LinearAttention modules.
[0018] Furthermore, the diffusion model is obtained by training based on a collected training data set by minimizing a loss function, and each sample of the training data set includes a hair tangent map.
[0019] Furthermore, the loss function is expressed as follows:
[0020]
[0021] In the formula, ∈ represents random noise, which obeys the normal distribution of N(0,1), t represents the time step, || 2 represents the L2 paradigm; It is calculated by the diffusion coefficient at different time steps, ∈ θ represents the noise prediction module of the diffusion model, ∈ θ (*) represents the output of the noise prediction module of the diffusion model, that is, the predicted noise, and x0 is the hair tangent map.
[0022] Furthermore, the first denoised image obtained by sampling at the previous time step is updated twice by combining the gradient descent method to obtain the second denoised image at the previous time step, specifically:
[0023]
[0024] In the formula, x t ′ -1 represents the first denoised image at the previous time step, t represents the time step, x t represents the input image of the current time step, y represents the hair tangent map guide data obtained in step 3, a represents the update step size, and x t-1 represents the second denoised image at the previous time step, Represents the derivative of the first-order loss function for the image at this time step.
[0025] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, a method for generating and optimizing a hair tangent map based on a diffusion model is implemented.
[0026] A storage medium containing computer executable instructions, which, when executed by a computer processor, can implement a method for generating and optimizing a hair tangent map based on a diffusion model.
[0027] A computer program product includes a computer program / instruction, which, when executed by a processor, implements the steps of a method for generating and optimizing a hair tangent map based on a diffusion model.
[0028] The benefit of the present invention is that the present invention uses a custom data set to train a diffusion model, and by guiding the generation and optimization of hair piece tangent maps, the missing art resources in the hair piece model are completed, and the utilization efficiency of existing art resources is improved. The implementation method is simple, the means are flexible, and the versatility is strong. It can ensure that the hair setting generates the corresponding hair piece model according to the demand, meet the requirements of the high-level sense of the hair model, and thus improve the quality of the hair model. And for different types of hair piece maps, corresponding data sets can be constructed to enhance the generation effect. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] Figure 1 It is a flow chart of the method for generating and optimizing the hair tangent map based on the diffusion model of the present invention;
[0030] Figure 2 It is a framework diagram of a method for generating and optimizing a hair tangent map based on a diffusion model according to an embodiment of the present invention;
[0031] Figure 3This is a diagram of the model output, true value, and guided data results for a specific case of the present invention. The first on the right is the guided data of the model, obtained using a Gabor Filter; the second on the right is the actual hair tangent map in a custom test set, used to verify the model effect; the third on the right is the output of the diffusion model after guidance, i.e., the optimized hair tangent map. DETAILED DESCRIPTION
[0032] In order to enable those skilled in the art to better understand the solution of the present invention, the technical solution in the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is only a part of the embodiment of the present invention, not all of the embodiments. Based on the embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0033] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example: a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0034] like Figure 1 As shown, a method for generating and optimizing a hair tangent map based on a diffusion model of the present invention comprises the following steps:
[0035] Step 1: Generate hair-piece tangent map guidance data corresponding to the tangent map to be generated or to be optimized; wherein, for the tangent map to be generated, if there is only one corresponding other type of map, then the corresponding hair-piece tangent map guidance data is generated based on Gabor Filter; if there are multiple corresponding other types of maps, then a corresponding hair-piece tangent map is generated for each other type of map based on Gabor Filter, and then the direction with the largest value in all directions of the corresponding pixel points of each hair-piece tangent map is weighted averaged to obtain the hair-piece tangent map guidance data of the tangent map to be generated; for the tangent map to be optimized, if there is only the tangent map to be optimized, then the tangent map to be optimized is directly used as the hair-piece tangent map guidance data; if there are other corresponding types of maps, then the tangent map to be optimized is directly used as the hair-piece tangent map guidance data or each other type of map is weighted averaged based on Gabor Filter. The filter generates a corresponding hair-piece tangent map, and then performs weighted average of the direction with the largest value among all directions of the corresponding pixel points of the tangent map to be optimized and each hair-piece tangent map, to obtain the hair-piece tangent map guide data of the tangent map to be optimized;
[0036] Step: 2: Input the randomly generated Gaussian noise image into the trained diffusion model, and perform reverse inference denoising step by step through the diffusion model, wherein, in each reverse inference process of the diffusion model, predict the noise of the input image of the current step and reversely upsample to obtain the first denoised image of the previous time step, and at the same time, reversely infer the estimated value of the denoised image of time step 0 based on the predicted noise, calculate the derivative of the estimated value and the first-order loss function of the hair tangent map guide data for the denoised image at the current time step, and combine the gradient descent method to perform a second update on the sampled first denoised image of the previous time step to obtain the second denoised image of the previous time step; use the second denoised image as the input image of the diffusion model of the previous time step until the time step is 0, and obtain the final tangent map.
[0037] The present invention guides the generation process of the diffusion model by processing the existing original art resources as the corresponding hair tangent map guide data, and finally generates a relatively high-quality hair tangent map to meet the requirements of the subsequent art pipeline for data quality. The effect of the present invention is further described below in conjunction with a specific embodiment.
[0038] like Figure 2 As shown, a method for generating and optimizing a hair piece tangent map based on a diffusion model of the present invention mainly includes two parts, one is to obtain a trained diffusion model; the other is to realize the generation and optimization of a hair piece tangent map based on the trained diffusion model; the following is an example of the process of obtaining a diffusion model through training and realizing the generation and optimization of a hair piece tangent map based on the trained diffusion model for the first time, specifically:
[0039] Step 1: Build a training data set;
[0040] This step is the data basis of the present invention, and the data set is independently constructed so that the diffusion model can learn the data distribution of similar hair tangent maps. This step can be obtained by collecting or building through modeling software; in a specific embodiment, the method of building through modeling software is as follows:
[0041] Batch generate three-dimensional hair that conforms to the common hair texture form through modeling software; convert the three-dimensional hair into strips composed of triangular facets in three-dimensional space; adjust rendering parameters and triangular facet coloring method, and render; organize and store; specifically includes the following sub-steps:
[0042] (1.1) Set the length, quantity, curvature, and diversity of the hair, batch generate 3D hair that conforms to the common hair texture form, and export it from ZBrush, convert it into obj format through blender and save it;
[0043] (1.2) reading the three-dimensional hair in obj format obtained in step (1.1) and converting it into a three-dimensional hair curve expressed by a continuous three-dimensional point list; shifting each hair curve S0 along the same direction to obtain a translated hair curve S1, and interlacing the points on the hair curve S0 before translation and the hair curve S1 after translation. Specifically: connecting a line segment from the first point of the hair curve S0 before translation to the first point of the hair curve S1 after translation, and connecting a line segment from the first point of the hair curve S1 after translation to the second point of the hair curve S0 before translation, repeating this process until there are no points that can be connected, and converting the hair curve into a strip composed of triangular facets;
[0044] (1.3) For the hair tangent map, the color of the triangle vertex is assigned to the tangent unit vector of the three-dimensional hair curve corresponding to the point. The RGB channels correspond to the vectors in three directions respectively, and the tangent map is obtained by rendering;
[0045] Furthermore, data enhancement can be performed on various types of maps in the data set obtained in step (1.3). The enhancement scheme includes randomly intercepting a square block, rotating, translating, and mirroring; the enhanced data are numbered and stored to obtain a training data set.
[0046] Step 2: Build and train the diffusion model;
[0047] The forward diffusion process and reverse sampling process of the traditional diffusion model are used as the overall structure of network training, and the model is trained using the data set constructed in step 1; specifically, the following sub-steps are included:
[0048] (2.1) Build a diffusion model. The diffusion model of the present invention includes a noise prediction module and a sampling module, wherein the noise prediction module is used to predict the noise of the input image of the current step, and the sampling module is used to reverse upsample based on the predicted noise to obtain the first denoised image of the previous time step and reversely infer the estimated value of the denoised image of time step 0. The main network structure of the noise prediction module is U-net, which includes ResnetBlock and LinearAttention modules, and uses U-Net to predict noise; further, the diffusion model of the present invention can also include a module for adding Gaussian noise, thereby realizing the required forward diffusion process and reverse inference process, the forward inference process is realized by the module for adding Gaussian noise, and the reverse inference process is mainly realized by the noise prediction module and the sampling module.
[0049] (2.2) Randomly sample a batch of training data, where each sample of the data is a tangent map of the hair in the data set. For each sample in the batch, randomly sample a time step and a Gaussian noise, and calculate the result of the sample at the time step through the forward diffusion process through the sample, time step, and Gaussian noise, i.e., the noise image;
[0050] (2.3) The result of forward diffusion and the time step are input into the noise prediction module to obtain the predicted noise. The predicted noise is differentiated from the second-order loss of the random noise obtained by real sampling, and the parameters of the neural network are updated using the gradient descent method. The loss function is as follows:
[0051]
[0052] In the formula, ∈ represents random noise, which obeys the normal distribution of N(0,1), t represents the time step, || 2 represents the L2 paradigm; β t is the diffusion coefficient at the current time step, which is set according to actual needs. In this embodiment, it is set to β t ∈{β1,β2,…,…β T}, setting method is in the interval [0.0001,0.02]β t It is proportional to the diffusion step number t. It is calculated by the diffusion coefficient at different time steps, ∈ θ represents the noise prediction module, ∈ θ (*) represents the output of the noise prediction module of the diffusion model, i.e., the predicted noise, and x0 is the hair tangent map in the training data.
[0053] (2.4) A batch of training samples is considered as one round, and several rounds of training are performed until the loss function converges or the set number of training rounds is reached, such as 200,000;
[0054] After obtaining the trained diffusion model, the hair tangent map generation and optimization can be realized.
[0055] Step 3: Generate hair-piece tangent map guide data corresponding to the tangent map to be generated or optimized; the present invention generates hair-piece tangent map guide data using different schemes for three different situations: there is only one kind of map, there are multiple maps, there are not only multiple types of maps but also low-quality tangent maps, i.e., tangent maps to be optimized;
[0056] Step three is one of the core invention parts. According to the existing data of the art resources that need to be completed, different hair tangent map guide data are constructed. The quality and construction method of the guide data will have a significant impact on the subsequent generation effect.
[0057] The step three includes the following sub-steps:
[0058] (3.1) Generate hair tangent map guidance data corresponding to the tangent map to be generated or optimized, wherein, for the tangent map to be generated, for the hair art resource with only one map, identify its tangent through a conventional Gabor Filter-based scheme to obtain the corresponding hair tangent map guidance data; specifically, use a conventional Gabor Filter convolution kernel to convolve the image, calculate the value of each pixel in each direction, and select the direction with the largest value in all directions of each pixel as the possible tangent direction of the hair corresponding to the pixel, and merge them as the hair tangent map guidance data; in this step, if the resolution or image length and width of the art resource that needs to regenerate or optimize the hair tangent map is inconsistent with the training data, the image needs to be segmented and the segmented image is used for operation; if there are multiple maps of different types, all maps are processed according to the scheme in step (3.1), and the direction with the largest value in all directions of the pixel corresponding to each map is weighted averaged to obtain the hair tangent map guidance data;
[0059] (3.2) For the tangent map to be optimized, that is, the art resource with low-quality tangent map, it can be directly used as the tangent map guidance data for the hair-cut according to the specific situation of the tangent map. You can also consider repeating step (3.2): for each other type of map, a corresponding hair-cut tangent map is generated based on the Gabor Filter, and then the direction with the largest value among all directions of the corresponding pixel points of the tangent map to be optimized and each hair-cut tangent map is weighted averaged to obtain the hair-cut tangent map guidance data of the tangent map to be optimized.
[0060] (3.3) For step (3.1), the hair tangent map guide data obtained in step (3.2) is standardized to be consistent with the input data format of the diffusion model;
[0061] Step 4: using the hair tangent map guide data obtained in step 3 to guide the reverse inference process of the diffusion model, so that the hair tangent map generated by the model is highly similar to the hair tangent map to be optimized or generated; after the reverse inference is completed, a hair tangent map that meets the quality requirements is obtained;
[0062] Step four is the second part of the core invention, and the focus is on data guidance in the diffusion model sampling process. This operation transforms the uncontrollable generation task into a controllable image generation and image optimization task.
[0063] The step 4 includes the following sub-steps:
[0064] (4.1) randomly generate a Gaussian noise image with a resolution that matches the training data, input it into the trained diffusion model, and gradually perform reverse inference denoising through the diffusion model. In this embodiment, reverse inference is performed 1000 times with a time step from 1000 to 0;
[0065] (4.2) For each reverse inference, the input image of the current step is used to obtain the predicted noise required for the reverse sampling process at that time step through an inference scheme consistent with the traditional diffusion model, and reverse sampling is performed to obtain the first denoised image of the previous time step.
[0066] (4.3) In step (4.2), based on the predicted noise, the estimated value of the denoised image inversely inferred to time step 0 can be calculated, and the first-order loss function of the estimated value and the hair tangent map guidance data obtained in step (3.4) is calculated as follows:
[0067]
[0068] In the formula, represents the noise obtained by the noise prediction module at the current time step, t represents the time step, It is calculated from the diffusion coefficient at different time steps, x t represents the input image at the current time step, is the estimated value of the denoised image at time step 0 estimated by the predicted noise data at the current time step.
[0069] (4.4) Calculate the derivative of the first-order loss function in step (4.3) for the image at this time step, use the update step size, and use the gradient descent method to perform a second update on the result obtained by reverse sampling. The formula is as follows:
[0070]
[0071] In the formula, x t ′ -1It represents the image of the previous time step obtained by the traditional sampling process at the current time step, that is, the first denoised image of the previous time step, t represents the time step, x t represents the input image of the current time step, y represents the hair tangent map guide data obtained in step 3, a represents the update step size, which is 0.7 in this embodiment, and x t-1 represents the second updated image, that is, the second denoised image of the previous time step, Represents the derivative of the first-order loss function for the image at this time step.
[0072] (4.5) The reverse inference process is repeated repeatedly, from time step 1000 to time step 0, to obtain the final hair tangent map.
[0073] Figure 3 This is a result diagram of the model output, true value and guided data of a specific case of the present invention. The first right figure is the guided data of the model, obtained using Gabor Filter; the second right figure is the real hair tangent map in the test set, which is used to verify the model effect; the third right figure is the output of the diffusion model after guidance, that is, the optimized hair tangent map. It can be seen that compared with the hair tangent map before optimization, the accuracy of the hair growth tangent is greatly improved.
[0074] Corresponding to the aforementioned embodiment of a hair-piece tangential map generation and optimization method based on a diffusion model, the present invention also provides an electronic device, comprising one or more processors, for implementing the aforementioned embodiment of a hair-piece tangential map generation and optimization method based on a diffusion model.
[0075] The electronic device of the present invention may be any device having data processing capability, and the device having data processing capability may be a device or apparatus such as a computer.
[0076] The device embodiment can be implemented by software, hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capability in which it is located to read the corresponding computer program instructions in the non-volatile memory into the memory for execution. From the hardware level, in addition to the processor, memory, network interface, and non-volatile memory, any device with data processing capability in which the device in the embodiment is located can also include other hardware according to the actual function of the device with data processing capability, which will not be described in detail.
[0077] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.
[0078] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.
[0079] An embodiment of the present invention further provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, a method for generating and optimizing a hair tangent map based on a diffusion model in the above embodiment is implemented.
[0080] The computer-readable storage medium may be an internal storage unit of any device with data processing capability described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be any device with data processing capability, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit of any device with data processing capability and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capability, and may also be used to temporarily store data that has been output or is to be output.
[0081] The specific embodiments described herein are merely examples of the present invention. A person skilled in the art may make various modifications or additions to the specific embodiments described or replace them in a similar manner without departing from the spirit of the present invention or exceeding the scope defined by the appended claims.
[0082] Although the terms such as hair tangent map and diffusion model are used more frequently in this article, the possibility of using other terms is not excluded. These terms are used only to more conveniently describe and explain the essence of the present invention; interpreting them as any additional limitation is contrary to the spirit of the present invention.
Claims
1. A method for generating and optimizing hair tangent maps based on a diffusion model, characterized in that: The following steps are involved: Generate hair-piece tangent map guidance data corresponding to the tangent map to be generated or to be optimized; wherein, for the tangent map to be generated, if there is only one corresponding other type of map, then the corresponding hair-piece tangent map guidance data is obtained based on the Gabor Filter; if there are multiple corresponding other types of maps, then a corresponding hair-piece tangent map is generated for each other type of map based on the GaborFilter, and then the direction with the largest value in all directions of the corresponding pixel points of each hair-piece tangent map is weighted averaged to obtain the hair-piece tangent map guidance data of the tangent map to be generated; for the tangent map to be optimized, if there is only the tangent map to be optimized, then the tangent map to be optimized is directly used as the hair-piece tangent map guidance data; if there are other corresponding types of maps, then the tangent map to be optimized is directly used as the hair-piece tangent map guidance data or each other type of map is weighted averaged based on the GaborFilter. The filter generates a corresponding hair-piece tangent map, and then performs weighted average of the direction with the largest value among all directions of the corresponding pixel points of the tangent map to be optimized and each hair-piece tangent map, to obtain the hair-piece tangent map guide data of the tangent map to be optimized; A randomly generated Gaussian noise image is input into a trained diffusion model, and reverse inference denoising is performed step by step through the diffusion model, wherein, in each reverse inference process of the diffusion model, the input image of the current step is predicted to have noise and reversely upsampled to obtain a first denoised image of the previous time step, and at the same time, an estimated value of the denoised image of time step 0 is reversely inferred based on the predicted noise, and the derivative of the estimated value and the first-order loss function of the hair tangent map guide data for the denoised image at the current time step is calculated, and the first denoised image of the previous time step obtained by sampling is updated twice in combination with the gradient descent method to obtain a second denoised image of the previous time step; the second denoised image is used as the input image of the diffusion model of the previous time step until the time step is 0, and the final tangent map is obtained.
2. The method according to claim 1, characterized in that: The diffusion model includes a noise prediction module and a sampling module, wherein the noise prediction module is used to predict the noise of the input image of the current step, and the sampling module is used to reversely upsample based on the predicted noise to obtain the first denoised image of the previous time step and reversely infer the estimated value of the denoised image to time step 0.
3. The method according to claim 2, characterized in that The first denoised image of the previous time step is obtained by reverse upsampling based on the predicted noise and the estimated value of the denoised image of the time step 0 is reversely inferred, specifically: α t =1-β t , In the formula, represents the noise obtained by the noise prediction module at the current time step, t represents the time step, β t is the diffusion coefficient at the current time step, It is calculated by the diffusion coefficient at different time steps, x t represents the input image at the current time step, is the estimated value of the denoised image at time step 0 estimated by the data at the current time step.
4. The method according to claim 2, characterized in that: The main network structure of the noise prediction module is U-net, which includes ResnetBlock and LinearAttention modules.
5. The method according to claim 1, characterized in that The diffusion model is obtained by training based on a collected training data set by minimizing a loss function, wherein each sample of the training data set includes a hair tangent map.
6. The method according to claim 1, characterized in that The loss function is expressed as follows: In the formula, ∈ represents random noise, which obeys the normal distribution of N(0,1), t represents the time step, || 2 represents the L2 paradigm; It is calculated by the diffusion coefficient at different time steps, ∈ θ represents the noise prediction module of the diffusion model, ∈ θ (*) represents the output of the noise prediction module of the diffusion model, that is, the predicted noise, and x0 is the hair tangent map.
7. The method according to claim 1, characterized in that The first denoised image obtained by sampling at the previous time step is updated twice by combining the gradient descent method to obtain the second denoised image at the previous time step, specifically: In the formula, x t ′ -1 represents the first denoised image at the previous time step, t represents the time step, x t represents the input image of the current time step, y represents the hair tangent map guide data obtained in step 3, a represents the update step size, and x t-1 represents the second denoised image at the previous time step, Represents the derivative of the first-order loss function for the image at this time step.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method for generating and optimizing a hair tangent map based on a diffusion model as described in any one of claims 1 to 7 is implemented.
9. A storage medium comprising computer executable instructions, wherein when the computer executable instructions are executed by a computer processor, the method for generating and optimizing a hair tangent map based on a diffusion model as claimed in any one of claims 1 to 7 is implemented.
10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the steps of a method for generating and optimizing a hair tangent map based on a diffusion model as described in any one of claims 1 to 7 are implemented.
Citation Information
Patent Citations
Image hair style redrawing method, device and equipment based on diffusion model
CN118196859A
Material map generation method and device
CN118247411A
Method and a system for generating a synthesized image of at least a portion of a head of hair
US20100299106A1
Cited By
Hair style replacement video generation method and device based on diffusion model
CN121032836A
A method and apparatus for generating hairstyle replacement videos based on a diffusion model
CN121032836B
