A method for generating and optimizing a film sheet tangent map based on a diffusion model

Abstract

The application discloses a kind of based on diffusion model's hair sheet tangent map generation and optimization method.This method is suitable for the common hair sheet map resource in the art resources of physical engine, can regenerate or optimize map in the case where hair sheet tangent map is missing or quality is not good.The scheme generates high-quality hair sheet map by training diffusion model, the model weight of the model reserves a large amount of prior information of hair sheet tangent map, and can be used to assist the generation and optimization of general hair sheet tangent map.In the sampling process of diffusion model, the method of adding tangent guide and mask guide is used, and the model prior information is used to regenerate high-quality tangent map according to the existing mask map and other maps, so that the hair setting can generate corresponding hair sheet model according to demand, meet the requirement of high-level feeling of hair model, and improve the quality of hair model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer technology, and in particular to a hair sheet tangent map generation and optimization method based on a diffusion model. BACKGROUND

[0002] Using a hair sheet model to cooperate with a hair sheet map to model and express hair, and applying it to a three-dimensional character model of a common game engine (Unreal Engine, Unity Engine), is one of the mainstream solutions in the industry for building a character hair model.

[0003] Because the art pipelines and purposes of different hair sheet models are not exactly the same, when resources are used, there are often cases where part of the map is not saved or missing, especially the tangent map of the hair sheet. However, in the use method of many hair sheet models, this kind of map is necessary, so a solution is needed that can generate or optimize the tangent map of the existing hair sheet.

[0004] So far, there have been some image filter-based solutions that can initially solve this problem, such as Gabor Filter. Gabor transform is essentially a special case of short-time Fourier transform, where the window function used is a Gaussian function. By applying a sliding time window to the signal at different time periods to perform Fourier transform, the local spectral characteristics of the signal are analyzed. Gabor Filter can encode features in images at different scales and directions, and has shown good performance in texture analysis, pattern recognition and computer vision tasks, and can capture and distinguish certain key visual features.

[0005] However, Gabor Filter has great limitations in the tangent identification of hair sheet maps, because each hair in the hair sheet map occupies multiple pixels, and the number of pixels occupied changes greatly with the thickness of the hair, which brings great difficulty to the parameter adjustment of Gabor Filter. Secondly, because the hair sheet map only contains the information of the hair after orthogonal projection through a single view angle, when the hair is thick, a large number of hairs completely overlap together, and the information loss in the overlapping part is serious, so the correct solution cannot be obtained using Gabor Filter, and therefore a model with data priori is needed to correct and complete the information of the Gabor Filter result.

[0006] Diffusion model is the mainstream model in the field of 2D image generation, which can learn the distribution of a class of data and generate randomly, and can be considered for this task, but as a generation model, the diffusion model must solve the problem of how to control the generation result when used. SUMMARY

[0007] The purpose of the present application is to solve the problems of the prior art, and provide a hair sheet tangent map generation and optimization method based on a diffusion model on the basis of the traditional Gabor Filter technology. The method trains the diffusion model using a customized data set, so that the model learns the relatively correct hair sheet tangent map data distribution, and uses the results of the Gabor Filter to guide the generation process of the diffusion model, and finally generates high-quality hair sheet tangent maps to meet the data quality requirements of the subsequent art pipeline.

[0008] The purpose of the present application is to solve the problems of the prior art, and provide a hair sheet tangent map generation and optimization method based on a diffusion model on the basis of the traditional Gabor Filter technology. The method trains the diffusion model using a customized data set, so that the model learns the relatively correct hair sheet tangent map data distribution, and uses the results of the Gabor Filter to guide the generation process of the diffusion model, and finally generates high-quality hair sheet tangent maps to meet the data quality requirements of the subsequent art pipeline.

[0009] A hair sheet tangent map generation and optimization method based on a diffusion model comprises the following steps:

[0010] Generate the hair sheet tangent map guide data corresponding to the tangent map to be generated or optimized. For the tangent map to be generated, if there is only one corresponding other type of map, generate the corresponding hair sheet tangent map guide data based on Gabor Filter; if there are multiple corresponding other type of maps, generate one corresponding hair sheet tangent map for each other type of map based on Gabor Filter, and then perform weighted averaging on the direction with the largest value among all directions of the pixel points of each hair sheet tangent map to obtain the hair sheet tangent map guide data of the tangent map to be generated. For the tangent map to be optimized, if there is only the tangent map to be optimized, directly use the tangent map to be optimized as the hair sheet tangent map guide data; if there is also a corresponding other type of map, directly use the tangent map to be optimized as the hair sheet tangent map guide data or generate one corresponding hair sheet tangent map for each other type of map based on Gabor Filter, and then perform weighted averaging on the direction with the largest value among all directions of the pixel points of the tangent map to be optimized and each hair sheet tangent map to obtain the hair sheet tangent map guide data of the tangent map to be optimized.

[0011] Input the randomly generated Gaussian noise image into the trained diffusion model, and gradually perform reverse inference denoising through the diffusion model. In each reverse inference process of the diffusion model, predict the noise of the input image at the current step and inversely upsample to obtain a first denoised image at the previous time step, and simultaneously inversely infer the predicted value of the denoised image at time step 0 based on the predicted noise, calculate the derivative of the first-order loss function of the predicted value and the hair sheet tangent map guide data with respect to the denoised image at the current time step, and combine the gradient descent method to update the first denoised image at the previous time step to obtain a second denoised image at the previous time step. Use the second denoised image as the input image of the diffusion model at the previous time step until the time step is 0, and obtain the final tangent map.

[0012] The application adopts a method of adding tangential guidance and mask guidance in the diffusion model sampling process, and can regenerate high-quality tangential maps according to existing mask maps and other maps by using model prior.

[0013] Further, the diffusion model comprises a noise prediction module and a sampling module, wherein the noise prediction module is used for predicting noise of an input image of a current step, and the sampling module is used for inversely upsampling the predicted noise to obtain a first denoised image of a previous time step and an estimated value of a denoised image of time step 0 inversely inferred.

[0014] Further, the inversely upsampling the predicted noise to obtain the first denoised image of the previous time step and the estimated value of the denoised image of the time step 0 inversely inferred is specifically:

[0015]

[0016] In the formula, represents noise obtained by the noise prediction module at the current time step, t represents a time step, and β T is a diffusion coefficient at the current time step, is calculated via different time step diffusion coefficients, x t represents an input image of the current time step, is an estimated value of the denoised image of the time step 0 estimated by data of the current time step.

[0017] Further, the network main structure of the noise prediction module is a U-net, and the structure comprises a ResnetBlock and a LinearAttention module.

[0018] Further, the diffusion model is obtained by training based on a collected training data set to minimize a loss function, and each sample of the training data set comprises a film tangential map.

[0019] Further, the loss function is represented as follows:

[0020]

[0021] In the formula, ∈ represents random noise, and is subject to a normal distribution N(0, 1); t represents a time step, and 2 represents an L2 norm; is calculated via different time step diffusion coefficients, ∈ θ represents a noise prediction module of the diffusion model, ∈ θ (*) represents an output of the noise prediction module of the diffusion model, i.e., predicted noise, and x0 is a film tangential map.

[0022] Further, the combined gradient descent method is used to update the first denoised image of the previous time step to obtain the second denoised image of the previous time step, specifically:

[0023]

[0024] In the formula, x t ′ -1 denotes the first denoised image of the previous time step, t denotes the time step, x t denotes the input image of the current time step, y denotes the film sheet tangential map guide data obtained by step three, a denotes the update step length, x t-1 denotes the second denoised image of the previous time step, denotes the derivative of the first-order loss function with respect to the image at the time step.

[0025] An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the computer program to implement the film sheet tangential map generation and optimization method based on the diffusion model.

[0026] A storage medium comprising computer executable instructions, which, when executed by a computer processor, implement the film sheet tangential map generation and optimization method based on the diffusion model.

[0027] A computer program product comprising computer programs / instructions, which, when executed by a processor, implement the steps of the film sheet tangential map generation and optimization method based on the diffusion model.

[0028] The benefit and effect of the present application is that the diffusion model is trained using a custom dataset, and the film sheet tangential map is generated and optimized by guidance, which realizes the completion of the missing art resources in the film model, improves the utilization efficiency of the existing art resources, and has a simple implementation method, flexible means, and strong universality, which can guarantee that the hair setting generates the corresponding film model according to the demand, meets the requirement of high-level feeling of the hair model, and further improves the quality of the hair model. And for different types of film sheet map, a corresponding dataset can be constructed to enhance the generation effect. BRIEF DESCRIPTION OF DRAWINGS

[0029] Figure 1 is the flowchart of the film sheet tangential map generation and optimization method based on the diffusion model of the present application;

[0030] Figure 2 is the framework diagram of the film sheet tangential map generation and optimization method based on the diffusion model of an embodiment of the present application;

[0031] Figure 3The figure on the right is the guide data of the model, obtained by using Gabor Filter; the figure in the middle is the real hair sheet tangent map in the self-defined test set, used for verifying the effect of the model; and the figure on the left is the output of the guided diffusion model, i.e. the optimized hair sheet tangent map. DETAILED DESCRIPTION

[0032] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should fall within the scope of protection of the present application.

[0033] It should be noted that the terms "first", "second", and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example: a process, method, system, product or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0034] As shown in Figure 1 A hair sheet tangent map generation and optimization method based on a diffusion model according to the present application includes the following steps:

[0035] Step 1: generate a sheet tangent map guide data corresponding to a to-be-generated or to-be-optimized tangent map; wherein, for the to-be-generated tangent map, if there is only one corresponding other type of map, the corresponding sheet tangent map guide data is obtained based on Gabor Filter; if there are multiple corresponding other type of maps, a corresponding sheet tangent map is generated for each other type of map based on Gabor Filter, and then the direction with the maximum value among all directions of the corresponding pixel points of each sheet tangent map is weighted and averaged to obtain the sheet tangent map guide data of the to-be-generated tangent map; for the to-be-optimized tangent map, if there is only the to-be-optimized tangent map, the to-be-optimized tangent map is directly taken as the sheet tangent map guide data; if there is also a corresponding other type of map, the to-be-optimized tangent map is directly taken as the sheet tangent map guide data or a corresponding sheet tangent map is generated for each other type of map based on Gabor Filter, and then the direction with the maximum value among all directions of the corresponding pixel points of the to-be-optimized tangent map and each sheet tangent map is weighted and averaged to obtain the sheet tangent map guide data of the to-be-optimized tangent map.

[0036] Step 2: input the randomly generated Gaussian noise image into the trained diffusion model, and gradually perform reverse inference denoising through the diffusion model, wherein, in the reverse inference process of each step of the diffusion model, the input image of the current step is predicted noise and is inversely up-sampled to obtain a first denoised image of the previous time step, and at the same time, based on the predicted noise, an estimated value of the denoised image at time step 0 is inversely inferred, the derivative of the first-order loss function of the estimated value and the sheet tangent map guide data with respect to the denoised image at the current time step is calculated, and the first denoised image obtained by sampling is updated twice to obtain a second denoised image at the previous time step by combining the gradient descent method; the second denoised image is taken as the input image of the diffusion model at the previous time step, until the time step is 0, and the final tangent map is obtained.

[0037] The present application guides the generation process of the diffusion model by processing the existing original art resources as the corresponding sheet tangent map guide data, and finally generates a relatively high-quality sheet tangent map, which meets the data quality requirements of the subsequent art pipeline. The effects of the present application will be further described below in conjunction with a specific embodiment.

[0038] As shown in Figure 2 A sheet tangent map generation and optimization method based on a diffusion model, mainly includes two parts, one is to obtain a trained diffusion model; the second is to realize sheet tangent map generation and optimization based on the trained diffusion model; the following takes the process of training the diffusion model and realizing sheet tangent map generation and optimization based on the trained diffusion model for the first time as an example for description, specifically:

[0039] Step one: Constructing training dataset

[0040] This step is the data foundation of the present application, and the data set is autonomously constructed to enable the diffusion model to learn the data distribution of similar hair slice tangent maps. This step can be obtained by collection or by modeling software; in a specific embodiment, the method of constructing by modeling software is as follows:

[0041] Generate three-dimensional hair strands in bulk that conform to common hair slice map morphologies through modeling software; convert the three-dimensional hair strands into strips composed of triangular facets in three-dimensional space; adjust the rendering parameters and the coloring method of the triangular facets, and perform rendering; organize and store; specifically including the following sub-steps:

[0042] (1.1) Set the length, number, curvature, and diversity degree parameters of the hair strands, generate three-dimensional hair strands in bulk that conform to common hair slice map morphologies, and export them from ZBrush, and convert them to obj format through blender for saving;

[0043] (1.2) Read the three-dimensional hair strands in obj format obtained in step (1.1), convert them into three-dimensional hair strand curves represented by a list of consecutive three-dimensional points; displace each hair strand curve S0 in the same direction to obtain a translated hair strand curve S1, and stagger the connection of the points on the pre-translation hair strand curve S0 and the post-translation hair strand curve S1, specifically: connect the first point of the pre-translation hair strand curve S0 to a line segment to the first point of the post-translation hair strand curve S1, connect the first point of the post-translation hair strand curve S1 to a line segment to the second point of the pre-translation hair strand curve S0, and repeat this process until there are no points that can be connected, and convert the hair strand curve into a strip composed of triangular facets;

[0044] (1.3) For hair slice tangent maps, assign the color of the triangular facet vertex to the tangent unit vector of the three-dimensional hair strand curve corresponding to the point, with the RGB channels corresponding to the vectors in the three directions, and perform rendering to obtain the tangent map;

[0045] Further, the various maps in the dataset obtained in step (1.3) can be subjected to data augmentation, and the augmentation schemes include random cropping of a square block, rotation, translation, and mirroring; the augmented data is numbered and stored to obtain the training dataset.

[0046] Step two: Constructing and training the diffusion model

[0047] Use the forward diffusion process and the reverse sampling process of the traditional diffusion model as the overall structure of network training, and use the dataset constructed in step one to train the model; specifically, including the following sub-steps:

[0048] (2.1) build a diffusion model, the diffusion model of the application includes a noise prediction module and a sampling module, wherein the noise prediction module is used to predict the noise of the input image of the current step, and the sampling module is used to obtain the first denoised image of the previous time step and the estimated value of the denoised image of time step 0 by reverse inference based on the predicted noise. The network main structure of the noise prediction module is U-net, which contains ResnetBlock, LinearAttention module, and uses U-Net to predict noise; further, the diffusion model of the application can also include a module for adding Gaussian noise, thereby realizing the required forward diffusion process and reverse inference process, the forward inference process is realized by the module for adding Gaussian noise, and the reverse inference process is mainly realized by the noise prediction module and the sampling module.

[0049] (2.2) randomly sample a batch of training data, and each sample of the data is a film cut tangent map in the data set. For each sample in the batch, a time step and a Gaussian noise are randomly sampled, and the result obtained by the sample, the time step and the Gaussian noise through the forward diffusion process at the time step, that is, the noise image, is calculated;

[0050] (2.3) input the result of the forward diffusion into the noise prediction module to obtain the predicted noise, and derive the second order loss of the predicted noise and the random noise actually sampled, and use gradient descent method to update the parameters of the neural network, wherein the loss function is as follows:

[0051]

[0052] In the formula, ∈ represents random noise, which is normally distributed as N(0, 1), t represents time step, || represents L2 norm; β 2 is the diffusion coefficient at the current time step, which is set according to actual requirements, and is set to β t in the embodiment. t ∈{β1,β2,…,…β T} is set in the interval [0.0001, 0.02] β t and the diffusion step number t are in a proportional relationship. is calculated by the diffusion coefficient at different time steps, ∈ θ represents the noise prediction module, ∈ θ (*) represents the output of the noise prediction module of the diffusion model, that is, the predicted noise, and x0 is the film cut tangent map in the training data.

[0053] (2.4) a batch of samples is regarded as a round of training, and a plurality of rounds of training are performed until the loss function converges or the set training rounds, for example, 200000, are reached;

[0054] After obtaining the trained diffusion model, the generation and optimization of the hair sheet tangent map can be realized.

[0055] Step three: generate the hair sheet tangent map guide data corresponding to the tangent map to be generated or optimized; the present application generates hair sheet tangent map guide data in different schemes for three different situations: only one type of map exists, multiple types of maps exist, and low-quality tangent maps exist, i.e., tangent maps to be optimized;

[0056] Step three is one of the core parts of the present application, and different hair sheet tangent map guide data is constructed according to the existing data of the art resources that need to be supplemented, and the quality and construction method of the guide data will have a significant impact on the subsequent generation effect.

[0057] The step three includes the following sub-steps:

[0058] (3.1) generating hair sheet tangent map guide data corresponding to the tangent map to be generated or optimized, wherein for the tangent map to be generated, for the hair sheet art resource with only one type of map, the tangent thereof is identified by a conventional Gabor Filter-based scheme to obtain the corresponding hair sheet tangent map guide data; specifically, a conventional Gabor Filter convolution kernel is used to convolve the image, the numerical value of each pixel in each direction is calculated, and the direction with the maximum numerical value in all directions of each pixel point is selected as the possible tangent of the hair of the pixel, and after merging, the hair sheet tangent map guide data is obtained; in this step, if the resolution or the image length and width of the art resource for which the hair sheet tangent map needs to be regenerated or optimized are inconsistent with the training data, the image needs to be cut, and the cut image is used for operation; if there are multiple types of maps, all the maps are processed according to the scheme in step (3.1), and the direction with the maximum numerical value in all directions of each map corresponding to the pixel point is weighted and averaged to obtain the hair sheet tangent map guide data;

[0059] (3.2) for the tangent map to be optimized, i.e., the art resource with low-quality tangent map, the tangent map is directly used as the hair sheet tangent map guide data according to the specific situation of the tangent map, or step (3.2) is repeated: a hair sheet tangent map corresponding to each type of map is generated based on Gabor Filter, and the direction with the maximum numerical value in all directions of each hair sheet tangent map corresponding to the pixel point is weighted and averaged to obtain the hair sheet tangent map guide data of the tangent map to be optimized.

[0060] (3.3) standardizing the hair sheet tangent map guide data obtained in steps (3.1) and (3.2) to make it consistent with the input data format of the diffusion model;

[0061] Step four: guide the reverse inference process of the diffusion model by the hair sheet tangent map guide data obtained in step three, so that the hair sheet tangent map generated by the model is highly approximate to the required optimized or generated hair sheet tangent map; the reverse inference ends, and a hair sheet tangent map with quality meeting the conditions is obtained;

[0062] Step four is the second part of the core invention, which focuses on data guidance during the sampling process of the diffusion model. This operation changes the uncontrollable generation task into a controllable image generation and image optimization task.

[0063] The step four includes the following sub-steps:

[0064] (4.1) Randomly generate a Gaussian noise image with a resolution meeting the training data, input it into the trained diffusion model, and gradually perform reverse inference denoising through the diffusion model. In this embodiment, 1000 reverse inferences from 1000 to 0 time steps are performed.

[0065] (4.2) For each reverse inference, the input image at the current step is used to obtain the predicted noise required for the reverse sampling process at this time step through the inference scheme consistent with the traditional diffusion model, and the first denoised image at the previous time step is obtained by reverse sampling.

[0066] (4.3) In step (4.2), the predicted noise can be used to calculate the estimated value of the denoised image at time step 0 of the reverse inference, and a first-order loss function of the estimated value and the hair sheet tangent map guide data obtained in step (3.4) is calculated, as follows:

[0067]

[0068] In the formula, x represents the noise obtained by the noise prediction module at the current time step, and t represents the time step, is calculated through the diffusion coefficient at different time steps, x t represents the input image at the current time step, is the estimated value of the denoised image at time step 0 estimated by the predicted noise data at the current time step.

[0069] (4.4) Calculate the derivative of the first-order loss function in step (4.3) with respect to the image at this time step, use an update step, and use the gradient descent method to update the result obtained by reverse sampling twice, as follows:

[0070]

[0071] In the formula, x t ′ -1denotes the first denoised image of the previous time step obtained by the traditional sampling process, t denotes the time step, x denotes the input image of the current time step t denotes the input image of the current time step, y denotes the film tangential map guiding data obtained by step three, a denotes the update step, 0.7 is used in the embodiment, x t-1 denotes the second denoised image of the previous time step, denotes the derivative of the first loss function with respect to the image of the time step.

[0072] (4.5) The looped reverse inference process is used to infer from time step 1000 to time step 0 to obtain the final film tangential map.

[0073] Figure 3 is the result image of the model output, the true value and the guiding data of a specific case of the application, the right one in the image is the guiding data of the model, which is obtained using Gabor Filter; the second from the right is the real film tangential map in the test set, which is used to verify the effect of the model; the third from the right is the output of the guided diffusion model, that is, the optimized film tangential map, it can be seen that the accuracy of the hair growth tangential direction has been greatly improved compared with the film tangential map before optimization.

[0074] Corresponding to the foregoing embodiment of the film tangential map generation and optimization method based on the diffusion model, the application also provides an electronic device comprising one or more processors for implementing the film tangential map generation and optimization method based on the diffusion model in the foregoing embodiment.

[0075] The electronic device of the application can be any device with data processing capability, and the device with data processing capability can be a device or apparatus such as a computer.

[0076] The device embodiment can be realized by software, hardware or a combination of software and hardware. For example, as a logical device, the corresponding computer program instructions in the non-volatile memory are read into the memory and executed by the processor of the device to form a hardware layer. In addition to the processor, memory, network interface and non-volatile memory, the device in the embodiment can also include other hardware according to the actual function of the device with data processing capability, which will not be described here.

[0077] The implementation process of the functions and roles of each unit in the above device is specifically described in the implementation process of the corresponding steps in the above method, which will not be described here.

[0078] For the apparatus embodiment, since it basically corresponds to the method embodiment, the relevant part can be seen from the part of the method embodiment. The apparatus embodiment described above is only illustrative, wherein the units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e., can be located in one place or distributed on multiple network units. Some or all of the modules can be selected to achieve the purpose of the present application according to actual needs. Those skilled in the art can understand and implement it without creative labor.

[0079] The embodiment of the present application also provides a computer readable storage medium, which stores a program, and the program is executed by a processor to realize the method for generating and optimizing a film sheet tangential map based on a diffusion model in the above embodiment.

[0080] The computer readable storage medium can be an internal storage unit of any data processing device, such as a hard disk or a memory. The computer readable storage medium can also be any data processing device, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. Further, the computer readable storage medium can include both an internal storage unit of any data processing device and an external storage device. The computer readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0081] The specific embodiments described herein are merely illustrative of the present application. Those skilled in the art can make various modifications or supplements to the described specific embodiments or use similar ways to replace them without departing from the spirit of the present application or exceeding the scope defined by the appended claims.

[0082] Although the terms such as film sheet tangential map and diffusion model are used more frequently herein, the possibility of using other terms is not excluded. The use of these terms is only for the convenience of describing and explaining the essence of the present application; any interpretation of them as any kind of additional limitation is contrary to the spirit of the present application.

Claims

1. A method for generating and optimizing hair tangent maps based on a diffusion model, characterized in that: The following steps are involved: Generate hair piece tangent map guidance data corresponding to the tangent map to be generated or to be optimized; wherein, for the tangent map to be generated, if there is only one corresponding other type of map, then generate the corresponding hair piece tangent map guidance data based on Gabor Filter; if there are multiple corresponding other type of maps, then generate a corresponding hair piece tangent map for each other type of map based on GaborFilter, and then perform weighted averaging on the direction with the largest value among all directions of the corresponding pixel points of each hair piece tangent map to obtain the hair piece tangent map guidance data of the tangent map to be generated; for the tangent map to be optimized, if there is only the tangent map to be optimized, then directly use the tangent map to be optimized as the hair piece tangent map guidance data; if there are corresponding other type of maps, then directly use the tangent map to be optimized as the hair piece tangent map guidance data or perform weighted averaging on each other type of map based on GaborFilter. The filter generates a corresponding hair tangent map, and then performs a weighted average of the direction with the largest value among all directions of the corresponding pixel points of the tangent map to be optimized and each hair tangent map, to obtain the hair tangent map guidance data of the tangent map to be optimized; A randomly generated Gaussian noise image is input into a trained diffusion model, and reverse inference denoising is performed step by step through the diffusion model. In each reverse inference process of the diffusion model, noise is predicted for the input image of the current time step and reversely upsampled to obtain a first denoised image of the previous time step. Simultaneously, an estimated value of the denoised image at time step 0 is reversely inferred based on the predicted noise. The derivative of the estimated value and the first-order loss function of the tangential map guide data with respect to the denoised image at the current time step is calculated. The first denoised image of the previous time step obtained by sampling is updated twice using the gradient descent method to obtain a second denoised image of the previous time step. The second denoised image is used as the input image of the diffusion model of the previous time step until time step 0 is reached, thereby obtaining a final tangential map.

2. The method according to claim 1, characterized in that The diffusion model includes a noise prediction module and a sampling module, wherein the noise prediction module is used to predict the noise of the input image of the current step, and the sampling module is used to reversely upsample based on the predicted noise to obtain the first denoised image of the previous time step and reversely infer the estimated value of the denoised image to time step 0.

3. The method according to claim 2, characterized in that The first denoised image of the previous time step is obtained by reverse upsampling based on the predicted noise, and the estimated value of the denoised image at time step 0 is reversely inferred, specifically: α t ＝1-β t ， Where, represents the noise obtained by the noise prediction module at the current time step, t represents the time step, β t is the diffusion coefficient at the current time step, It is calculated by the diffusion coefficient at different time steps, x t represents the input image at the current time step, is the estimated value of the denoised image at time step 0 estimated by the data at the current time step.

4. The method according to claim 2, characterized in that The main network structure of the noise prediction module is U-net, which includes ResnetBlock and LinearAttention modules.

5. The method according to claim 1, wherein The diffusion model is obtained by training based on a collected training data set by minimizing a loss function, wherein each sample of the training data set includes a hair tangent map.

6. The method according to claim 1, characterized in that The loss function is expressed as follows: Where, ∈ represents random noise, which obeys the normal distribution of N(0,1), t represents the time step, || 2 represents the L2 paradigm; It is calculated by the diffusion coefficient at different time steps, ∈ θ represents the noise prediction module of the diffusion model, ∈ θ (*) represents the output of the noise prediction module of the diffusion model, that is, the predicted noise, and x0 is the hair tangent map.

7. The method according to claim 1, characterized in that The gradient descent method is combined to perform a secondary update on the first denoised image obtained by sampling at the previous time step to obtain the second denoised image at the previous time step, specifically: Where x t ′ -1 represents the first denoised image at the previous time step, t represents the time step, x t represents the input image of the current time step, y represents the hair tangent map guide data obtained in step 3, a represents the update step size, x t-1 represents the second denoised image at the previous time step, Represents the derivative of the first-order loss function with respect to the image at that time step.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method for generating and optimizing a hair tangent map based on a diffusion model according to any one of claims 1 to 7 is implemented.

9. A storage medium comprising computer executable instructions, wherein when executed by a computer processor, the computer executable instructions implement the method for generating and optimizing hair tangent maps based on a diffusion model according to any one of claims 1 to 7.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the steps of the method for generating and optimizing a hair tangent map based on a diffusion model as claimed in any one of claims 1 to 7 are implemented.

Images