A human posture scene restoration method based on adaptive network enhanced diffusion model

Abstract

The present invention discloses a human posture scene restoration method based on an adaptive network enhanced diffusion model, which belongs to the field of image generation in computer vision. In this method, each set of training data is represented as a five-tuple (A (N) ,B (N) ,y,Z A ,Z B ), and use the dot matrix A (N) ,B (N) Grayscale images A0 and B0 are made respectively. Secondly, the difference between the primary and secondary adapters in the diffusion model is obtained by the diffusion model, and the loss function is calculated according to the difference, and the residual loss is calculated. Finally, the global optimization function is obtained according to the loss function and the residual loss function, and the output character posture graph is optimized. The present invention eliminates the computational overhead required for training to find the weights between loss functions through a trial method, realizes the training method of the pre-trained model function subdivision, and makes the output character posture graph more accurate and stable.

Description

Technical Field

[0001] The present invention belongs to the field of image generation of computer vision, and in particular relates to a method for restoring a human body posture scene based on an adaptation network enhanced diffusion model. Background Art

[0002] The image generation task in the field of computer vision was first completed by the Generative Adversarial Network (GAN), but in recent years it has gradually been replaced by the Stable Diffusion model, which can generate corresponding pictures based on the input text information. However, the models under the above two frameworks are still unable to generate specific scenes, specific objects, and people or objects in specific outlines based on given inputs. Most of the models that directly generate portraits or object images in traditional modes focus on how to accurately understand text prompts and thus realize image operations under more precise text prompts (such as generating images for input text and modifying images according to text requirements).

[0003] The key to stable image generation is to pass the required information to the machine in a reasonable form and encode it to generate output. Fundamentally, this is because human language descriptions have very large uncertainties from the perspective of the machine, and the actual fuzzy semantics are far beyond imagination. The ControlNet architecture and the T2I-Adapter architecture provide a diffusion model with image constraints using small models such as semantic segmentation and key gesture recognition, and provide an adapter model that injects additional constraints into the diffusion model; however, such models will be confused with text guidance when injected. Specifically, when the same small model is used to give different image constraints when the same text guidance is used, the scene and even the style mode of the final generated content will drift greatly, showing low robustness. Summary of the invention

[0004] In view of the above problems, the present invention proposes a method for restoring human posture scenes based on an adaptive network enhanced diffusion model. The method performs a joint training method of the primary and secondary adapters (Primary-Adapter and Secondary-Adapter) based on the architecture of T2I-Adapter, and at the same time derives and simplifies the optimization objectives of the training, constructs a model based on a fine-tuning adapter, and realizes embedding a small model such as key posture recognition and image semantic segmentation through the adapter. According to the information obtained from the analysis of the small model, the image is distinguished according to the content of the text and the background scene with unchanged detection information of the small model is restored (for example, for the small model of key posture recognition, the background scene guided by the text is restored for different key postures, while the objects in the key posture features are kept unchanged). The invention aims to manipulate the locked large weights through the fine-tuning model, sample high-dimensional pixel distribution with a more reliable and controllable small model, and drive higher-demand image generation tasks.

[0005] The present invention mainly introduces a secondary adapter (S-ad) and fine-tunes the secondary adapter under the conditions of fixed stable-diffusion weights and pre-trained primary adapter (P-ad) weights, so as to achieve the effect of the secondary adapter outputting stable data for the primary adapter. At the same time, the fine-tuned secondary adapter has stability for text; in conjunction with the feature extraction model of Google-Dreambooth in terms of generation, it can easily achieve: image generation of any specified scene (text description), any specified person (image feature input), and any specified posture (key posture).

[0006] In the present invention, two local optimization objectives are first constructed, and then the two objectives are mapped to the same scale through transformation for joint training; at the same time, this method is promoted in the main and secondary adapter models of Stable-Diffusion, so that it can adapt to the function of driving Stable-Diffusion using different small models.

[0007] The present invention provides the following technical solutions:

[0008] Step 1. Each set of training data is represented as a five-tuple (A (N) , B (N) , y, Z A , Z B ), where y is the text description label, A (N) , B (N) The number of elements in both The dot matrix, is the image data, where the number 3 means that the image is stored in RGB format, w and h represent the width and height of the image respectively; δ (·) represents the model of the coordinate point set of the character joint base points in the input image, with δ as the weight.

[0009] As a supplementary note, the text description annotation y∈{y A ,y B}, here y A ,y B Represents image Z A , Z B The English text description corresponding to the image content (so two quintuples are actually generated here).

[0010] Through the model δ (·) The process of calculating the joint base point set in the input image Y is described as:

[0011]

[0012] Afterwards Indicates the coordinate position of the i-th base point in the w×h coordinate system, and uses y A,i Indicates the ordinate position of the i-th base point in image A, y B，i Similarly, similar to the definition of x A,i , x B,i Through this model, we get a The points in the lattice are normalized (i.e. compressed) using the normalization function defined below. Specifically, for a point The normalized representation of is:

[0013]

[0014] Here w =w,b h =h, is an adjustable bias (one of the hyperparameters).

[0015] For the entire lattice, the normalization (compression) process is expressed as:

[0016]

[0017] Combining equations (1) and (3), it can be expressed in the form of a composite function:

[0018]

[0019] According to the above definition, the number of elements in the data quintuple is The dot matrix A (N) , B(N) It is calculated as follows:

[0020]

[0021] in

[0022] According to the image Z A , Z B The points obtained In subsequent steps, they are rendered into grayscale images.

[0023] Step 2. Put the two dot matrices obtained in step 1 into Create grayscale images A0 and B0 respectively: expand the compressed dot matrix geometrically to [0, u]×[0, u], where u=min{w, h}; in addition, set the grayscale values ​​of the coordinate positions corresponding to the points that have appeared in the dot matrix to 0, and set the remaining positions to 255, and obtain two grayscale images A0 and B0 respectively.

[0024] The original image Z A , Z B The text conversion model is passed in (converting the text content into vector encoding), and the vector encoding of the obtained text is τ(y). The five-tuple of each data becomes:

[0025] (A0,B0,τ(y),Z A , Z B ) (7)

[0026] Also set adjustable parameters represents the total diffusion step length of the diffusion model; here we take y = y A Or y = y B , two pieces of data can be obtained.

[0027] In fact, in this process, the grayscale images A0 and B0 produced from the original image are the image data required for the final training. The data set used in the actual training is recorded as TRAIN, and for all grayscale image combinations A0 and B0:

[0028]

[0029] Step 3. Describe the denoising process (sampling process, or inference process) of the diffusion model as:

[0030]

[0031] In the above formula And α t ∈[0,1] is an adjustable parameter in the diffusion model.

[0032] In the entire diffusion model, t traverses T, T-1, …, 2, 1 (the adjustable parameter in the diffusion model is α t ,σ t ∈[0,1]), there are T equations in total, which are called “the tth equation” according to the value of t; where ∈ θ It is the pre-trained U-Net model in the diffusion model, with θ as the weight. It indicates that the denoising process samples the generated image in the latent space at the t step, and takes

[0033] Further, here It means that the mean of each dimension is 0, and the covariance matrix is ​​the unit matrix I, that is, a high-dimensional normal distribution; in the present invention, the dimensions of all I are the same.

[0034] To decode the latent space image representation into a real picture, we need to use the decoder of the variational autoencoder (VAE). This model is denoted as De(·). The real image Z at time t t It is expressed as:

[0035]

[0036] In the denoising process of the diffusion model using the adapter, the image in the latent space is represented as:

[0037]

[0038] Where X0 represents the image condition input, y represents the text input, and F Φ represents the adapter model with Φ as weight.

[0039] In the above diffusion model, the grayscale image A0 obtained in step 2 is passed to the primary adapter. Get vector Pass the grayscale image B0 to the secondary adapter Get vector They are injected into the diffusion model according to the method in the T2I-Adapter model, where Φ1 and Φ2 represent the weights used by the adapter model.

[0040] In addition, the diffusion model used in the present invention uses U-Net as a denoising network, and uses CLIP, a contrastive language image pre-training, as an encoder to convert text content into a vector. Classifier-Free Guidance is used to continuously inject these two vectors into the Denoising U-Net in each time step of the diffusion model, fix the weight θ of Stable-Diffusion and the weight Φ1 of the primary adapter, and set the weight Φ2 of the secondary adapter as a learnable parameter. The two diffusion generation paths calculate the prediction loss of the noise. The derivation process is as follows:

[0041] For convenience, the generation path where the main adapter is located is recorded as line 1, and the generation path where the secondary adapter is located is recorded as line 2. The loss used is:

[0042]

[0043] here:

[0044]

[0045] Use with represents the mathematical expectation, Indicates that the time step t follows a uniform distribution on [1,T], ∈~(0,I) indicates that the random noise ∈ follows a random normal distribution, ∈ θ Represents the noise prediction value calculated by the U-Net network with θ as the weight in the diffusion model.

[0046] For route 1, the image sampled by the diffusion model at time step t is recorded as Similar definition Z B,t .

[0047] In order to achieve the goal of controlling the portrait posture through the joint base points, it is necessary to ensure that the images obtained by sampling the predicted noise in routes 1 and 2 have the same pixel distribution as much as possible, except for the pixels with different distributions caused by different joint base points. Since the diffusion model uses gradient quantization to represent the score of the diffusion direction, the difference in the diffusion noise in route 1 and route 2 at a certain time step can be expressed as:

[0048]

[0049] Here p(·) represents the probability distribution. Indicates the gradient, combined with the Tweedie formula, we can know:

[0050]

[0051] as well as:

[0052]

[0053] This quantitatively represents the diffusion direction within the model, and the difference in diffusion direction at time step t for the diffusion model after the two adapter injection vectors are respectively passed can be expressed as:

[0054]

[0055] Therefore, according to equations 15 and 16, the optimization of the diffusion direction of line 1 and line 2 is designed, and the loss function is set:

[0056]

[0057]

[0058] On the other hand, considering that the difference in diffusion results after injection of the main and secondary adapters in line 1 and line 2 should actually be limited to the posture marked by the joint base point, it is also necessary to limit the posture from the loss function to ensure that:

[0059]

[0060] The right side of the above equation is actually a constant. To satisfy the constraints of equation (19), we can choose to optimize:

[0061]

[0062] The constant term can be discarded to obtain the residual loss function:

[0063]

[0064] So far, two loss functions for local optimization are obtained through equations (18) and (21).

[0065] Step 4. In this step, the global optimization function is determined. For equations (18) and (21), the two local loss functions cannot be added directly because of their different scales. To address this problem, most traditional methods add a coefficient to one of them (the coefficient is tested for a suitable value during the training process) and then add two or more functions for local optimization. In this step, a new method is used to directly convert the two loss functions for local optimization to the same scale to optimize the output character posture graph.

[0066] Since the model obtained by optimizing the main adapter allows the Stable-Diffusion model to stably distribute pixels to the joint base point positions (calibrating the posture of the portrait) when sampling in fixed weights, it is equivalent to completing an optimization problem with constraints when training the adapter.

[0067] Set the global optimization function to:

[0068]

[0069] As a test of the correctness of the derived model, we can check whether the global loss after a sufficiently large number of iterative training has converged to the theoretical value:

[0070]

[0071] The present invention has the following advantages: the present invention transforms two local optimization functions to the same scale, thereby eliminating the computational overhead required for finding weights between loss functions through trial and error training; the direct optimization of the loss function is transformed into optimizing the supremum (minimum upper bound), and a training method for realizing functional subdivision of the pre-trained model by comparing the generation effects of each step in the diffusion model is proposed, so that the output character posture graph is more accurate and stable. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 is a training flow chart of the present invention;

[0073] Figure 2 Vector injection mode for a single adapter;

[0074] Figure 3 It is a schematic diagram of the results of the present invention. DETAILED DESCRIPTION

[0075] The present invention is explained in detail below with reference to the accompanying drawings.

[0076] A human posture scene restoration method based on adaptive network enhanced diffusion model, such as Figure 1 As shown, the specific steps include:

[0077] Step 1. Each set of training data is represented as a five-tuple (A (N) , B (N) , y, Z A , Z B ), where y is the text description label, A (N) , B (N) The number of elements in both The dot matrix, is the image data, where the number 3 means that the image is stored in RGB format, w and h represent the width and height of the image respectively; δ (·) represents the model of the coordinate point set of the character joint base points in the input image, with δ as the weight.

[0078] As a supplementary note, the text description annotation y∈{y A ,y B}, here y A ,y B Represents image Z A , Z B The English text description corresponding to the image content (so two quintuples are actually generated here).

[0079] Through the model δ (·) The process of calculating the joint base point set in the input image Y is described as:

[0080]

[0081] Afterwards Indicates the coordinate position of the i-th base point in the w×h coordinate system, and uses y A，i Indicates the ordinate position of the i-th base point in image A, y B，i Similarly, similar to the definition of x A,i , x B,i Through this model, we get a The points in the lattice are normalized (i.e. compressed) using the normalization function defined below. Specifically, for a point The normalized representation of is:

[0082]

[0083] Here w =w,b h =h, is an adjustable bias (one of the hyperparameters).

[0084] For the entire lattice, the normalization (compression) process is expressed as:

[0085]

[0086] Combining equations (1) and (3), it can be expressed in the form of a composite function:

[0087]

[0088] According to the above definition, the number of elements in the data quintuple is The dot matrix A (N) , B(N) It is calculated as follows:

[0089]

[0090] in

[0091] According to the image Z A , Z B The points obtained In subsequent steps, they are rendered into grayscale images.

[0092] Step 2. Put the two dot matrices obtained in step 1 into Create grayscale images A0 and B0 respectively: expand the compressed dot matrix geometrically to [0, u]×[0, u], where u=min{w, h}; in addition, set the grayscale values ​​of the coordinate positions corresponding to the points that have appeared in the dot matrix to 0, and set the remaining positions to 255, and obtain two grayscale images A0 and B0 respectively.

[0093] The original image Z A , Z B The text conversion model is passed in (converting the text content into vector encoding), and the vector encoding of the obtained text is τ(y). The five-tuple of each data becomes:

[0094] (A0,B0,T(y),Z A , Z B ) (7)

[0095] Also set adjustable parameters represents the total diffusion step length of the diffusion model; here we take y = y A Or y = y B , two pieces of data can be obtained.

[0096] In fact, in this process, the grayscale images A0 and B0 produced from the original image are the image data required for the final training. The data set used in the actual training is recorded as TRAIN, and for all grayscale image combinations A0 and B0:

[0097]

[0098] Step 3. Describe the denoising process (sampling process, or inference process) of the diffusion model as:

[0099]

[0100] In the above formula And α t ∈[0, 1] is an adjustable parameter in the diffusion model.

[0101] In the entire diffusion model, t traverses T, T-1, ..., 2, 1 (the adjustable parameter in the diffusion model is α t , σ t ∈[0,1]), there are T equations in total, which are called “the tth equation” according to the value of t; where ∈ θ It is the pre-trained U-Net model in the diffusion model, with θ as the weight. It indicates that the denoising process samples the generated image in the latent space at the t step, and takes

[0102] Further, here It means that the mean of each dimension is 0, and the covariance matrix is ​​the unit matrix I, that is, a high-dimensional normal distribution; in the present invention, all I have the same dimension

[0103] To decode the latent space image representation into a real picture, we need to use the decoder of the variational autoencoder (VAE). This model is denoted as De(·). The real image Z at time t t It is expressed as:

[0104]

[0105] In the denoising process of the diffusion model using the adapter, the image in the latent space is represented as:

[0106]

[0107] Where X0 represents the image condition input, y represents the text input, and F Φ represents the adapter model with Φ as weight.

[0108] In the above diffusion model, the grayscale image A0 obtained in step 2 is passed to the primary adapter. Get vector Pass the grayscale image B0 to the secondary adapter Get vector They are injected into the diffusion model according to the method in the T2I-Adapter model, where Φ1 and Φ2 represent the weights used by the adapter model. The vector injection of a single adapter is as follows: Figure 2 shown.

[0109] In addition, the diffusion model used in the present invention uses U-Net as a denoising network, and uses CLIP, a contrastive language image pre-training, as an encoder to convert text content into a vector. Classifier-Free Guidance is used to continuously inject these two vectors into the Denoising U-Net at each time step of the diffusion model, fix the weight θ of Stable-Diffusion and the weight Φ1 of the primary adapter, and set the weight Φ2 of the secondary adapter as a learnable parameter. The two diffusion generation paths calculate the prediction loss of the noise. The derivation process is as follows:

[0110] For convenience, the generation path where the main adapter is located in the figure is recorded as line 1, and the generation path where the secondary adapter is located is recorded as line 2. The loss used is:

[0111]

[0112] here:

[0113]

[0114] Use with represents the mathematical expectation, Indicates that the time step t follows a uniform distribution on [1,T], ∈~(0,I) indicates that the random noise ∈ follows a random normal distribution, ∈ θ Represents the noise prediction value calculated by the U-Net network with θ as the weight in the diffusion model.

[0115] For route 1, the image sampled by the diffusion model at time step t is recorded as Similar definition Z B,t .

[0116] In order to achieve the goal of controlling the portrait posture through the joint base points, it is necessary to ensure that the images obtained by sampling the predicted noise in routes 1 and 2 have the same pixel distribution as much as possible, except for the pixels with different distributions caused by different joint base points. Since the diffusion model uses gradient quantization to represent the score of the diffusion direction, the difference in the diffusion noise in route 1 and route 2 at a certain time step can be expressed as:

[0117]

[0118] Here p(·) represents the probability distribution. Indicates the gradient, combined with the Tweedie formula, we can know:

[0119]

[0120] as well as:

[0121]

[0122] This quantitatively represents the diffusion direction within the model, and the difference in diffusion direction at time step t for the diffusion model after the two adapter injection vectors are respectively passed can be expressed as:

[0123]

[0124] Therefore, according to equations 15 and 16, the optimization of the diffusion direction of line 1 and line 2 is designed, and the loss function is set:

[0125]

[0126] On the other hand, considering that the difference in diffusion results after injection of the main and secondary adapters in line 1 and line 2 should actually be limited to the posture marked by the joint base point, it is also necessary to limit the posture from the loss function to ensure that:

[0127]

[0128] The right side of the above equation is actually a constant. To satisfy the constraints of equation (19), we can choose to optimize:

[0129]

[0130] The constant term can be discarded to obtain the residual loss function:

[0131]

[0132] So far, two loss functions for local optimization are obtained through equations (18) and (21):

[0133] In addition, in the diffusion model described by equations (9) and (11), the process of deriving equations (15) and (16) using the Tweedie formula is as follows:

[0134] For high-dimensional Gaussian variables where μ z represents the mean of the n-dimensional variable z, Σ z Represents the covariance matrix between variables of each dimension in variable z. The Tweedie formula shows that:

[0135]

[0136] here, represents the mathematical expectation, represents the gradient, and p(·) represents the probability density.

[0137] In the diffusion model:

[0138] Conditional Probability

[0139] Substituting into the Tweedie formula:

[0140]

[0141] Here you can Abbreviated as

[0142] In a pre-trained diffusion model, x t The best generation of Therefore:

[0143]

[0144] Compared with the formula describing the sampling process:

[0145]

[0146] Here∈ θ It represents a fully trained denoising network, which is the convolutional neural network U-Net in the diffusion model (in the diffusion model, the noise prediction must be completed first, and then the image is recovered according to the denoising process formula).

[0147] Step 4. In this step, the global optimization function is determined. For equations (18) and (21), the two local loss functions cannot be added directly because of their different scales. To address this problem, most traditional methods add a coefficient to one of them (the coefficient is tested for a suitable value during the training process) and then add two or more functions for local optimization. In this step, a new method is used to directly convert the two loss functions for local optimization to the same scale to optimize the output character posture graph.

[0148] Since the model obtained by optimizing the main adapter enables the Stable-Diffusion model to stably distribute pixels to the joint base point positions (calibrating the posture of the portrait) when sampling in fixed weights, it is equivalent to completing an optimization problem with constraints when training the adapter. Take line 1 as an example:

[0149]

[0150] This can be written as:

[0151]

[0152]

[0153] Here λ>0.

[0154] In formula (12), only Φ1 is a learnable parameter. Combined with the previous analysis, we have:

[0155]

[0156] Therefore, and The optimal Φ1 is the same, that is:

[0157]

[0158] This shows that:

[0159]

[0160] Similarly, for line 2:

[0161]

[0162] To simplify the representation, the mathematical expectation is The data quintuple (A0, B0, τ(y), Z) is omitted in the subscript A ,Z B )

[0163] For formula (18), note that:

[0164]

[0165]

[0166] The first scaling here uses the triangle inequality in L2 linear space, and the second scaling uses the Cauchy inequality.

[0167] Consider the part of the upper bound of equation (18) where the gradient with respect to Φ2 is not zero:

[0168]

[0169] Combining equations (18) and (21), the global optimization function is set as:

[0170]

[0171] Here it is called contrast error. When calculating this error later, the coefficient of the previous term is recorded as Q, that is:

[0172]

[0173] As a test of the correctness of the derived model, we can check whether the global loss after a sufficiently large number of iterative training has converged to the theoretical value:

[0174]

[0175] Embodiment 1:

[0176] Figure 1 The training architecture of the overall model is used to achieve the stabilization of the weight of the secondary adapter relative to the primary adapter, that is, the stabilization fine-tuning training method of the secondary adapter relative to the primary adapter in the diffusion model. When training the sample stability relative to the user's given data set, it is necessary to provide input data in the form of images of key poses. In fact, since the training is based on the key poses of the image obtained through noise prediction and sampling at each step in the diffusion model, it is not necessary to ensure the similarity of the content in the source image when obtaining the key pose. According to this analysis, the specific content of the images in the data set has little effect on the training results, but in order to avoid the final training result of forcibly diffusing the text description content to unreasonable key poses under the action of the secondary adapter, the image description (Image Caption) model is used to generate the description of the images in the data set.

[0177] Step 1. Select the public datasets MPII-Human Pose Estimation, Cricket Shots Dataset and Yogo Posture Dataset as the original image datasets, aiming to obtain key posture data with large differences. The number of images exceeds 30,000, and 3,000 images are randomly selected when making the training dataset.

[0178] For each original image Z in the image dataset A (1 of 3000 images), use the pre-trained vit-gpt2-image-captioning model to generate a description of the data in the image and annotate the image content.

[0179] Considering the practical problems in the calculation of the machine video memory, we first set the downsampling factor factor = 4, divide the length and width of the image by the value of factor and round it up as the target shape for downsampling, use bilinear interpolation (you can also use the nearest neighbor interpolation method, the resampling method based on local pixels, the cubic interpolation method based on the 4×4 pixel neighborhood, or the Lanczos interpolation method based on the 8×8 pixel neighborhood) for downsampling, and use the sampling result to replace the original image Z A (That is, all original images are scaled down proportionally).

[0180] For each image Z A, use vit-gpt2-image-captioning (a model for image annotation text description of input images) to randomly select 1500 images with max_length=30 and beams=5 (max_length and beams are parameters in the vit-gpt2-image-captioning model) to obtain the text description y corresponding to the image A .

[0181] Take the image processed by the above method as input, and use the openpose model to obtain the corresponding key pose dot matrix output: Image output A0.

[0182] Step 2. According to the steps in step 1, obtain the original image Z A , Z B , grayscale images A0, B0 describing the posture, and the corresponding text y describing the image A ,y B . The actual data set used for training is recorded as TRAIN, which is:

[0183]

[0184] Step 3. The five data in each quintuple are the input of the entire model. After inputting these data, they are injected into the U-Net in the diffusion model according to the ControlNet architecture and the method provided by T2I-Adapter. For each time step t, calculate the current comparison error:

[0185]

[0186] here:

[0187]

[0188] θ is a fixed parameter of the stable diffusion model, Φ1 is a fixed main adapter parameter, Φ2 is a learnable parameter of the secondary adapter, and the function De(·) represents decoding the sampling result using a variational autoencoder (VAE) decoder.

[0189] Step 4. Loss optimization, repeat the process of step 3, take T = 50 to calculate the single-step loss value, and obtain the total loss:

[0190]

[0191] Furthermore, the above α t(t=1,2,...,T) is an adjustable parameter in the diffusion model, which is taken as an arithmetic progression with an initial value of 0.00002 and a final value of 1.0.

[0192] The experimental results are as follows Figure 3 As shown in the figure, the experiment shows that by optimizing the detection model for the human joint base points and embedding the adapter, the model superiority of the former can be transferred to the diffusion model. Compared with the model directly trained under the ControlNet architecture or T2I-Adapter, slight changes in the input of the joint base points will affect the model's response to the text; the secondary adapter retains the ability of the primary adapter to adsorb pixels based on the text, while preventing the partial information encoded by the text from being erased when the vector is injected into the denoising network, so that given the same text, the generated image only changes in the human posture. In the multi-adapter mode, the input of the adapter is no longer limited to the fixed point of the human posture, but can be a target detection and recognition model such as semantic segmentation and key posture. The corresponding L2 norm loss is defined as the loss calculation in the corresponding model.

[0193] Since the adapter only interferes with the generation during injection in the image generation process, the generation ability of the overall diffusion model is still determined by the stable diffusion model with locked weights. The secondary adapter also interferes with the pixel distribution position, but is relatively stable compared to the text.

[0194] The original adapter can input the generation process of the constraint influence diffusion model in the form of an image under the ControlNet or T2I-Adapter architecture, so that the generation result is close to the constraint image; the secondary adapter weights trained according to the present invention improve the stability during constraint injection, reduce the impact of constraint injection on the generation main line, and greatly improve the model's ability to restore scenes through text.

[0195] The method proposed in the present invention takes about 5 hours on 4 NVIDIA-RTX4090 graphics cards (single graphics card memory 24GB, each graphics card has 12-core CPU and 90GB memory, floating point computing power: single precision 82.58TFLOPS / semi-precision 165.2Tensor TFLOPS), and the total loss finally converges to the theoretical value (error 0.04%). By comparing the generation effect of the diffusion model under different frameworks and guidance methods and the fine-tuning method in the present invention, the superiority of the present invention can be shown.

[0196] The experimental results are as follows Figure 3As shown, using the "stable-diffusion" model, CLIP is pre-trained with contrasting language images to encode the input text and diffuse it in the latent space. After inputting "a girl waving her left hand", the result a shown is generated. The image content is somewhat different from the text. Using the T2I-Adapter model under the ControlNet architecture, the input "a girl waving her left hand" is pre-trained with CLIP to encode the input text and diffuse it in the latent space. At the same time, the adapter encodes the key posture of the input and projects it into the latent space, and adds it to the diffusion results of each step to generate the result b shown. The key posture of the rendering is consistent with the input posture. Still using the T2I-Adapter model under the ControlNet architecture, using the main adapter fine-tuned by the main and sub-adapter joint training method of the present invention, the input "a girl, waving her left hand" is pre-trained with the contrasting language image CLIP to encode the input text and then diffuse it in the latent space. At the same time, the adapter encodes the input key posture and projects it into the latent space, and adds it to the diffusion result of each step to generate the result c shown. The key posture of the rendering is basically consistent with the input posture.

Claims

1. A human posture scene restoration method based on an adaptive network enhanced diffusion model, characterized in that: The following steps are involved: Step 1. Each set of training data is represented as a five-tuple (A (N) ,B (N) ,y,Z A ,Z B ), where y is the text description label, A (N) ,B (N) The number of elements in both The dot matrix, is the image data, where the number 3 means that the image is stored in RGB format, w and h represent the width and height of the image respectively; δ (·) represents the model of the coordinate point set of the character joint base points in the input image, with δ as the weight; according to the image Z A ,Z B The obtained dot matrix A (N) ,B (N) ; Step 2. Use dot matrix A (N) ,B (N) Create grayscale images A0 and B0 respectively; Step 3. Obtain the difference between the primary and secondary adapters in the diffusion model from the diffusion model, calculate the loss function based on the difference, and calculate the surplus loss; the specific process is as follows: 3.

1. The reasoning process of the diffusion model is described as: In the above formula And α t ∈[0,1] is an adjustable parameter in the diffusion model; In the diffusion model, time step t traverses T, T-1, ..., 2, 1, a total of T equations, where ∈ θ is the pre-trained U-Net model in the diffusion model, with θ as the weight; It indicates that the denoising process samples the generated image in the latent space at the t step, and takes It means that the mean of each dimension is 0, and the covariance matrix is ​​the unit matrix I, that is, a high-dimensional normal distribution; 3.

2. Decode the latent space image representation in the diffusion model into a real image using the decoder of the variational autoencoder, denoted as De(·), the real image Z at time t t It is expressed as: In the denoising process of the diffusion model using the adapter, the image in the latent space is represented as: Where X0 represents the image condition input, y represents the text input, and F Φ represents the adapter model with Φ as weight; In the above diffusion model, the grayscale image A0 is passed to the main adapter Get vector Pass the grayscale image B0 to the secondary adapter Get vector They are injected into the diffusion model according to the method in the T2I-Adapter model, and Φ1 and Φ2 represent the weights used by the adapter model; 3.

3. The diffusion model uses U-Net as the denoising network and uses CLIP, a pre-trained contrastive language image, as an encoder to convert text content into vectors, using category-free guidance; The generated path where the main adapter is located is recorded as line 1, and the generated path where the secondary adapter is located is recorded as line 2; for line 1, the image sampled by the diffusion model at time step t is recorded as Similar definition Z B,t ; The difference between the primary and secondary adapters in route 1 and route 2 is expressed as: Combined with the Tweedie formula: The expected loss function is obtained from the above formula: 3.

4. Set the surplus loss function and restrict the posture from the loss function. The surplus loss function is: Step 4. Obtain a global optimization function based on the loss function and the surplus loss function, and optimize the output character posture graph.

2. The human body posture scene restoration method based on the adaptive network enhanced diffusion model according to claim 1 is characterized in that: In step 1, the dot matrix A is obtained (N) ,B (N) The specific process is: 1.

1. Text description annotation y∈{y A ,y B },y A ,y B Represents image Z A ,Z B English text description corresponding to the image content; 1.

2. Through model Ψ δ (·) The process of calculating the joint base point set in the input image Y is described as: use Indicates the coordinate position of the i-th base point in the w×h coordinate system, and uses y A,i Indicates the ordinate position of the i-th base point in image A, y B,i Similarly, similar to the definition of x A,i ,x B,i Through this model, we get a points; use the normalization function defined below to normalize the points in the lattice, that is, compress them. The normalized representation of is: in For adjustable bias; For the entire lattice, the compression process is expressed as: Combining equations (1) and (3), it can be expressed in the form of a composite function: Then the dot matrix The calculation method is as follows: in 3. According to claim 2, the method for restoring a human posture scene based on an adaptive network enhanced diffusion model is characterized in that the specific process of step 2 is as follows: Expand the compressed lattice proportionally to [0,i]×[0,i], u=min{w,h}; The grayscale values ​​of the coordinate positions corresponding to the points that have appeared in the dot matrix are set to 0, and the remaining positions are set to 255, and two grayscale images A0 and B0 are obtained respectively.

4. The method for restoring a human posture scene based on an adaptive network enhanced diffusion model according to claim 3, characterized in that step 2 further comprises: respectively converting the original image Z A ,Z B The text conversion model is passed in, and the vector encoding of the obtained text is τ(y). The five-tuple of each data becomes: (A0, B0, τ(y), Z A ,Z B ), while setting adjustable parameters represents the total diffusion step length of the diffusion model; y = y A Or y = y B , get two pieces of data; The grayscale images A0 and B0 made from the original images are the image data required for the final training. The data set used in the actual training is recorded as TRAIN. For all grayscale image combinations A0 and B0:

5. The method for restoring a human posture scene based on an adaptive network enhanced diffusion model according to claim 4, characterized in that: The global optimization function described in step 4 is as follows:

6. The method for restoring a human posture scene based on an adaptive network enhanced diffusion model according to claim 5, characterized in that: Also included in step 4, the theoretical value of global loss convergence is: Determine whether the model is sufficiently trained.

Images