Training method, prediction method and defect image generation method for noise prediction

Abstract

Training a second diffusion network through a training method, a prediction method and a defect image generation method for noise prediction; obtaining a first mask feature and a second mask feature; obtaining a new first image feature according to the second mask feature; performing step-by-step noise addition on the new first image feature according to a preset total noise addition step number to obtain a third image feature; inputting the third image feature and the first mask feature into a second diffusion network to obtain high-dimensional noise predicted by the second diffusion network, and further obtaining fifth high-dimensional noise; gradually denoising the third image feature according to the obtained fifth high-dimensional noise to obtain a first denoised feature map; and obtaining a target image according to the first de-noising feature map. According to the technical scheme, the image meeting the user requirement is generated in an auxiliary mode through additional information such as the point cloud data, the information diversity is enhanced on the basis of guaranteeing the information integrity, the target image generation effect is enhanced, and therefore high-quality images are generated under pixel-level precision control.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a training method, a prediction method and a defect image generation method for noise prediction. Background Art

[0002] In recent years, deep learning has become the focus of attention in various fields at home and abroad. Deep learning includes supervised learning and unsupervised learning. In the field of computer vision, supervised learning refers to training neural networks through one-to-one correspondence between images and annotation information, so that they can complete tasks such as classification, object detection and semantic segmentation. Unsupervised learning only requires images and does not require additional annotation information for training.

[0003] In most scenarios, supervised learning works better than unsupervised learning. However, supervised learning requires a large amount of images and annotation information. Annotating images is time-consuming and laborious. In addition, the number of defect images in industrial scenarios such as defect detection is small, which makes it difficult to collect enough sample data. These factors greatly limit the verification and implementation of deep learning algorithms.

[0004] Existing technologies often make up for the shortage of actual defect samples by synthesizing defect images. In the case of a limited number of original defect samples, the diffusion model can create more and more diverse defect images, greatly enriching the sample data. However, since the diffusion model usually relies on text information to guide the generation of images, it is not accurate enough when processing pixel-level image details. In application scenarios that require high realism and detailed texture expression, such as industrial defect detection, the images generated by the diffusion model may be difficult to achieve the expected image quality. Secondly, the diffusion model often contains a huge amount of parameters, making it difficult to train in specific fields. That is, the traditional diffusion network has problems such as the inability to accurately process image details and the difficulty of training caused by the huge amount of parameters.

[0005] Therefore, in order to overcome the shortcomings of traditional diffusion networks, it is necessary to design a network structure with few parameters and high-precision processing of image details; further, on this basis, we can think about how to predict noise based on this network, and even generate high-quality defective images based on the obtained noise prediction results. Summary of the invention

[0006] In view of the above problems, the present application provides a training method, a prediction method and a defect image generation method for noise prediction, which are described in detail below.

[0007] According to the first aspect, some embodiments provide a training method for noise prediction, comprising:

[0008] Acquire a second diffusion network formed by adding a control branch network to a pre-trained first diffusion network, wherein the first diffusion network is a preset diffusion network;

[0009] The second diffusion network is trained for one or more rounds according to a training set, wherein the training set includes one or more sets of training data, and any set of training data includes an image pair consisting of a grayscale image and a height image, and a mask image; in the training process of any training round, the second diffusion network is trained according to any set of training data in the training set, including:

[0010] A data processing step, in a current training round, obtaining an initial first image feature according to an image pair in current training data, and obtaining a first mask feature according to a mask image in current training data;

[0011] A noisy step, obtaining a first high-dimensional noise; when the current number of noisy steps is the first number of noisy steps, the initial first image feature is denoised according to the first high-dimensional noise, to obtain a first image feature after the noisy operation corresponding to the first number of noisy steps; when the current number of noisy steps is not the first number of noisy steps, the first image feature after the noisy operation corresponding to the previous number of noisy steps is continuously noisy according to the first high-dimensional noise, to obtain a first image feature after the noisy operation corresponding to the current number of noisy steps;

[0012] A fusion processing step, inputting the first image feature after the denoising operation corresponding to the current denoising step number and the first mask feature into the second diffusion network, and in the second diffusion network, respectively performing fusion processing on the first image feature after the denoising operation corresponding to the current denoising step number and the first mask feature; wherein the second diffusion network is used to perform noise prediction on the input of the second diffusion network to output high-dimensional noise when the current denoising step number is equal to the preset total denoising step number of the current training round;

[0013] The training judgment step includes: if the current number of noise addition steps is less than the preset total number of noise addition steps of the current training round, the current number of noise addition steps is increased by 1 step to update the current number of noise addition steps, and the training is continued from the noise addition step; if the current number of noise addition steps is equal to the preset total number of noise addition steps of the current training round, the high-dimensional noise output by the second diffusion network is used as the second high-dimensional noise; the loss function of the second diffusion network is obtained according to the first high-dimensional noise and the second high-dimensional noise; if the loss function converges, the training is stopped.

[0014] In some embodiments, the training determination step further includes:

[0015] If the loss function has not converged, the training is continued from the data processing step according to the next set of training data in the training set; if the current training data is the last set of training data in the training set, the training of the current training round is stopped.

[0016] In some embodiments, the training determination step further includes:

[0017] After stopping the training of the current training round, the second diffusion network after the current training round is used as the second diffusion network used in the training process of the next training round, and the training of the next training round is started from the data processing step according to the first set of training data in the training set.

[0018] In some embodiments, the training determination step further includes:

[0019] Before stopping the training of the current training round and starting the training of the next training round, it is also determined whether a preset maximum training round is reached. If reached, the training is stopped.

[0020] In some embodiments, the step of continuing to add noise to the first image feature after the noise addition operation corresponding to the previous noise addition step number according to the first high-dimensional noise to obtain the first image feature after the noise addition operation corresponding to the current noise addition step number includes:

[0021] For the t-th number of noise addition steps, where t is a positive integer, the noise intensity of the first high-dimensional noise corresponding to the t-th number of noise addition steps is obtained according to the preset intensity of the first high-dimensional noise corresponding to the first number of noise addition steps and the preset intensity of the first high-dimensional noise corresponding to the last number of noise addition steps;

[0022] The noise intensity of the first high-dimensional noise corresponding to the t-th noisy step under the square root is used as the reference weight of the first high-dimensional noise corresponding to the t-th noisy step; the difference between 1 and the noise intensity of the first high-dimensional noise corresponding to the t-th noisy step is calculated, and the difference under the square root is used as the reference weight of the first image feature after the noisy operation corresponding to the t-1-th noisy step; the first image feature after the noisy operation corresponding to the t-1-th noisy step and the first high-dimensional noise corresponding to the t-th noisy step are multiplied by the corresponding reference weights respectively, and the multiplication result is used as the first image feature after the noisy operation corresponding to the t-1-th noisy step.

[0023] In some embodiments, obtaining the noise intensity of the first high-dimensional noise corresponding to the t-th number of noise addition steps according to the preset intensity of the first high-dimensional noise corresponding to the first number of noise addition steps and the preset intensity of the first high-dimensional noise corresponding to the last number of noise addition steps includes:

[0024] Calculate the square difference between the preset intensity of the first high-dimensional noise corresponding to the first noise addition step number and the preset intensity of the first high-dimensional noise corresponding to the last noise addition step number, and obtain the difference between the preset total number of noise addition steps in the current training round and 1; take the ratio between the obtained square difference and the obtained difference as the noise intensity change of a single noise addition;

[0025] Multiply the difference between the current number of noise addition steps and 1 by the change in noise intensity of a single noise addition, and use the resulting product as the cumulative change in noise intensity corresponding to the current number of noise addition steps; add the cumulative change in noise intensity corresponding to the current number of noise addition steps to the square of the preset intensity of the first high-dimensional noise corresponding to the first number of noise addition steps, and then use the resulting addition result under the square root as the noise intensity of the first high-dimensional noise corresponding to the tth number of noise addition steps.

[0026] In some embodiments, in the second diffusion network, the first image feature after the noise addition operation corresponding to the current number of noise addition steps and the first mask feature are fused separately, including:

[0027] The fusion process includes a first fusion process and a second fusion process;

[0028] Obtaining a current noise step vector according to the current noise step number; for any network module in the second diffusion network, first performing a first fusion process on the input content of any network module and the current noise step vector to obtain a first fusion result of any network module;

[0029] Any one of the training data sets further includes point cloud data and / or natural language data of an object corresponding to the image, and a fourth image feature corresponding to the current training data is obtained according to the point cloud data or natural language data or grayscale image in the current training data; a first general feature is obtained, and any one of the first general feature and the fourth image feature corresponding to the current training data is used as an attention parameter corresponding to the current training data; a second mask feature corresponding to the current training data can be obtained according to the mask image in the current training data;

[0030] The first fusion result of any one of the network modules is subjected to a second fusion process with the attention parameters corresponding to the current training data and the second mask feature to obtain the second fusion result of any one of the network modules, and the second fusion result of any one of the network modules is used as the output content of any one of the network modules.

[0031] In some embodiments, the input content of any one of the network modules includes:

[0032] If any one of the network modules is the first encoding module in the first diffusion network, the input content of any one of the network modules is the first image feature after the noise addition operation corresponding to the current noise addition step number; if any one of the network modules is the encoding module in the first diffusion network and is not the first encoding module, or if any one of the network modules is the intermediate module in the first diffusion network, the input content of any one of the network modules is the output content of the previous encoding module in the first diffusion network;

[0033] If any one of the network modules is the first coding module in the control branch network, the input content of any one of the network modules is the first mask feature; if any one of the network modules is the coding module in the control branch network and is not the first coding module, or if any one of the network modules is an intermediate module in the control branch network, the input content of any one of the network modules is the output content of the previous coding module in the control branch network;

[0034] If any one of the network modules is the first decoding module in the first diffusion network, the previous decoding module of any one of the network modules is the intermediate module; if any one of the network modules is the decoding module of the first diffusion network and is not the first decoding module, the previous decoding module of any one of the network modules in the first diffusion network is obtained; the output content of the encoding module with the same size as the previous decoding module in the control branch network is convolved with a convolution layer to obtain a branch fusion result; the branch fusion result, the output content of the previous decoding module and the output content of the encoding module with the same size as the previous decoding module in the first diffusion network are fused, and the obtained fusion result is used as the input content of any one of the network modules.

[0035] In some embodiments, obtaining a current noise adding step number vector according to a current noise adding step number includes:

[0036] Set a preset length of the noise step vector, and obtain the logarithmic function value corresponding to the current noise step; for the kth element in the noise step vector, calculate the ratio between k-1 and the preset length; calculate the product between the obtained logarithmic function value and the obtained ratio, and use the exponential function value corresponding to the inverse of the obtained product as the kth element value in the current noise step vector.

[0037] In some embodiments, obtaining a loss function of a second diffusion network according to the first high-dimensional noise and the second high-dimensional noise includes:

[0038] For any element value of the second high-dimensional noise, calculate the logarithmic function value of the arbitrary element value, multiply the obtained logarithmic function value with the element value of the first high-dimensional noise at the corresponding position, and obtain the multiplication result corresponding to the arbitrary element value; obtain the cumulative sum of the multiplication results corresponding to all element values ​​of the second high-dimensional noise, and use the inverse of the obtained cumulative sum as the loss function of the second diffusion network.

[0039] According to the second aspect, some embodiments provide a noise prediction method, comprising:

[0040] Acquire a second diffusion network trained by the training method;

[0041] The predicted high-dimensional noise is obtained according to the second diffusion network, including:

[0042] Acquire a first mask feature and a first image feature;

[0043] The first image feature is gradually denoised, and when the number of denoising steps reaches a preset total number of denoising steps, the first image feature after denoising is obtained, the first image feature after denoising and the first mask feature are input into a second diffusion network, and in the second diffusion network, the first mask feature and the first image feature after denoising are respectively fused; the output of the second diffusion network is used as the predicted high-dimensional noise.

[0044] According to the third aspect, some embodiments provide a defect image generating method, comprising:

[0045] Acquire a second diffusion network trained by the training method;

[0046] Acquire a mask image, and obtain a first mask feature and a second mask feature according to the mask image;

[0047] Obtaining a new first image feature according to the second mask feature;

[0048] Stepwise adding noise to the new first image feature according to a preset total number of steps for adding noise, and recording the first image feature after the last adding noise as the third image feature;

[0049] Performing a denoising operation on the third image feature according to the first mask feature and the second diffusion network, and stopping the denoising when the number of denoising times is equal to the preset total number of denoising steps to obtain a first denoising feature map;

[0050] A target image is obtained according to the first denoising feature map.

[0051] In some embodiments, obtaining a new first image feature according to the second mask feature includes:

[0052] Get comprehensive feature vector;

[0053] For any pixel point in the new first image feature, when the pixel value of the pixel point at the corresponding position in the second mask feature is 1, the foreground standard deviation vector in the comprehensive feature vector is multiplied by the element value of the corresponding dimension of the standard high-dimensional noise, and the sum of the multiplied vector and the foreground mean vector is used as the high-dimensional feature vector corresponding to any pixel point in the new first image feature; when the pixel value of the pixel point at the corresponding position in the second mask feature is 0, the high-dimensional feature vector corresponding to any pixel point in the new first image feature is obtained according to the comprehensive feature vector or the reference image feature.

[0054] In some embodiments, obtaining the comprehensive feature vector includes:

[0055] In any training round of the training method, the first feature vector is iteratively updated once in the data processing step of each set of training data, wherein:

[0056] For the current training data, in the data processing step, the background area and foreground area of ​​the initial first image feature corresponding to the current training data and the mask image in the current training data are obtained, and the first feature vector corresponding to the current training data is obtained according to the background area and foreground area of ​​the initial first image feature;

[0057] According to the first eigenvector corresponding to the current training data and the first eigenvector updated in the last iteration, the first eigenvector updated in the current iteration is obtained; wherein, if the current training data is the first set of training data in the training set, the first eigenvector updated in the last iteration is a 0 vector;

[0058] The first feature vector after the last iteration update in any training round of the second diffusion network is used as the comprehensive feature vector.

[0059] In some embodiments, obtaining a first feature vector corresponding to current training data according to the background area and the foreground area of ​​the initial first image feature includes:

[0060] The first feature vector corresponding to the current training data includes four vectors, and the four vectors are respectively the background mean vector, background standard deviation vector, foreground mean vector and foreground standard deviation vector of the initial first image feature; wherein the background mean vector of the initial first image feature is the mean vector of the high-dimensional feature vector corresponding to all pixels in the background area of ​​the initial first image feature; the background standard deviation vector of the initial first image feature is the standard deviation vector of the high-dimensional feature vector corresponding to all pixels in the background area of ​​the initial first image feature; the foreground mean vector of the initial first image feature is the mean vector of the high-dimensional feature vector corresponding to all pixels in the foreground area of ​​the initial first image feature; the foreground standard deviation vector of the initial first image feature is the standard deviation vector of the high-dimensional feature vector corresponding to all pixels in the foreground area of ​​the initial first image feature.

[0061] In some embodiments, obtaining the first feature vector updated in the current iteration according to the first feature vector corresponding to the current training data and the first feature vector updated in the previous iteration includes:

[0062] In the g-th iterative update of a training round, for any first eigenvector, a momentum coefficient is set, and the momentum coefficient is used as the reference weight of the any first eigenvector after the g-1-th iterative update, and the difference between 1 and the momentum coefficient is used as the reference weight of the first eigenvector corresponding to the g-th group of training data; the product of the any first eigenvector obtained after the g-1-th iterative update, the first eigenvector corresponding to the g-th group of training data and the corresponding reference weight is calculated respectively, and the sum of the two products is used as the any first eigenvector after the g-th iterative update.

[0063] In some embodiments, when the pixel value of the pixel point at the corresponding position in the second mask feature is 0, obtaining the high-dimensional feature vector corresponding to any pixel point in the new first image feature according to the comprehensive feature vector or the reference image feature includes:

[0064] For any pixel point in the new first image feature, when the pixel value of the pixel point at the corresponding position in the second mask feature is 0, the background standard deviation vector in the comprehensive feature vector is multiplied by the element value on the corresponding dimension of the standard high-dimensional noise, and the sum of the multiplied vector and the background mean vector is used as the high-dimensional feature vector corresponding to any pixel point in the new first image feature; or, a reference image feature is obtained according to a preset image pair; the high-dimensional feature vector corresponding to the corresponding pixel point in the reference image feature is used as the high-dimensional feature vector corresponding to any pixel point in the new first image feature.

[0065] In some embodiments, performing a denoising operation on the third image feature according to the first mask feature and the second diffusion network includes:

[0066] For the t'th denoising operation, the third image feature after the last denoising operation and the first mask feature are input into the second diffusion network, and the third image feature after the last denoising operation and the first mask feature are respectively fused in the second diffusion network; the third high dimensional noise and the fourth high dimensional noise corresponding to the t'th denoising operation are obtained according to the output of the second diffusion network; the fifth high dimensional noise corresponding to the t'th denoising operation is obtained according to the third high dimensional noise and the fourth high dimensional noise corresponding to the t'th denoising operation; the third image feature after the last denoising operation is denoised according to the fifth high dimensional noise corresponding to the t'th denoising operation, to obtain the third image feature after the t'th denoising operation.

[0067] In some embodiments, the fusing process of the third image feature after the last denoising operation and the first mask feature in the second diffusion network respectively includes:

[0068] Obtain a denoising step vector corresponding to the t'th denoising operation, and for any network module in the second diffusion network, first perform a first fusion process on the input content of any one of the network modules and the denoising step vector corresponding to the t'th denoising operation to obtain a first fusion result of the any one of the network modules; then perform a second fusion process on the first fusion result of the any one of the network modules, the attention parameter and the second mask feature to obtain a second fusion result of the any one of the network modules; wherein the third image feature after the last denoising operation is the input content of the first encoding module of the first diffusion network in the second diffusion network, and the first mask feature is the input content of the first encoding module of the control branch network in the second diffusion network.

[0069] In some embodiments, obtaining the denoising step number vector corresponding to the t'th denoising operation includes: taking the result of subtracting t' from a preset total denoising step number and adding 1 as the denoising step number corresponding to the t'th denoising operation; and taking the denoising step number vector corresponding to the denoising step number as the denoising step number vector corresponding to the denoising operation.

[0070] In some embodiments, obtaining the third high-dimensional noise and the fourth high-dimensional noise corresponding to the t'th denoising operation according to the output of the second diffusion network includes:

[0071] The fourth image feature and the first general feature are obtained; for the t'th denoising operation, the attention parameter used by any network module in the second diffusion network for the second fusion processing is set to the fourth image feature, the third image feature after the last denoising operation and the first mask feature are input into the second diffusion network, and the high-dimensional noise predicted by the second diffusion network is recorded as the third high-dimensional noise corresponding to the t'th denoising operation; the attention parameter used by any network module in the second diffusion network for the second fusion processing is set to the first general feature, and the third image feature after the last denoising operation and the first mask feature are input into the second diffusion network again, and the high-dimensional noise predicted by the second diffusion network is recorded as the fourth high-dimensional noise corresponding to the t'th denoising operation.

[0072] According to the training method for noise prediction in the above embodiment, the grayscale image and the height map are first channel-superimposed to extract the first image feature, the first mask feature and the second mask feature are extracted according to the mask image, and different types of global features of the object, i.e., the fourth image feature, are extracted according to the point cloud, grayscale image and natural language of the object. Then, through contrast learning, the global feature is made to contain not only the basic information of the object itself, but also the relevant information between the objects; finally, the second diffusion network composed of the first diffusion network and the control branch network is trained according to the first image feature, the first mask feature, the second mask feature and the fourth image feature or the first general feature, and the parameters inside the first diffusion network are fixed, and only the parameters in the control branch network and some learnable parameters are updated, thereby reducing the amount of parameters inside the network during the training process, reducing the difficulty of training, and achieving precise fine-tuning without changing the original diffusion model; at the same time, the control branch network provides additional information for the first diffusion network, thereby improving the phenomenon that the first diffusion network is difficult to achieve pixel-level image generation;

[0073] According to the noise prediction method of the above embodiment, firstly, a first mask feature and a first image feature are obtained; the obtained first image feature is gradually denoised to obtain a first image feature after denoising; then the first image feature after denoising and the first mask feature are input into a second diffusion network, and in the second diffusion network, the first mask feature and the first image feature after denoising are fused respectively; finally, the output of the second diffusion network is used as the predicted high-dimensional noise; in this process, a second fusion process is performed with attention parameters in different network modules, and by setting the attention parameters, a noise prediction result under conditions (i.e., when additional information such as point cloud is involved) or unconditional (i.e., when additional information such as point cloud is not involved);

[0074] According to the defect image generation method of the above embodiment, whether to require the defect sample data to have a specified background area to generate a new first image feature can be selected according to actual needs, so that while enriching the defect sample data, it can better meet user needs and make the final target image more diverse in form; in addition, since the second diffusion network used in the target image acquisition process can input additional information such as point clouds, and the mask image contains rich location information, it can enhance the diversity of information while ensuring the integrity of the information, thereby enhancing the generation effect of the target image, and then achieving the generation of high-quality target images under pixel-level precision control. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Figure 1 is a method flow chart of a training method for noise prediction;

[0076] Figure 2 is a method flow chart of a noise prediction method;

[0077] Figure 3 A method flow chart of a defect image generation method;

[0078] Figure 4 is the training flow chart of the second diffusion network;

[0079] Figure 5 Schematic diagram of the network structure of the second diffusion network. DETAILED DESCRIPTION

[0080] The present invention is further described in detail below by specific embodiments in conjunction with the accompanying drawings. Wherein similar elements in different embodiments adopt associated similar element numbers. In the following embodiments, many detailed descriptions are for making the present application better understood. However, those skilled in the art can easily recognize that some features can be omitted in different situations, or can be replaced by other elements, materials, methods. In some cases, some operations related to the present application are not shown or described in the specification, this is to avoid the core part of the present application being overwhelmed by too much description, and for those skilled in the art, it is not necessary to describe these related operations in detail, and they can fully understand the related operations according to the description in the specification and the general technical knowledge in the art.

[0081] In addition, the features, operations or characteristics described in the specification can be combined in any appropriate manner to form various implementations. At the same time, the steps or actions in the method description can also be interchanged or adjusted in a manner that is obvious to those skilled in the art. Therefore, the various sequences in the specification and the drawings are only for the purpose of clearly describing a certain embodiment and are not meant to be a required sequence, unless otherwise specified that a certain sequence must be followed.

[0082] The serial numbers of the components in this document, such as "first", "second", etc., are only used to distinguish the objects described and do not have any order or technical meaning. The "connection" and "coupling" mentioned in this application, unless otherwise specified, include direct and indirect connections (couplings).

[0083] In an embodiment of the present invention, the grayscale image and the height map are firstly channel-superimposed to extract the first image feature, the first mask feature and the second mask feature are extracted according to the mask image, and different types of global features of the object, i.e., the fourth image feature, are extracted according to the point cloud, the grayscale image and the natural language of the object. Then, through contrast learning, the global feature is made to contain not only the basic information of the object itself, but also the relevant information between the objects. Finally, the second diffusion network composed of the first diffusion network and the control branch network is trained according to the first image feature, the first mask feature, the second mask feature and the fourth image feature or the first general feature, and the control branch network is used to provide additional information for the first diffusion network. information, thereby improving the phenomenon that the first diffusion network is difficult to achieve pixel-level image generation; then, according to actual needs, it is selected whether to require the defect sample data to have a specified background area to generate a new first image feature, so that while enriching the defect sample data, it can better meet user needs and make the final target image more diverse in form; in addition, since the second diffusion network used in the target image acquisition process can input additional information such as point clouds, and the mask image contains rich location information, it can enhance the diversity of information on the basis of ensuring the integrity of the information, thereby enhancing the generation effect of the target image, and then achieving the generation of high-quality target images under pixel-level precision control.

[0084] Please refer to Figure 1 In some embodiments, a training method for noise prediction is provided, which comprises the following steps:

[0085] Step S100: training a feature extraction network to obtain a first image feature extraction network, a first feature decoding network, a first mask feature extraction network, and a second feature extraction network.

[0086] The feature extraction network involved in this embodiment includes a first image feature extraction network, a first feature decoding network, a first mask feature extraction network and a second feature extraction network;

[0087] In some embodiments, the process of training the first image feature extraction network and the first feature decoding network includes: firstly training a variational autoencoder, which is referred to as the first image variational autoencoder, and using the encoder in the first image variational autoencoder as the first image feature extraction network, and using the decoder in the first image variational autoencoder as the first feature decoding network;

[0088] The basic unit of the data set used to train the first image variational autoencoder is a training image pair, each training image pair contains a grayscale image and a height map; the result of channel superposition of the grayscale image and the height map in an image pair is recorded as δ, at this time, the grayscale information and the height information correspond to a channel respectively, so the result of the channel superposition can be regarded as a two-channel image, and then the result of the channel superposition is input into the first image variational autoencoder, and the first image variational autoencoder will output the reconstructed grayscale image and height map, that is, the output of the first image variational autoencoder can also be regarded as a reconstructed two-channel image;

[0089] According to the distance function between the output and input of the first image variational autoencoder, as the loss function of the first image variational autoencoder, in this embodiment, the distance function used to train the first image variational autoencoder is the Euclidean distance, then the loss function of the first image variational autoencoder is loss θ It can be expressed as:

[0090] loss θ =(θ vae (δ)-δ) 2

[0091] Among them, θ vae is the first image variational autoencoder, δ is the result of channel superposition of the grayscale image and the height map, and it is used as the input of the first image variational autoencoder; θ vae (δ) represents the output of the first image variational autoencoder.

[0092] It should be noted that if a grayscale image lacks a pre-made corresponding height map, a depth estimation algorithm such as depthanything can be used to process the grayscale image to obtain the height map corresponding to the grayscale image.

[0093] In some embodiments, the process of obtaining the first mask feature extraction network includes: retraining a new variational autoencoder, and calling it a first mask variational autoencoder, and using the encoder of the first mask variational autoencoder as the first mask feature extraction network.

[0094] The data set used for training the first mask variational autoencoder is a mask image data set composed of a large number of mask images, wherein the pixel points with a pixel value of 1 in the mask image belong to the foreground area, and the pixel points with a pixel value of 0 represent the background area; after the mask image is input into the first mask variational autoencoder, the first mask variational autoencoder will output a reconstructed mask image, and the distance function between the output and input of the first mask variational autoencoder is used as the loss function to train the first mask variational autoencoder; in this embodiment, the distance function used for training the first mask variational autoencoder is the Euclidean distance; then the loss function of the first mask variational autoencoder is It can be expressed as:

[0095]

[0096] in, is the first mask variational autoencoder, γ is the mask image, which is the input of the first mask variational autoencoder; Represents the output of the first masked variational autoencoder.

[0097] In some embodiments, the process of obtaining the second feature extraction network includes:

[0098] The second feature extraction network includes a second point cloud feature extraction network, a second image feature extraction network, and a second text feature extraction network;

[0099] For the second point cloud feature extraction network, the preprocessed point cloud data is input into the point cloud neural network for feature extraction. The point cloud neural network used in this embodiment is a pre-trained PointNet++ network. The number of feature vectors obtained is recorded as r. The obtained r feature vectors are used as tokens, and a trainable category representation is added. The r feature vectors and the trainable category representation are input into the encoder of the transformer model. The feature corresponding to the category representation in the output of the last layer of the encoder network is used as the global feature of the point cloud, thereby realizing the feature extraction of the point cloud data. The trained second point cloud feature extraction network can be considered to have a strong ability to extract semantic information from the point cloud. Among them, the category representation is a basic concept of the transformer model and will not be repeated here.

[0100] For the second grayscale feature extraction network and the second text feature extraction network, the pre-trained weights of image encoders such as open-clip and natural language encoders are generally used as the pre-trained weights of the second grayscale feature extraction network and the second text feature extraction network, and the weights of the second grayscale feature extraction network and the second text feature extraction network are fixed, which is equivalent to the second grayscale feature extraction network and the second text feature extraction network being fixed models, and only the second point cloud feature extraction network is trained; it is also possible to train both the second grayscale feature extraction network and the second text feature extraction network while training the second point cloud feature extraction network; wherein the data set required for the training of the second grayscale feature extraction network is a data set composed of grayscale images, and the data set required for the training of the second text feature extraction network is a data set composed of natural language descriptions; thereby, the global features of the point cloud are obtained according to the second point cloud feature extraction network, the global features of the grayscale image are obtained according to the second grayscale feature extraction network, and the global features of the natural language are obtained according to the second text feature extraction network;

[0101] In addition, in order to make the representations of different modal data from the same object close to each other and make the representations of different objects far away from each other, this embodiment constructs positive sample pairs and negative sample pairs through contrastive learning, where the positive sample pairs are the representations of different modal data of the same object, which in this embodiment refers to the global features of the point cloud, the global features of the grayscale image, and the global features of the natural language of the same object; and the negative sample pairs are the representations of different modal data of different objects; the loss function loss of the contrastive learning model is D It can be expressed as:

[0102]

[0103] Wherein, N represents the number of basic units used in training. In this embodiment, the point cloud, grayscale image, and natural language description representing the same object are one basic unit; αi represents the global features of the point cloud corresponding to the i-th basic unit; f βi Indicates the global features of natural language corresponding to the i-th basic unit; represents the global feature of the grayscale image corresponding to the i-th basic unit; f αj represents the global features of the point cloud corresponding to the jth basic unit; f βj Indicates the global features of natural language corresponding to the jth basic unit; f χj represents the global feature of the grayscale image corresponding to the jth basic unit; ι is a learnable parameter; S iu It represents the u-th comparison similarity corresponding to the i-th basic unit; exp() is the exponential function; log() is the logarithmic function; “·” represents the dot product symbol.

[0104] Among them, fαi ·f βi It is used to represent the similarity measure between the positive sample pairs consisting of the global features of the point cloud corresponding to the i-th basic unit and the global features of its corresponding natural language, f αi ·f βj Represents the similarity measure between the global features used to represent the point cloud corresponding to the i-th basic unit and the negative sample pairs composed of the global features of the natural language corresponding to other basic units, and so on; different contrast similarities corresponding to a single basic unit are obtained according to the differences in the global features used to constitute the positive sample pairs and the negative sample pairs, and the loss function of contrast learning is obtained according to all contrast similarities corresponding to all basic units;

[0105] Through contrastive learning, the representation of different global features in the feature space can be made richer and more discriminative. At this time, the representation of these global features not only contains the basic information of the object, but also contains the similarities and differences between objects, thereby improving the performance of the model in subsequent tasks.

[0106] Step S110: obtaining a second diffusion network formed by adding a control branch network to the pre-trained first diffusion network.

[0107] The first diffusion network in this embodiment is a preset diffusion network, which includes two parts: an encoding network and a decoding network. The encoding network includes a plurality of encoding modules, each of which is composed of a downsampling operation, a plurality of convolutions, and a plurality of nonlinear activation functions, wherein the downsampling operation may be a pooling operation with a step size greater than 1, a convolution operation with a step size greater than 1, an interpolation-based resolution enlargement operation, etc.; the decoding network includes a plurality of decoding modules, each of which is composed of an upsampling operation, a plurality of convolutions, and a plurality of nonlinear activation functions, wherein the upsampling operation may be a transposed convolution, an interpolation-based resolution reduction operation, etc.;

[0108] All weight parameters of the pre-trained first diffusion network are "frozen", that is, the weight parameters are not updated, and then a coding network of a control branch network with the same network structure and parameters as the coding network of the first diffusion network is constructed. The obtained network and the corresponding zero-initialized convolutional layer constitute the control branch network; the control branch network is jump-connected to the pre-trained first diffusion network, thereby forming the network structure of the second diffusion network, such as Figure 5 As shown in the figure; this method greatly reduces the number of parameters in the network during training and reduces the difficulty of training.

[0109] Step S120: performing one or more rounds of training on the second diffusion network according to the training set to obtain a second diffusion network.

[0110] In this embodiment, the training set used to train the second diffusion network includes one or more sets of training data, and any set of training data includes an image pair consisting of a set of grayscale images and height images, a mask image, and a point cloud of an object corresponding to the image pair, and natural language data, wherein the mask image can be randomly selected from the mask image data set; then, in the training process of any training round, the second diffusion network is trained according to any set of training data in the training set, such as Figure 4 As shown, including:

[0111] Data processing step S121, in the current training round, the initial first image features are obtained according to the image pairs in the current training data, and the first mask features are obtained according to the mask images in the current training data; that is, the image pairs in the current training data are input into the first image feature extraction network, and the output results of the first image feature extraction network are used as the initial first image features corresponding to the current training data, and the mask images in the current training data are input into the first mask feature extraction network to obtain the first mask features corresponding to the current training data;

[0112] Noise adding step S122, obtaining the first high-dimensional noise ε 1 , where the first high-dimensional noise is a standard high-dimensional noise, each dimension of which obeys the standard Gaussian distribution, that is, ε 1 ~N(0,I), where I represents the identity matrix, and each dimension of which follows a standard Gaussian distribution with a mean of 0 and a variance of 1;

[0113] First, the total number of noise addition steps in the current training round is set, wherein the total number of noise addition steps corresponding to each training round can be randomly set within a preset range, that is, the total number of noise addition steps corresponding to each training round is randomly set within a range of 2 to a preset value, and the preset value can be set according to the needs. In the present embodiment, the preset value set in the training process of the second diffusion network is 30;

[0114] Then, when the current number of noise adding steps is the first number of noise adding steps, the first high-dimensional noise is used to add noise to the initial first image feature, so as to obtain the first image feature after the noise adding operation corresponding to the first number of noise adding steps; when the current number of noise adding steps is not the first number of noise adding steps, the first high-dimensional noise is used to continue adding noise to the first image feature after the noise adding operation corresponding to the previous number of noise adding steps, so as to obtain the first image feature after the noise adding operation corresponding to the current number of noise adding steps;

[0115] For example, assuming that the current number of noise addition steps is t, the first image feature after the noise addition operation corresponding to the current number of noise addition steps is obtained according to the first high-dimensional noise and the first image feature after the noise addition operation corresponding to the previous number of noise addition steps t-1. The specific process is: according to the preset intensity of the first high-dimensional noise corresponding to the first number of noise addition steps and the preset intensity of the first high-dimensional noise corresponding to the last number of noise addition steps, the noise intensity of the first high-dimensional noise corresponding to the t-th number of noise addition steps is obtained; and then the noise intensity of the first high-dimensional noise corresponding to the t-th number of noise addition steps under the square root is used as the first high-dimensional noise corresponding to the t-th number of noise addition steps. The reference weight of the noise; calculate the difference between 1 and the noise intensity of the first high-dimensional noise corresponding to the t-th noise addition step, and use the difference under the square root as the reference weight of the first image feature after the noise addition operation corresponding to the t-1-th noise addition step; multiply the first image feature after the noise addition operation corresponding to the t-1-th noise addition step and the first high-dimensional noise corresponding to the t-th noise addition step by the corresponding reference weights respectively, and use the multiplication result as the first image feature after the noise addition operation corresponding to the t-th noise addition step, then the first image feature after the noise addition operation corresponding to the t-th noise addition step can be expressed as:

[0116]

[0117] Among them, x t Indicates that the current number of noise addition steps corresponds to the first image feature after the noise addition operation, that is, the tth number of noise addition steps corresponds to the first image feature after the noise addition operation; x t-1 represents the first image feature after the previous noise addition step number corresponds to the noise addition operation, that is, the first image feature after the t-1th noise addition step number corresponds to the noise addition operation; ε 1 represents the first high-dimensional noise; β t is the noise intensity of the first high-dimensional noise when the current noise addition step number t corresponds to the noise addition operation;

[0118] The noise intensity of the first high-dimensional noise in the noise addition operation corresponding to the current t-th noise addition step is obtained according to the preset intensity of the first high-dimensional noise in the first noise addition and the preset intensity of the last noise addition. The specific process is: calculate the square difference between the preset intensity of the first high-dimensional noise corresponding to the first noise addition step and the preset intensity of the first high-dimensional noise corresponding to the last noise addition step, and obtain the difference between the preset total number of noise addition steps in the current training round and 1; use the ratio between the obtained square difference and the obtained difference as the noise intensity change of a single noise addition; multiply the difference between the current number of noise addition steps and 1 by the noise intensity change of a single noise addition, and use the obtained product as the cumulative change of the noise intensity corresponding to the current number of noise addition steps; add the cumulative change of the noise intensity corresponding to the current number of noise addition steps and the square of the preset intensity of the first high-dimensional noise corresponding to the first noise addition step, and then use the sum obtained under the square root as the noise intensity of the first high-dimensional noise corresponding to the t-th noise addition step, then the noise intensity of the first high-dimensional noise corresponding to the t-th noise addition step can be expressed as:

[0119]

[0120] Among them, β t is the noise intensity of the first high-dimensional noise when the current t-th noise adding step corresponds to the noise adding operation; β 1 β is the preset strength of the first high-dimensional noise added when the first noise addition step corresponds to the noise addition operation, that is, the preset strength of the first high-dimensional noise when the noise is added for the first time; T is the preset strength of the first high-dimensional noise added in the noise addition operation corresponding to the Tth noise addition step, that is, the preset strength of the first high-dimensional noise in the last noise addition; T is the total number of noise addition steps in the current training round; Indicates the change in noise intensity after a single noise addition; Indicates the cumulative change in noise intensity corresponding to the current number of noise addition steps;

[0121] In a fusion processing step S123, the first image feature and the first mask feature after the denoising operation corresponding to the current denoising step number are input into the second diffusion network, and in the second diffusion network, the first image feature and the first mask feature after the denoising operation corresponding to the current denoising step number are fused respectively;

[0122] In some embodiments, the fusion process includes a first fusion process and a second fusion process, and the specific process includes:

[0123] First, the current noise step vector is obtained according to the current noise step number, wherein the method for obtaining the current noise step vector is as follows: for the kth element in the current noise step vector, the logarithmic function value corresponding to the current noise step is obtained; for the kth element in the noise step vector, the ratio between k-1 and the preset length of the noise step vector is calculated; the product between the obtained logarithmic function value and the obtained ratio is calculated, and the exponential function value corresponding to the inverse of the obtained product is used as the kth element value in the current noise step vector. Then, the kth element in the current noise step vector can be expressed as:

[0124]

[0125] Among them, τ k Represents the kth element value in the current noise step vector; t is the current noise step; L is the preset length of the noise step vector, which is also the number of elements in the noise step vector, and this value is equal to the dimension of the first image feature; log() is a logarithmic function, and e is a natural constant, which is also the base of the exponential function.

[0126] Then, for any network module in the second diffusion network, the input content of the network module is first fused with the current noise step vector to obtain the first fusion result of the network module; that is, the current noise step vector is copied and expanded into a feature map of the same size as the input content of the network module, and the result of adding each element in the obtained feature map to the element at the corresponding position in the input content of the network module is used as the element value at the corresponding position in the first fusion result of the network module; for example: assuming that the current noise step vector is a 1×1×L vector, the input of the network module The size of the input content is H×W×C, where L=C, and the current noise step vector is copied and expanded in the H dimension and the W dimension, that is, the current noise step vector is copied H×W times to obtain a H×W×C feature map, and the obtained feature map is added to the elements at the corresponding positions of the input content of the network module, and the obtained addition result is used as the element value at the corresponding position in the first fusion result of the network module, and each element in the first fusion result of the network module is processed to obtain the first fusion result of the network module; the first fusion processing of the input content of the network module is completed;

[0127] Then, the first fusion result of the network module is subjected to a second fusion process with the attention parameters corresponding to the current training data and the second mask feature to obtain the second fusion result of the network module, and the output content of the network module is obtained according to the second fusion result of the network module; since each encoding module in the diffusion network is composed of one downsampling operation, several convolutions and several nonlinear activation functions, and each decoding module is composed of one upsampling operation, several convolutions and several nonlinear activation functions, the corresponding network module will also perform corresponding upsampling or downsampling operations, convolution operations, and calculations of nonlinear activation functions on the second fusion result to obtain the output content of the corresponding network module, which will not be repeated here.

[0128] The second mask feature is obtained by scaling the mask image through a scaling image function. The interpolation method of the scaling image function used in this embodiment is nearest neighbor interpolation, and the scaling factor is 0.25, that is, the length and width of the scaled mask image are both 0.25 times of the original.

[0129] In some embodiments, a method for obtaining an attention parameter corresponding to current training data includes: obtaining a fourth image feature corresponding to the current training data according to point cloud data or natural language data or grayscale image in the current training data, that is, inputting the point cloud data or natural language data or grayscale image in the current training data into the corresponding second feature extraction network, that is, inputting the point cloud data in the current training data into the second point cloud feature extraction network, or inputting the natural language data in the current training data into the second text feature extraction network, or inputting the grayscale image in the current training data into the second grayscale feature extraction network, and the global features of the point cloud or the grayscale image or the natural language obtained thereby are collectively referred to as fourth image features; then obtaining a first general feature, which is a learnable parameter having the same length as the fourth image feature; and using any one of the obtained first general feature and the fourth image feature corresponding to the current training data as the attention parameter corresponding to the current training data;

[0130] It should be noted that since the first image feature obtained by the first image feature extraction network may not be complete, the combination of additional information, namely the fourth image feature, can ensure the integrity of the information and enhance the diversity of the information. For example, grayscale information can reflect texture information and guide height information reconstruction. The first general feature is a feature structure with strong feature extraction capabilities. Using it as an attention parameter can obtain more stable and accurate noise prediction results.

[0131] Then, the process of performing a second fusion processing on the first fusion result of the network module, the attention parameter corresponding to the current training data, and the second mask feature includes: firstly multiplying the attention parameter by the first fusion result of the network module to obtain an attention scoring result. For example, assuming that the attention parameter is a 1×1×C vector, the size of the first fusion result of the network module is H×W×C, and the attention scoring result obtained after attention scoring is H×W×1; multiplying the attention parameter by the attention scoring result to obtain an attention convergence result, and the attention convergence result obtained at this time is H×W×C; multiplying the second mask feature by the attention convergence result to obtain a mask operation result, and the size of the mask operation result obtained at this time is still H×W×C; adding the mask operation result to the elements at the corresponding positions of the first fusion result of the network module, thereby obtaining the second fusion result of the network module; thus, the second fusion processing of the first fusion result of the network module is completed, and the feature fusion within a single network module is completed;

[0132] In addition, since the first diffusion network and the control branch network are jump-connected in this embodiment, the input contents of different network modules in the second diffusion network are also different, such as Figure 5 As shown, for any network module in the second diffusion network, if the network module is the first encoding module in the first diffusion network, the corresponding Figure 5 In the "Diffusion Encoding Module_1", the input content of the network module is the first image feature after the current number of noise addition steps corresponding to the noise addition operation; if the network module is the encoding module in the first diffusion network and is not the first encoding module, for example Figure 5 In the diffusion coding module 3, or if the network module is an intermediate module in the first diffusion network, the corresponding Figure 5 In the diffusion intermediate module 4, the input content of the network module is the output content of the previous coding module in the first diffusion network, that is, the input content of the diffusion coding module 3 is the output content corresponding to the diffusion coding module 2; the input content of the diffusion intermediate module 4 is the output content corresponding to the diffusion coding module 4;

[0133] If the network module is the first encoding module in the control branch network, the corresponding Figure 5 In the “control coding module_1”, the input content of this network module is the first mask feature. It should be noted that since there is a 0-initialized convolution layer before the first coding module of the control branch network, the input content of this network module actually refers to the first mask feature after the convolution operation with the 0-initialized convolution layer;

[0134] If the network module is a coding module in the control branch network and is not the first coding module, for example Figure 5"Control Encoding Module_3" in the control branch network, or if the network module is an intermediate module in the control branch network, the corresponding Figure 5 In the “Control Intermediate Module_4”, the input content of this network module is the output content of the previous coding module in the control branch network; that is, the input content of “Control Coding Module_3” is the output content corresponding to “Control Coding Module_2”; the input content of “Control Intermediate Module_4” is the output content corresponding to “Control Coding Module_4”;

[0135] If the network module is the first decoding module in the first diffusion network, the corresponding Figure 5 In the "diffusion decoding module_4", the previous decoding module of the network module refers to the middle module of the first diffusion network; if the network module is the decoding module of the first diffusion network and is not the first decoding module, Figure 5 Taking the "diffusion decoding module_3" in the example, the previous decoding module of the network module in the first diffusion network is obtained, that is, Figure 5 "Diffusion decoding module_4" in the control branch network; convolve the output content of the encoding module with the same size as the previous decoding module "Diffusion decoding module_4" in the control branch network with a zero-initialized convolution layer to obtain a branch fusion result; then fuse the obtained branch fusion result, the output content of the previous decoding module and the output content of the encoding module with the same size as the previous decoding module in the first diffusion network, and use the obtained fusion result as the input content of the network module;

[0136] The specific process of obtaining the input content of "diffusion decoding module_3" is as follows: first, the output content of "control coding module_3" is convolved with a zero-initialized convolutional layer to obtain a branch fusion result, and then the obtained branch fusion result is fused with the output content of "diffusion decoding module_4" and the output content of "diffusion coding module_4", and the fusion result obtained is used as the input content of "diffusion decoding module_3", and so on;

[0137] It should be noted that when the network module is the first decoding module "diffusion decoding module_4" in the first diffusion network, the previous decoding module of the network module refers to the intermediate module "diffusion intermediate module_4" of the first diffusion network. Since the intermediate module does not have a corresponding encoding module, the first decoding module in the first diffusion network does not need the "output content of the encoding module with the same size as the previous decoding module in the first diffusion network" mentioned above to participate in the fusion in the process of obtaining its input content. That is, for the first decoding module in the first diffusion network, the process of obtaining the input content of the network module is: first, the output content of the "control intermediate module_4" is convolved with a zero-initialized convolution layer to obtain a branch fusion result, and then the obtained branch fusion result is fused with the output content of the "diffusion intermediate module_4", and the fusion result obtained is used as the input content of the "diffusion decoding module_4";

[0138] The second diffusion network is used to perform noise prediction on the input of the second diffusion network to output high-dimensional noise when the current number of noise addition steps is equal to the preset total number of noise addition steps of the current training round;

[0139] It should be noted that, during the training process of the second diffusion network, the attention parameter used by the first diffusion network is the fourth image feature, and the attention parameter in the control branch network can be the fourth image feature or the first general feature, which can be set according to actual conditions.

[0140] Training judgment step S124, if the current number of noise adding steps is less than the preset total number of noise adding steps of the current training round, then the current number of noise adding steps is increased by 1 step to update the current number of noise adding steps, and the training is continued from the noise adding step S122; if the current number of noise adding steps is equal to the preset total number of noise adding steps of the current training round, then the high-dimensional noise output by the second diffusion network is used as the second high-dimensional noise; the loss function of the second diffusion network is obtained according to the first high-dimensional noise and the second high-dimensional noise, and back propagation is performed according to the loss function to update the weight parameters in the second diffusion network. At this time, since the weight parameters in the first diffusion network are fixed, the weight parameters updated during the training process include the weight parameters in the control branch network and some learnable parameters, such as the first general feature;

[0141] Among them, the loss function of the second diffusion network can use a loss function such as cross entropy and mean square error to represent the degree of distribution similarity. The loss function used in this embodiment is a cross entropy loss function, and its acquisition process is: for any element value of the second high-dimensional noise, calculate the logarithmic function value of the element value, multiply the obtained logarithmic function value with the element value of the first high-dimensional noise at the corresponding position, and obtain the multiplication result corresponding to the element value; obtain the cumulative sum of the corresponding multiplication results of all element values ​​of the second high-dimensional noise, and use the inverse of the obtained cumulative sum as the loss function of the second diffusion network, then the loss function of the second diffusion network can be expressed as:

[0142]

[0143] Among them, ε 1a represents the ath element value of the first high-dimensional noise; ε 2a is the ath element value of the second high-dimensional noise; m is the total number of elements contained in the first high-dimensional noise and the second high-dimensional noise;

[0144] It should be noted that the initial first image feature is obtained after the first image feature extraction network extracts features from the input data. Therefore, each pixel in the first image feature corresponds to a high-dimensional feature vector, and the first high-dimensional noise will apply a noise disturbance value to the element at each position in the first image feature, and the corresponding noise disturbance values ​​at different positions obey the standard Gaussian distribution;

[0145] If the loss function converges, the training is stopped, thereby obtaining a trained second diffusion network; if the loss function does not converge, the training is continued from the data processing step S121 according to the next set of training data in the training set; if the current training data is the last set of training data in the training set, the training of the current training round is stopped, and after the training of the current training round is stopped, the second diffusion network after the end of the current training round is used as the second diffusion network used in the training process of the next training round, and then the training of the next training round is started from the data processing step S121 according to the first set of training data in the training set; in addition, before stopping the training of the current training round and starting the training of the next training round, it is also determined whether the preset maximum training round is reached. If the maximum training round is reached, the training is stopped and the obtained second diffusion network is adjusted. For example, when the second diffusion network cannot converge, it may be because the network parameters are set unreasonably. At this time, the network parameters of the second diffusion network need to be adjusted, and then the adjusted second diffusion network is re-trained for multiple rounds according to the training set, wherein the maximum training round can be set according to actual needs. In this embodiment, it can be set to 10,000 times.

[0146] In this embodiment, the grayscale image and the height map are first channel-superimposed to extract the first image feature, the first mask feature and the second mask feature are extracted according to the mask image, and different types of global features of the object, namely, the fourth image feature, are extracted according to the point cloud, grayscale image and natural language of the object. Then, through contrast learning, the global feature is made to contain not only the basic information of the object itself, but also the relevant information between the objects. Finally, the second diffusion network composed of the first diffusion network and the control branch network is trained according to the first image feature, the first mask feature, the second mask feature and the fourth image feature or the first general feature. By fixing the parameters inside the first diffusion network, only the parameters in the control branch network and some learnable parameters are adjusted. The learning parameters are updated, thereby reducing the number of parameters inside the network during training and reducing the difficulty of training, thereby achieving precise fine-tuning without changing the original diffusion model; at the same time, the control branch network provides additional information for the first diffusion network, thereby improving the phenomenon that the first diffusion network is difficult to achieve pixel-level image generation; and because it is fused with attention parameters and the like in different network modules of the second diffusion network, by setting the attention parameters, the obtained second diffusion network can complete conditional (i.e., when additional information such as point cloud is involved) or unconditional (i.e., when additional information such as point cloud is not involved) noise prediction tasks, so as to subsequently denoise the noisy image and generate high-quality images.

[0147] Please refer to Figure 2 In some embodiments, a noise prediction method is disclosed, which includes the following steps:

[0148] Step S200: Acquire a second diffusion network trained by a training method.

[0149] In some embodiments, the training method mentioned in step S200 may refer to the training method for noise prediction mentioned above, such as the second diffusion network obtained through steps S100 to S120.

[0150] Step S210: Acquire a first mask feature and a first image feature.

[0151] In some embodiments, the first mask feature may be obtained by inputting a mask image into a first mask feature extraction network. The mask image may be set according to actual needs or may be randomly selected from a mask image data set.

[0152] In some embodiments, the first image feature may be obtained by inputting an image pair consisting of a grayscale image and a height image into a first image feature extraction network. The image pair may be set according to actual needs or may be randomly selected from a data set consisting of training image pairs.

[0153] Step S220: gradually adding noise to the obtained first image feature, and when the number of steps of adding noise reaches a preset total number of steps of adding noise, the first image feature after adding noise is obtained.

[0154] The first image feature is gradually denoised according to a random high-dimensional noise, wherein the random high-dimensional noise is a standard high-dimensional noise, and a preset total number of denoising steps is set, for example, the preset total number of denoising steps is set to 20, and the value can be set as needed; when the number of denoising steps reaches the preset total number of denoising steps, the denoising is stopped, indicating that the denoising process of the first image feature is completed, and the first image feature after the denoising is completed is obtained. Exemplarily, the denoising process of gradually denoising the first image feature according to a random high-dimensional noise is:

[0155] When the current number of noise adding steps is the first number of noise adding steps, random high-dimensional noise is used to add noise to the first image feature, so as to obtain the first image feature after the noise adding operation corresponding to the first number of noise adding steps; when the current number of noise adding steps is not the first number of noise adding steps, random high-dimensional noise is used to continue adding noise to the first image feature after the noise adding operation corresponding to the previous number of noise adding steps, so as to obtain the first image feature after the noise adding operation corresponding to the current number of noise adding steps;

[0156] For example, assuming that the current number of noisy steps is t, the first image feature after the noisy operation corresponding to the current number of noisy steps is obtained according to the random high-dimensional noise and the first image feature after the noisy operation corresponding to the previous number of noisy steps t-1. The specific process is: according to the preset intensity of the random high-dimensional noise corresponding to the first number of noisy steps and the preset intensity of the random high-dimensional noise corresponding to the last number of noisy steps, the noise intensity of the random high-dimensional noise corresponding to the t-th number of noisy steps is obtained; and then the noise intensity of the random high-dimensional noise corresponding to the t-th number of noisy steps under the quadratic root is used as the random high-dimensional noise corresponding to the t-th number of noisy steps. The reference weight of the noise; calculate the difference between 1 and the noise intensity of the random high-dimensional noise corresponding to the t-th noisy step, and use the difference under the square root as the reference weight of the first image feature after the noisy operation corresponding to the t-1-th noisy step; multiply the first image feature after the noisy operation corresponding to the t-1-th noisy step and the random high-dimensional noise corresponding to the t-th noisy step with the corresponding reference weights respectively, and use the multiplication result as the first image feature after the noisy operation corresponding to the t-th noisy step, then the first image feature after the noisy operation corresponding to the t-th noisy step can be expressed as:

[0157]

[0158] Among them, y t Indicates that the current number of noise addition steps corresponds to the first image feature after the noise addition operation, that is, the tth number of noise addition steps corresponds to the first image feature after the noise addition operation; y t-1represents the first image feature after the previous noise addition step number corresponds to the noise addition operation, that is, the first image feature after the t-1th noise addition step number corresponds to the noise addition operation; ε 0 represents random high-dimensional noise; ω t The noise intensity of the random high-dimensional noise during the noise addition operation corresponding to the current noise addition step number t;

[0159] The noise intensity of the random high-dimensional noise during the noise addition operation corresponding to the current t-th noise addition step is obtained according to the preset intensity of the random high-dimensional noise during the first noise addition and the preset intensity of the last noise addition. The specific process is: calculate the square difference between the preset intensity of the random high-dimensional noise corresponding to the first noise addition step and the preset intensity of the random high-dimensional noise corresponding to the last noise addition step, and obtain the difference between the preset total number of noise addition steps in the current training round and 1; take the ratio between the obtained square difference and the obtained difference as the noise intensity change of a single noise addition; multiply the difference between the current number of noise addition steps and 1 by the noise intensity change of a single noise addition, and take the obtained product as the cumulative change of the noise intensity corresponding to the current number of noise addition steps; add the cumulative change of the noise intensity corresponding to the current number of noise addition steps and the square of the preset intensity of the random high-dimensional noise corresponding to the first noise addition step, and then take the sum obtained under the square root as the noise intensity of the random high-dimensional noise corresponding to the t-th noise addition step, then the noise intensity of the random high-dimensional noise corresponding to the t-th noise addition step can be expressed as:

[0160]

[0161] Among them, ω t is the noise intensity of the random high-dimensional noise corresponding to the current t-th noise addition step during the noise addition operation; ω 1 is the preset intensity of the random high-dimensional noise added when the first noise addition step corresponds to the noise addition operation, that is, the preset intensity of the random high-dimensional noise when the noise is added for the first time; ω Q is the preset intensity of the random high-dimensional noise added during the noise addition operation corresponding to the Qth noise addition step, that is, the preset intensity of the random high-dimensional noise during the last noise addition; Q is the preset total number of noise addition steps.

[0162] Step S230: inputting the first image feature after noise addition and the first mask feature into the second diffusion network, and performing fusion processing on the first mask feature and the first image feature after noise addition in the second diffusion network.

[0163] In some embodiments, the fusion process includes a first fusion process and a second fusion process, wherein:

[0164] First, the current denoising step number vector is obtained according to the current denoising step number. Since the first image feature after denoising at this time is the first image feature after the denoising step number reaches the preset total denoising step number, the denoising step number at this time is equal to the preset total denoising step number. In addition, if the second diffusion network is used to perform a denoising operation on the first image feature after denoising, the corresponding denoising step number may be changed accordingly according to the change of the denoising step number, thereby changing the current denoising step number vector.

[0165] Then, for any network module in the second diffusion network, the input content of the network module is first fused with the current noise step vector to obtain a first fusion result of the network module; the first fusion result of the network module is then fused with the corresponding attention parameter and the second mask feature to obtain a second fusion result of the network module, and the second fusion result of the network module is used as the output content of the network module;

[0166] In some embodiments, the second mask feature is obtained by scaling the mask image using a scaling image function;

[0167] In some embodiments, the fourth image feature is obtained based on the point cloud data or natural language data or grayscale image of the object corresponding to the image pair used when the first image feature is obtained, that is, the obtained point cloud data or natural language data or grayscale image is input into the corresponding second feature extraction network, and the global feature output by the corresponding second feature extraction network is used as the fourth image feature;

[0168] Obtaining a first general feature, illustratively, the first general feature may be obtained in the second diffusion network training process; using any one of the obtained first general feature and the fourth image feature as an attention parameter; performing a second fusion process on the first fusion result of the network module, the attention parameter and the second mask feature, and then obtaining the output content of the single network module, thereby completing the feature fusion within the single network module;

[0169] In addition, due to the jump connection between the first diffusion network and the control branch network, the output contents of different network modules in the second diffusion network will also be merged, which will not be described here;

[0170] It should be noted that since the noise step vector, attention parameter, and second mask feature are used when each network module performs fusion processing, from an overall perspective, the input content of the second diffusion network includes the first mask feature, the first image feature after noise addition, the noise step vector, attention parameter, and the second mask feature.

[0171] Step S240: using the output of the second diffusion network as predicted high-dimensional noise.

[0172] In this embodiment, the first mask feature and the first image feature are first obtained; the obtained first image feature is gradually denoised to obtain the first image feature after denoising; the first image feature after denoising and the first mask feature are then input into the second diffusion network, and in the second diffusion network, the first mask feature and the first image feature after denoising are respectively fused; finally, the output of the second diffusion network is used as the predicted high-dimensional noise; in this process, the second fusion processing is performed with the attention parameters in different network modules, and by setting the attention parameters, the noise prediction results under conditions (i.e., the case where additional information such as point cloud is involved) or unconditional (i.e., the case where additional information such as point cloud is not involved) can be obtained.

[0173] Please refer to Figure 3 In some embodiments, a defect image generation method is disclosed, which includes the following steps:

[0174] Step S300: Acquire a second diffusion network trained by a training method.

[0175] In some embodiments, the training method mentioned in step S300 may refer to the training method for noise prediction mentioned above, such as the second diffusion network obtained through steps S100 to S120.

[0176] Step S310: obtaining a first mask feature and a second mask feature according to the mask image.

[0177] In some embodiments, a mask image is obtained, and the mask image can be set according to actual needs or randomly selected from a mask image data set;

[0178] The mask image is input into a first mask feature extraction network to obtain a first mask feature; and the mask image is scaled by a scaling image function to obtain a second mask feature.

[0179] Step S320: obtaining a new first image feature according to the second mask feature.

[0180] First, a comprehensive feature vector is obtained, wherein the comprehensive feature vector is obtained in a training process of a second diffusion network obtained by a training method;

[0181] In some embodiments, the training method mentioned in step S320 may refer to the training method for noise prediction mentioned above, such as the second diffusion network obtained through steps S100 to S120.

[0182] In some embodiments, in any training round of the second diffusion network, the first feature vector is further iteratively updated in the data processing step of each set of training data, wherein in any training round, for the current training data, the background area and foreground area of ​​the initial first image feature are obtained in the data processing step according to the initial first image feature corresponding to the current training data and the mask image in the current training data;

[0183] In some embodiments, if the corresponding pixel value of a pixel point in the mask image in the current training data is 1, it is considered that the corresponding pixel point in the first image feature belongs to the foreground area; if the corresponding pixel value of a pixel point in the mask image in the current training data is 0, it is considered that the corresponding pixel point in the first image feature belongs to the background area;

[0184] Obtaining a first feature vector corresponding to current training data according to the background area and the foreground area of ​​the initial first image feature;

[0185] In some embodiments, the first feature vector corresponding to the current training data includes four vectors, namely, the background mean vector, the background standard deviation vector, the foreground mean vector and the foreground standard deviation vector of the initial first image feature; wherein, for the initial first image feature corresponding to the current training data, its background mean vector is the mean vector of the high-dimensional feature vector corresponding to all pixels in the background area of ​​the initial first image feature; its background standard deviation vector is the standard deviation vector of the high-dimensional feature vector corresponding to all pixels in the background area of ​​the initial first image feature; its foreground mean vector is the mean vector of the high-dimensional feature vector corresponding to all pixels in the foreground area of ​​the initial first image feature; its foreground standard deviation vector is the standard deviation vector of the high-dimensional feature vector corresponding to all pixels in the foreground area of ​​the initial first image feature;

[0186] According to the first eigenvector corresponding to the current training data and the first eigenvector updated in the last iteration, the first eigenvector updated in the current iteration is obtained; wherein, if the current training data is the first set of training data in the training set, the first eigenvector updated in the last iteration is a 0 vector;

[0187] In the g-th iterative update, for any first eigenvector, firstly, a momentum coefficient is preset, and the momentum coefficient is used as the reference weight of any first eigenvector corresponding to the g-1-th iterative update, and the difference between 1 and the momentum coefficient is used as the reference weight of the first eigenvector corresponding to the g-th group of training data; the product of any first eigenvector obtained after the g-1-th iterative update, the first eigenvector corresponding to the g-th group of training data and the corresponding reference weight is calculated respectively, and the sum of the two products is used as any first eigenvector after the g-th iterative update;

[0188] Taking the background mean vector as an example, let the momentum coefficient ρ be used as the reference weight of the background mean vector after the g-1th iteration update, and the difference between 1 and the momentum coefficient, that is, 1-ρ, as the reference weight of the background mean vector corresponding to the g-th group of training data; calculate the product of the background mean vector obtained after the g-1th iteration update, the background mean vector corresponding to the g-th group of training data and the corresponding reference weight respectively, and take the sum of the two products as the background mean vector after the g-th iteration update; then the background mean vector after the g-th iteration update can be expressed as:

[0189] μ g =ρ×μ g-1 +(1-ρ)×c g

[0190] Among them, μ g represents the background mean vector after the g-th iteration update; μ g-1 represents the background mean vector after the g-1th iteration update; ρ is the momentum coefficient; c g Represents the background mean vector corresponding to the g-th group of training data;

[0191] The same processing is performed on the other three first eigenvectors, that is, the processing method for the background standard deviation vector, the foreground mean vector and the foreground standard deviation vector is the same as the processing method for the background mean vector.

[0192] The first feature vector after the last iteration update in a training round of the second diffusion network is used as the comprehensive feature vector.

[0193] It should be noted that in any training round, all training data in the entire training set need to be processed in sequence, and each time a group of training data is processed, the corresponding first eigenvector will be obtained, and then the first eigenvector will be updated. In other words, each update of the comprehensive feature vector in the acquisition process is actually only related to the training data. Therefore, the comprehensive feature vectors obtained in different training rounds are the same.

[0194] Then, for any pixel point in the new first image feature, when the pixel value of the pixel point at the corresponding position in the second mask feature is 1, the foreground standard deviation vector in the comprehensive feature vector is multiplied by the element value of the corresponding dimension of the standard high-dimensional noise, and the sum of the multiplied vector and the foreground mean vector is used as the high-dimensional feature vector corresponding to the pixel point in the new first image feature; when the pixel value of the pixel point at the corresponding position in the second mask feature is 0, the high-dimensional feature vector corresponding to any pixel point in the new first image feature is obtained according to the comprehensive feature vector or the reference image feature;

[0195] Among them, when the pixel value of the pixel point at the corresponding position in the second mask feature is 0, the background standard deviation vector in the comprehensive feature vector is multiplied by the element value on the corresponding dimension of the standard high-dimensional noise, and the sum of the multiplied vector and the background mean vector is used as the high-dimensional feature vector corresponding to any pixel point in the new first image feature; or, a preset image pair is obtained, which can be set by the user; then a reference image feature is obtained based on the preset image pair, that is, the preset image pair is input into the first image feature extraction network, and the output result of the first image feature extraction network is called the reference image feature; then the high-dimensional feature vector corresponding to the corresponding pixel point in the reference image feature is used as the high-dimensional feature vector corresponding to any pixel point in the new first image feature.

[0196] When the mask image and the first eigenvector are directly used to generate a new first image feature, the defect position can be made to appear at different positions by selecting different mask images, thereby greatly enriching the diversity of the defect sample data; when the new first image feature is generated according to the preset image pair, the mask image and the first eigenvector, the final defect sample can have the background image actually required. At this time, by selecting different mask images, so that the second mask feature is also changed, the defect position can also be changed. While enriching the defect sample data, it can also better meet the user's usage needs, thereby migrating different defect areas to any background image and generating corresponding annotations, thereby synthesizing images and annotations used for supervised learning, greatly simplifying the process of collecting and annotating images, and making deep learning projects easier to implement.

[0197] Step S330: gradually add noise to the new first image feature according to a preset total number of noise adding steps to obtain a third image feature.

[0198] In some embodiments, the step S330 involved in gradually adding noise to the new first image feature according to the preset total number of noise adding steps can refer to the process of obtaining the first image feature after noise addition in the above step S220, and the obtained first image feature after noise addition is called the third image feature. In this embodiment, the preset total number of noise adding steps is still recorded as T.

[0199] Step S340: performing a denoising operation on the third image feature according to the first mask feature and the second diffusion network to obtain a first denoising feature map.

[0200] First, the first mask feature is input into the control branch network, the third image feature is input into the first diffusion network, and the input third image feature and the first mask feature are fused in the second diffusion network. In this process, the high-dimensional noise used for denoising the third image feature is obtained by changing the attention parameter used by each network module in the second diffusion network in the second fusion process, and the third image feature is denoised according to the obtained high-dimensional noise, and then the first mask feature and the denoised third image feature are re-input into the second diffusion network, and so on, so as to complete the step-by-step denoising of the third image feature until the denoising step number is equal to the preset total denoising step number in step S330; wherein, the denoising operation corresponding to the first denoising step number is to process the third image feature input into the first diffusion network, that is, the third image feature obtained in step S330; the denoising operation corresponding to other denoising step numbers is to process the third image feature after the last denoising operation;

[0201] Taking the denoising operation corresponding to the t'th denoising step as an example, first obtain the denoising step vector corresponding to each denoising step, and the denoising step vector corresponding to each denoising step is obtained by first obtaining the denoising step corresponding to the denoising step, and then obtaining the denoising step vector according to the obtained denoising step;

[0202] In some embodiments, the number of denoising steps corresponding to the t'th denoising operation is equal to the total number of denoising steps minus t' plus 1, that is, the number of denoising steps q corresponding to the t'th denoising operation is T-t'+1; the corresponding denoising step number vector is obtained according to the number of denoising steps corresponding to the t'th denoising operation;

[0203] The first mask feature and the third image feature after the last denoising operation are input into the second diffusion network, and the third image feature and the first mask feature after the last denoising operation are fused in the second diffusion network respectively; in the denoising process, for any network module in the second diffusion network, the input content of the network module is first fused with the denoising step vector corresponding to the t'th denoising operation to obtain the first fusion result of the network module; the first fusion result of the network module is then fused with the attention parameter and the second mask feature for the second time to obtain the second fusion result of the network module; wherein the third image feature after the last denoising operation is the input content of the first encoding module of the first diffusion network in the second diffusion network, and the first mask feature is the input content of the first encoding module of the control branch network in the second diffusion network, wherein the input content of the first encoding module of the control branch network here also refers to the first mask feature after the convolution operation with the 0-initialized convolution layer.

[0204] Then, the third high-dimensional noise and the fourth high-dimensional noise corresponding to the t'th denoising operation are obtained according to the output of the second diffusion network, and the fourth image feature and the first general feature are first obtained, wherein the fourth image feature can be obtained according to the point cloud data or natural language data or grayscale image of the object corresponding to the image used when obtaining the first image feature, that is, the obtained point cloud data or natural language data or grayscale image is input into the corresponding second feature extraction network, and the global feature output by the corresponding second feature extraction network is used as the fourth image feature; the first general feature can be obtained in the training process of the second diffusion network;

[0205] For the t'th denoising operation, the attention parameter used by any network module in the second diffusion network for the second fusion processing is set to the fourth image feature, the third image feature after the last denoising operation and the first mask feature are input into the second diffusion network, and the high-dimensional noise predicted by the second diffusion network is recorded as the third high-dimensional noise corresponding to the t'th denoising operation; then the attention parameter used by any network module in the second diffusion network for the second fusion processing is set to the first general feature, and the third image feature after the last denoising operation and the first mask feature are input into the second diffusion network again, and the high-dimensional noise predicted by the second diffusion network is recorded as the fourth high-dimensional noise corresponding to the t'th denoising operation.

[0206] Obtaining a fifth high-dimensional noise corresponding to the t'th denoising operation according to the third high-dimensional noise and the fourth high-dimensional noise corresponding to the t'th denoising operation, wherein the difference between the element values ​​at the same position of the third high-dimensional noise and the fourth high-dimensional noise is used as the element value of the fifth high-dimensional noise at the corresponding position;

[0207] Then, the third image feature after the previous denoising operation is denoised according to the fifth high-dimensional noise corresponding to the t'th denoising operation, to obtain the third image feature after the t'th denoising operation; wherein the denoising operation can be regarded as solving a stochastic differential equation, and a variety of stochastic differential equation solving methods can be used, such as the Euler sampling method, the Heun sampling method, etc., and this embodiment uses the Euler sampling method for denoising, and the method for obtaining the third image feature after the t'th denoising operation is:

[0208]

[0209] Wherein, q is the number of denoising steps corresponding to the t'th denoising operation, i.e., q=T-t'+1, which means that q and t' are in one-to-one correspondence; F represents the third image feature, F(q) represents the third image feature after the last denoising operation, i.e., the third image feature after the t'-1th denoising operation; F(q-1) represents the third image feature after the t'th denoising operation; a q is the denoising intensity of the t'th denoising operation; ε5 (q) represents the fifth high-dimensional noise corresponding to the t'th denoising operation; ε(q) is the standard high-dimensional noise corresponding to the t'th denoising operation, that is, ε(q)~N(0,I), I represents the unit matrix, each dimension of which obeys the standard Gaussian distribution with a mean of 0 and a variance of 1; p is a preset value used to represent the randomness of the stochastic differential equation solution process. The value can be set according to actual needs. In this embodiment, it is set to 1; is the multiplication result of the denoising intensity from the 0th denoising to the current t'th denoising operation, that is,

[0210] When t'=T, that is, the total number of denoising operations is equal to the preset total number of denoising steps in step S330, denoising is stopped after the current denoising operation is completed, and the third image feature obtained after the last denoising operation is completed is recorded as the first denoised image feature.

[0211] Step S350: Obtain a target image according to the first denoising feature map.

[0212] In some embodiments, the first denoising feature map is input into a first feature decoding network, and the output content of the first feature decoding network is the target image.

[0213] In this embodiment, first, according to actual needs, it is selected whether to require the defect sample data to have a specified background area to generate a new first image feature, so that while enriching the defect sample data, it can better meet user needs and make the final target image more diverse in form; in addition, since the second diffusion network used in the target image acquisition process can input additional information such as point clouds, and the mask image contains rich location information, it can enhance the diversity of information while ensuring the integrity of the information, thereby enhancing the generation effect of the target image, and then achieving the generation of high-quality target images under pixel-level precision control.

[0214] Those skilled in the art will appreciate that all or part of the functions of the various methods in the above-mentioned embodiments can be implemented by hardware or by computer programs. When all or part of the functions in the above-mentioned embodiments are implemented by computer programs, the program can be stored in a computer-readable storage medium, and the storage medium can include: read-only memory, random access memory, disk, optical disk, hard disk, etc., and the program is executed by a computer to implement the above-mentioned functions. For example, the program is stored in the memory of the device, and when the program in the memory is executed by the processor, all or part of the above-mentioned functions can be implemented. In addition, when all or part of the functions in the above-mentioned embodiments are implemented by computer programs, the program can also be stored in a storage medium such as a server, another computer, disk, optical disk, flash disk or mobile hard disk, and can be downloaded or copied and saved in the memory of the local device, or the system of the local device is updated, and when the program in the memory is executed by the processor, all or part of the functions in the above-mentioned embodiments can be implemented.

[0215] The above specific examples are used to illustrate the present invention, which is only used to help understand the present invention and is not intended to limit the present invention. For those skilled in the art, according to the concept of the present invention, some simple deductions, modifications or substitutions can be made.

Claims

1. A training method for noise prediction, characterized in that: include: Acquire a second diffusion network formed by adding a control branch network to a pre-trained first diffusion network, wherein the first diffusion network is a preset diffusion network; Performing one or more rounds of training on the second diffusion network according to a training set, wherein the training set includes one or more sets of training data, and any set of training data includes an image pair consisting of a grayscale image and a height image, and a mask image; In any training round, the second diffusion network is trained according to any set of training data in the training set, including: A data processing step, in a current training round, obtaining an initial first image feature according to an image pair in current training data, and obtaining a first mask feature according to a mask image in current training data; A noisy step, obtaining a first high-dimensional noise; when the current number of noisy steps is the first number of noisy steps, the initial first image feature is denoised according to the first high-dimensional noise, to obtain a first image feature after the noisy operation corresponding to the first number of noisy steps; when the current number of noisy steps is not the first number of noisy steps, the first image feature after the noisy operation corresponding to the previous number of noisy steps is continuously noisy according to the first high-dimensional noise, to obtain a first image feature after the noisy operation corresponding to the current number of noisy steps; A fusion processing step, inputting the first image feature after the denoising operation corresponding to the current denoising step number and the first mask feature into the second diffusion network, and fusing the first image feature after the denoising operation corresponding to the current denoising step number and the first mask feature in the second diffusion network respectively; wherein the second diffusion network is used to perform noise prediction on the input of the second diffusion network to output high-dimensional noise when the current denoising step number is equal to the preset total denoising step number of the current training round; The training judgment step includes: if the current number of noise addition steps is less than the preset total number of noise addition steps of the current training round, the current number of noise addition steps is increased by 1 step to update the current number of noise addition steps, and the training is continued from the noise addition step; if the current number of noise addition steps is equal to the preset total number of noise addition steps of the current training round, the high-dimensional noise output by the second diffusion network is used as the second high-dimensional noise; the loss function of the second diffusion network is obtained according to the first high-dimensional noise and the second high-dimensional noise; if the loss function converges, the training is stopped.

2. The method according to claim 1, characterized in that The training judgment step also includes: If the loss function has not converged, the training is continued from the data processing step according to the next set of training data in the training set; if the current training data is the last set of training data in the training set, the training of the current training round is stopped.

3. The method according to claim 2, characterized in that The training judgment step also includes: After stopping the training of the current training round, the second diffusion network after the current training round is used as the second diffusion network used in the training process of the next training round, and the training of the next training round is started from the data processing step according to the first set of training data in the training set.

4. The method according to claim 3, characterized in that The training judgment step also includes: Before stopping the training of the current training round and starting the training of the next training round, it is also determined whether a preset maximum training round is reached. If reached, the training is stopped.

5. The method according to claim 1, characterized in that The step of continuing to add noise to the first image feature after the noise addition operation corresponding to the previous noise addition step number according to the first high-dimensional noise to obtain the first image feature after the noise addition operation corresponding to the current noise addition step number includes: For the t-th number of noise addition steps, where t is a positive integer, the noise intensity of the first high-dimensional noise corresponding to the t-th number of noise addition steps is obtained according to the preset intensity of the first high-dimensional noise corresponding to the first number of noise addition steps and the preset intensity of the first high-dimensional noise corresponding to the last number of noise addition steps; The noise intensity of the first high-dimensional noise corresponding to the t-th noisy step under the square root is used as the reference weight of the first high-dimensional noise corresponding to the t-th noisy step; the difference between 1 and the noise intensity of the first high-dimensional noise corresponding to the t-th noisy step is calculated, and the difference under the square root is used as the reference weight of the first image feature after the noisy operation corresponding to the t-1-th noisy step; the first image feature after the noisy operation corresponding to the t-1-th noisy step and the first high-dimensional noise corresponding to the t-th noisy step are multiplied by the corresponding reference weights respectively, and the multiplication result is used as the first image feature after the noisy operation corresponding to the t-1-th noisy step.

6. The method according to claim 5, characterized in that The step of obtaining the noise intensity of the first high-dimensional noise corresponding to the t-th number of noise addition steps according to the preset intensity of the first high-dimensional noise corresponding to the first number of noise addition steps and the preset intensity of the first high-dimensional noise corresponding to the last number of noise addition steps comprises: Calculate the square difference between the preset intensity of the first high-dimensional noise corresponding to the first noise addition step number and the preset intensity of the first high-dimensional noise corresponding to the last noise addition step number, and obtain the difference between the preset total number of noise addition steps in the current training round and 1; take the ratio between the obtained square difference and the obtained difference as the noise intensity change of a single noise addition; Multiply the difference between the current number of noise addition steps and 1 by the change in noise intensity of a single noise addition, and use the resulting product as the cumulative change in noise intensity corresponding to the current number of noise addition steps; add the cumulative change in noise intensity corresponding to the current number of noise addition steps to the square of the preset intensity of the first high-dimensional noise corresponding to the first number of noise addition steps, and then use the resulting addition result under the square root as the noise intensity of the first high-dimensional noise corresponding to the tth number of noise addition steps.

7. The method according to claim 1, characterized in that In the second diffusion network, the first image feature after the noise addition operation corresponding to the current noise addition step number and the first mask feature are respectively fused, including: The fusion process includes a first fusion process and a second fusion process; Obtaining a current noise step vector according to the current noise step number; for any network module in the second diffusion network, first performing a first fusion process on the input content of any network module and the current noise step vector to obtain a first fusion result of any network module; Any one of the training data sets further includes point cloud data and / or natural language data of an object corresponding to the image, and a fourth image feature corresponding to the current training data is obtained according to the point cloud data or natural language data or grayscale image in the current training data; a first general feature is obtained, and any one of the first general feature and the fourth image feature corresponding to the current training data is used as an attention parameter corresponding to the current training data; a second mask feature corresponding to the current training data can be obtained according to the mask image in the current training data; The first fusion result of any one of the network modules is subjected to a second fusion process with the attention parameters corresponding to the current training data and the second mask feature to obtain the second fusion result of any one of the network modules, and the output content of any one of the network modules is obtained based on the second fusion result of any one of the network modules.

8. The method according to claim 7, characterized in that The input content of any one of the network modules includes: If any one of the network modules is the first encoding module in the first diffusion network, the input content of any one of the network modules is the first image feature after the noise addition operation corresponding to the current noise addition step number; if any one of the network modules is the encoding module in the first diffusion network and is not the first encoding module, or if any one of the network modules is the intermediate module in the first diffusion network, the input content of any one of the network modules is the output content of the previous encoding module in the first diffusion network; If any one of the network modules is the first coding module in the control branch network, the input content of any one of the network modules is the first mask feature; if any one of the network modules is the coding module in the control branch network and is not the first coding module, or if any one of the network modules is an intermediate module in the control branch network, the input content of any one of the network modules is the output content of the previous coding module in the control branch network; If any one of the network modules is the first decoding module in the first diffusion network, the previous decoding module of any one of the network modules is the intermediate module; if any one of the network modules is the decoding module of the first diffusion network and is not the first decoding module, the previous decoding module of any one of the network modules in the first diffusion network is obtained; the output content of the encoding module with the same size as the previous decoding module in the control branch network is convolved with a convolution layer to obtain a branch fusion result; the branch fusion result, the output content of the previous decoding module and the output content of the encoding module with the same size as the previous decoding module in the first diffusion network are fused, and the obtained fusion result is used as the input content of any one of the network modules.

9. The method according to claim 7, characterized in that The step of obtaining a current noise adding step number vector according to the current noise adding step number includes: Set a preset length of the noise step vector, and obtain the logarithmic function value corresponding to the current noise step; for the kth element in the noise step vector, calculate the ratio between k-1 and the preset length; calculate the product between the obtained logarithmic function value and the obtained ratio, and use the exponential function value corresponding to the inverse of the obtained product as the kth element value in the current noise step vector.

10. The method according to claim 1, characterized in that The step of obtaining a loss function of a second diffusion network according to the first high-dimensional noise and the second high-dimensional noise includes: For any element value of the second high-dimensional noise, calculate the logarithmic function value of the arbitrary element value, multiply the obtained logarithmic function value with the element value of the first high-dimensional noise at the corresponding position, and obtain the multiplication result corresponding to the arbitrary element value; obtain the cumulative sum of the multiplication results corresponding to all element values ​​of the second high-dimensional noise, and use the inverse of the obtained cumulative sum as the loss function of the second diffusion network.

11. A noise prediction method, characterized in that: include: Acquire a second diffusion network trained by the training method according to claims 1 to 10; The predicted high-dimensional noise is obtained according to the second diffusion network, including: Acquire a first mask feature and a first image feature; The first image feature is gradually denoised, and when the number of denoising steps reaches a preset total number of denoising steps, the first image feature after denoising is obtained, the first image feature after denoising and the first mask feature are input into a second diffusion network, and in the second diffusion network, the first mask feature and the first image feature after denoising are respectively fused; the output of the second diffusion network is used as the predicted high-dimensional noise.

12. A defect image generation method, characterized in that: include: Acquire a second diffusion network trained by the training method according to claims 1 to 10; Acquire a mask image, and obtain a first mask feature and a second mask feature according to the mask image; Obtaining a new first image feature according to the second mask feature; Stepwise adding noise to the new first image feature according to a preset total number of steps for adding noise, and recording the first image feature after the last adding noise as the third image feature; Performing a denoising operation on the third image feature according to the first mask feature and the second diffusion network, and stopping the denoising when the number of denoising times is equal to the preset total number of denoising steps to obtain a first denoising feature map; A target image is obtained according to the first denoising feature map.

13. The method according to claim 12, characterized in that The obtaining a new first image feature according to the second mask feature comprises: Get comprehensive feature vector; For any pixel point in the new first image feature, when the pixel value of the pixel point at the corresponding position in the second mask feature is 1, the foreground standard deviation vector in the comprehensive feature vector is multiplied by the element value of the corresponding dimension of the standard high-dimensional noise, and the sum of the multiplied vector and the foreground mean vector is used as the high-dimensional feature vector corresponding to any pixel point in the new first image feature; when the pixel value of the pixel point at the corresponding position in the second mask feature is 0, the high-dimensional feature vector corresponding to any pixel point in the new first image feature is obtained according to the comprehensive feature vector or the reference image feature.

14. The method according to claim 13, characterized in that The obtaining of the comprehensive feature vector comprises: In any one of the training rounds as claimed in claims 1 to 10, the first feature vector is iteratively updated once in the data processing step of each set of training data, wherein: For the current training data, in the data processing step, the background area and foreground area of ​​the initial first image feature corresponding to the current training data and the mask image in the current training data are obtained, and the first feature vector corresponding to the current training data is obtained according to the background area and foreground area of ​​the initial first image feature; According to the first eigenvector corresponding to the current training data and the first eigenvector updated in the last iteration, the first eigenvector updated in the current iteration is obtained; wherein, if the current training data is the first set of training data in the training set, the first eigenvector updated in the last iteration is a 0 vector; The first feature vector after the last iteration update in any training round of the second diffusion network is used as the comprehensive feature vector.

15. The method according to claim 14, characterized in that The step of obtaining a first feature vector corresponding to current training data according to the background area and the foreground area of ​​the initial first image feature includes: The first feature vector corresponding to the current training data includes four vectors, and the four vectors are respectively the background mean vector, background standard deviation vector, foreground mean vector and foreground standard deviation vector of the initial first image feature; wherein the background mean vector of the initial first image feature is the mean vector of the high-dimensional feature vectors corresponding to all pixels in the background area of ​​the initial first image feature; the background standard deviation vector of the initial first image feature is the standard deviation vector of the high-dimensional feature vectors corresponding to all pixels in the background area of ​​the initial first image feature; the foreground mean vector of the initial first image feature is the mean vector of the high-dimensional feature vectors corresponding to all pixels in the foreground area of ​​the initial first image feature; the foreground standard deviation vector of the initial first image feature is the standard deviation vector of the high-dimensional feature vectors corresponding to all pixels in the foreground area of ​​the initial first image feature.

16. The method according to claim 14, characterized in that The step of obtaining the first feature vector updated in the current iteration according to the first feature vector corresponding to the current training data and the first feature vector updated in the previous iteration includes: In the qth iterative update of a training round, for any first eigenvector, a momentum coefficient is set, and the momentum coefficient is used as the reference weight of the any first eigenvector after the q-1th iterative update, and the difference between 1 and the momentum coefficient is used as the reference weight of the first eigenvector corresponding to the gth group of training data; the product of the any first eigenvector obtained after the g-1th iterative update, the first eigenvector corresponding to the gth group of training data and the corresponding reference weight is calculated respectively, and the sum of the two products is used as the any first eigenvector after the gth iterative update.

17. The method according to claim 13, characterized in that When the pixel value of the pixel point at the corresponding position in the second mask feature is 0, obtaining a high-dimensional feature vector corresponding to any pixel point in the new first image feature according to the comprehensive feature vector or the reference image feature includes: For any pixel point in the new first image feature, when the pixel value of the pixel point at the corresponding position in the second mask feature is 0, the background standard deviation vector in the comprehensive feature vector is multiplied by the element value on the corresponding dimension of the standard high-dimensional noise, and the sum of the multiplied vector and the background mean vector is used as the high-dimensional feature vector corresponding to any pixel point in the new first image feature; or, a reference image feature is obtained according to a preset image pair; the high-dimensional feature vector corresponding to the corresponding pixel point in the reference image feature is used as the high-dimensional feature vector corresponding to any pixel point in the new first image feature.

18. The method according to claim 12, characterized in that The performing a denoising operation on the third image feature according to the first mask feature and the second diffusion network includes: For the t'th denoising operation, the third image feature after the last denoising operation and the first mask feature are input into the second diffusion network, and the third image feature after the last denoising operation and the first mask feature are respectively fused in the second diffusion network; the third high dimensional noise and the fourth high dimensional noise corresponding to the t'th denoising operation are obtained according to the output of the second diffusion network; the fifth high dimensional noise corresponding to the t'th denoising operation is obtained according to the third high dimensional noise and the fourth high dimensional noise corresponding to the t'th denoising operation; the third image feature after the last denoising operation is denoised according to the fifth high dimensional noise corresponding to the t'th denoising operation, to obtain the third image feature after the t'th denoising operation.

19. The method according to claim 18, characterized in that The fusing process of the third image feature after the last denoising operation and the first mask feature in the second diffusion network respectively includes: Obtain a denoising step vector corresponding to the t'th denoising operation, and for any network module in the second diffusion network, first perform a first fusion process on the input content of any one of the network modules and the denoising step vector corresponding to the t'th denoising operation to obtain a first fusion result of the any one of the network modules; then perform a second fusion process on the first fusion result of the any one of the network modules, the attention parameter and the second mask feature to obtain a second fusion result of the any one of the network modules; wherein the third image feature after the last denoising operation is the input content of the first encoding module of the first diffusion network in the second diffusion network, and the first mask feature is the input content of the first encoding module of the control branch network in the second diffusion network.

20. The method of claim 19, wherein: The obtaining of the denoising step number vector corresponding to the t'th denoising operation includes: taking the result of subtracting t' from the preset total denoising step number and then adding 1 as the denoising step number corresponding to the t'th denoising operation; and taking the denoising step number vector corresponding to the denoising step number as the denoising step number vector corresponding to the t'th denoising operation.

21. The method of claim 18, wherein: The step of obtaining the third high-dimensional noise and the fourth high-dimensional noise corresponding to the t'th denoising operation according to the output of the second diffusion network includes: The fourth image feature and the first general feature are obtained; for the t'th denoising operation, the attention parameter used by any network module in the second diffusion network for the second fusion processing is set to the fourth image feature, the third image feature after the last denoising operation and the first mask feature are input into the second diffusion network, and the high-dimensional noise predicted by the second diffusion network is recorded as the third high-dimensional noise corresponding to the t'th denoising operation; the attention parameter used by any network module in the second diffusion network for the second fusion processing is set to the first general feature, and the third image feature after the last denoising operation and the first mask feature are input into the second diffusion network again, and the high-dimensional noise predicted by the second diffusion network is recorded as the fourth high-dimensional noise corresponding to the t'th denoising operation.

22. A computer-readable storage medium, characterized in that: A computer program is stored on the medium, and the computer program can be executed by a processor to implement the method according to any one of claims 1 to 21.

Images