Training method, prediction method, and defect image generation method for noise prediction

Abstract

The application discloses a training method, a prediction method and a defect image generation method for noise prediction. The training method comprises the following steps: training a second diffusion network; obtaining a first mask feature and a second mask feature; obtaining a new first image feature according to the second mask feature; gradually adding noise to the new first image feature according to a preset total number of noise adding steps to obtain a third image feature; inputting the third image feature and the first mask feature into the second diffusion network to obtain high-dimensional noise predicted by the second diffusion network, and then obtaining fifth high-dimensional noise; gradually removing noise from the third image feature according to the obtained fifth high-dimensional noise to obtain a first denoising feature map; and obtaining a target image according to the first denoising feature map. Through the additional information such as point cloud data, an image meeting the user demand is generated, on the basis of ensuring the integrity of information, the diversity of information is enhanced, and the generation effect of the target image is enhanced, so that the generation of the high-quality image under the pixel-level precision control is realized.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and more specifically to training methods, prediction methods, and defect image generation methods for noise prediction. Background Technology

[0002] In recent years, deep learning has become a focus of attention in various fields both domestically and internationally. Deep learning includes two categories: supervised learning and unsupervised learning. In the field of computer vision, supervised learning refers to training neural networks through a one-to-one correspondence between images and labeled information, enabling them to perform tasks such as classification, object detection, and semantic segmentation. Unsupervised learning only requires images and does not require additional labeled information for training.

[0003] In most scenarios, supervised learning performs better than unsupervised learning. However, supervised learning requires a large amount of images and annotation information. Annotating images is time-consuming and laborious, and the number of defect images in industrial scenarios such as defect detection is small, making it difficult to collect enough sample data. These factors greatly limit the verification and implementation of deep learning algorithms.

[0004] Existing technologies often compensate for the insufficient number of actual defect samples by synthesizing defect images. In particular, when the number of original defect samples is limited, diffusion models can create more and more diverse defect images, greatly enriching the sample data. However, since diffusion models usually rely on textual information to guide image generation, they are not precise enough in processing pixel-level image details. In application scenarios that require high realism and detailed texture representation, such as industrial defect detection, images generated by diffusion models may fail to achieve the expected image quality. Secondly, diffusion models often contain a huge number of parameters, making them difficult to train in specific domains. In other words, traditional diffusion networks suffer from problems such as inaccurate processing of image details and the difficulty of training due to the huge number of parameters.

[0005] Therefore, in order to overcome the shortcomings of traditional diffusion networks, it is necessary to design a network structure with fewer parameters that can achieve high-precision processing of image details. Furthermore, based on this, we can consider how to perform noise prediction based on the network, and even generate high-quality defect images based on the obtained noise prediction results. Summary of the Invention

[0006] In view of the above problems, this application provides a training method, a prediction method, and a defect image generation method for noise prediction, which are described in detail below.

[0007] According to the first aspect, some embodiments provide a training method for noise prediction, comprising:

[0008] Obtain a second diffusion network formed by adding a control branch network to a pre-trained first diffusion network, wherein the first diffusion network is a preset diffusion network;

[0009] The second diffusion network is trained for one or more rounds based on a training set, wherein the training set includes one or more sets of training data, and any set of training data includes image pairs consisting of grayscale images and height maps, and mask images; during the training process of any training round, the second diffusion network is trained based on any set of training data in the training set, including:

[0010] The data processing steps are as follows: In the current training round, the initial first image features are obtained based on the image pairs in the current training data, and the first mask features are obtained based on the mask image in the current training data.

[0011] In the noise addition step, a first high-dimensional noise is obtained; if the current noise addition step is the first noise addition step, the initial first image feature is noise-added according to the first high-dimensional noise to obtain the first image feature after noise addition operation corresponding to the first noise addition step; if the current noise addition step is not the first noise addition step, the first image feature after noise addition operation corresponding to the previous noise addition step is noise-added again according to the first high-dimensional noise to obtain the first image feature after noise addition operation corresponding to the current noise addition step.

[0012] The fusion processing step involves inputting the first image feature after noise addition corresponding to the current noise addition step number and the first mask feature into the second diffusion network. In the second diffusion network, the first image feature after noise addition corresponding to the current noise addition step number and the first mask feature are fused respectively. The second diffusion network is used to perform noise prediction on the input of the second diffusion network to output high-dimensional noise when the current noise addition step number is equal to the preset total noise addition step number of the current training round.

[0013] In the training judgment step, if the current noise addition step is less than the preset total noise addition step for the current training round, then the current noise addition step is increased by 1 step to update the current noise addition step, and training continues from the noise addition step; if the current noise addition step is equal to the preset total noise addition step for the current training round, then the high-dimensional noise output by the second diffusion network is used as the second high-dimensional noise; the loss function of the second diffusion network is obtained based on the first high-dimensional noise and the second high-dimensional noise; if the loss function converges, then training stops.

[0014] In some embodiments, the training and judgment step further includes:

[0015] If the loss function does not converge, training continues from the data processing step based on the next set of training data in the training set; if the current training data is the last set of training data in the training set, training for the current training round is stopped.

[0016] In some embodiments, the training and judgment step further includes:

[0017] After stopping the training of the current training round, the second diffusion network after the end of the current training round is used as the second diffusion network for the training process of the next training round, and the training of the next training round starts from the data processing step based on the first set of training data in the training set.

[0018] In some embodiments, the training and judgment step further includes:

[0019] Before stopping the current training round and starting the next training round, it also checks whether the preset maximum number of training rounds has been reached. If it has, then training stops.

[0020] In some embodiments, the step of further adding noise to the first image features after the noise addition operation corresponding to the previous noise addition step number based on the first high-dimensional noise, to obtain the first image features after the noise addition operation corresponding to the current noise addition step number, includes:

[0021] For the t-th noise addition step, where t is a positive integer, the noise intensity of the first high-dimensional noise corresponding to the t-th noise addition step is obtained based on the preset intensity of the first high-dimensional noise corresponding to the first noise addition step and the preset intensity of the first high-dimensional noise corresponding to the last noise addition step.

[0022] The noise intensity of the first high-dimensional noise corresponding to the t-th noise addition step under the square root is used as the reference weight of the first high-dimensional noise corresponding to the t-th noise addition step; the difference between 1 and the noise intensity of the first high-dimensional noise corresponding to the t-th noise addition step is calculated, and the difference under the square root is used as the reference weight of the first image feature after noise addition operation corresponding to the (t-1)-th noise addition step; the first image feature after noise addition operation corresponding to the (t-1)-th noise addition step and the first high-dimensional noise corresponding to the t-th noise addition step are multiplied by the corresponding reference weights, and the multiplication result is used as the first image feature after noise addition operation corresponding to the t-th noise addition step.

[0023] In some embodiments, obtaining the noise intensity of the first high-dimensional noise corresponding to the t-th noise-adding step based on the preset intensity of the first high-dimensional noise corresponding to the first noise-adding step and the preset intensity of the first high-dimensional noise corresponding to the last noise-adding step includes:

[0024] Calculate the squared difference between the preset intensity of the first high-dimensional noise corresponding to the first noise addition step and the preset intensity of the first high-dimensional noise corresponding to the last noise addition step, and obtain the difference between the preset total number of noise addition steps in the current training round and 1; use the ratio between the obtained squared difference and the obtained difference as the noise intensity change of a single noise addition.

[0025] Multiply the difference between the current noise addition step number and 1 by the noise intensity change of a single noise addition, and use the product as the cumulative noise intensity change corresponding to the current noise addition step number; add the cumulative noise intensity change corresponding to the current noise addition step number to the square of the preset intensity of the first high-dimensional noise corresponding to the first noise addition step number, and then use the square root result as the noise intensity of the first high-dimensional noise corresponding to the t-th noise addition step number.

[0026] In some embodiments, the second diffusion network performs fusion processing on the first image features after the noise addition operation corresponding to the current noise addition step and the first mask features, including:

[0027] The fusion process includes a first fusion process and a second fusion process;

[0028] The current noise step number vector is obtained based on the current noise step number; for any network module in the second diffusion network, the input content of the network module is first fused with the current noise step number vector to obtain the first fusion result of the network module;

[0029] The training data set further includes point cloud data and / or natural language data of the corresponding objects of the image pair. The fourth image feature corresponding to the current training data is obtained based on the point cloud data, natural language data, or grayscale image in the current training data. The first general feature is obtained, and either the first general feature or the fourth image feature corresponding to the current training data is used as the attention parameter corresponding to the current training data. The second mask feature corresponding to the current training data can be obtained based on the mask image in the current training data.

[0030] The first fusion result of any one of the network modules is combined with the attention parameters and second mask features corresponding to the current training data to perform a second fusion process, thereby obtaining the second fusion result of any one of the network modules. The second fusion result of any one of the network modules is then used as the output content of any one of the network modules.

[0031] In some embodiments, the input content of any one of the network modules includes:

[0032] If any of the network modules is the first encoding module in the first diffusion network, the input of any of the network modules is the first image feature after the noise addition operation corresponding to the current noise addition step; if any of the network modules is an encoding module in the first diffusion network but is not the first encoding module, or if any of the network modules is an intermediate module in the first diffusion network, the input of any of the network modules is the output of the previous encoding module in the first diffusion network;

[0033] If any of the network modules is the first encoding module in the control branch network, the input content of the network module is the first mask feature; if any of the network modules is an encoding module in the control branch network but is not the first encoding module, or if any of the network modules is an intermediate module in the control branch network, the input content of the network module is the output content of the previous encoding module in the control branch network.

[0034] If any of the network modules is the first decoding module in the first diffusion network, then the previous decoding module of any of the network modules is an intermediate module; if any of the network modules is a decoding module of the first diffusion network but is not the first decoding module, then the previous decoding module of any of the network modules in the first diffusion network is obtained; the output of the encoding module in the control branch network with the same size as the previous decoding module is convolved with a convolutional layer to obtain a branch fusion result; then the branch fusion result, the output of the previous decoding module, and the output of the encoding module in the first diffusion network with the same size as the previous decoding module are fused, and the obtained fusion result is used as the input of any of the network modules.

[0035] In some embodiments, obtaining the current noise-adding step vector based on the current noise-adding step number includes:

[0036] Set the preset length of the noise step vector, obtain the logarithmic function value corresponding to the current noise step; for the k-th element in the noise step vector, calculate the ratio between k-1 and the preset length; calculate the product between the obtained logarithmic function value and the obtained ratio, and take the exponential function value corresponding to the opposite of the obtained product as the value of the k-th element in the current noise step vector.

[0037] In some embodiments, obtaining the loss function of the second diffusion network based on the first high-dimensional noise and the second high-dimensional noise includes:

[0038] For any element value of the second high-dimensional noise, calculate the logarithmic function value of the arbitrary element value, multiply the obtained logarithmic function value by the element value of the first high-dimensional noise at the corresponding position, and obtain the multiplication result corresponding to the arbitrary element value; obtain the sum of the multiplication results corresponding to all element values ​​of the second high-dimensional noise, and use the negative of the obtained sum as the loss function of the second diffusion network.

[0039] According to the second aspect, some embodiments provide a noise prediction method, including:

[0040] Obtain the second diffusion network trained using the aforementioned training method;

[0041] The predicted high-dimensional noise obtained from the second diffusion network includes:

[0042] Obtain the first mask features and the first image features;

[0043] The first image feature is gradually denoised. When the number of denoising steps reaches the preset total number of denoising steps, the denoised first image feature is obtained. The denoised first image feature and the first mask feature are input into the second diffusion network. In the second diffusion network, the first mask feature and the denoised first image feature are fused respectively. The output of the second diffusion network is used as the predicted high-dimensional noise.

[0044] According to a third aspect, some embodiments provide a method for generating a defect image, including:

[0045] Obtain the second diffusion network trained using the aforementioned training method;

[0046] Obtain a mask image, and obtain a first mask feature and a second mask feature based on the mask image;

[0047] A new first image feature is obtained based on the second mask feature;

[0048] The new first image feature is gradually denoised according to the preset total number of denoising steps, and the first image feature after the last denoising step is recorded as the third image feature.

[0049] Denoising is performed on the third image features based on the first mask features and the second diffusion network. When the number of denoising steps is equal to the preset total number of noise addition steps, denoising is stopped to obtain the first denoised feature map.

[0050] The target image is obtained based on the first denoised feature map.

[0051] In some embodiments, obtaining new first image features based on the second mask features includes:

[0052] Obtain the comprehensive feature vector;

[0053] For any pixel in the new first image feature, when the pixel value of the corresponding pixel in the second mask feature is 1, the foreground standard deviation vector in the comprehensive feature vector is multiplied by the element value of the corresponding dimension of the standard high-dimensional noise, and the sum of the resulting vector and the foreground mean vector is used as the high-dimensional feature vector corresponding to any pixel in the new first image feature; when the pixel value of the corresponding pixel in the second mask feature is 0, the high-dimensional feature vector corresponding to any pixel in the new first image feature is obtained according to the comprehensive feature vector or by referring to the image feature.

[0054] In some embodiments, obtaining the comprehensive feature vector includes:

[0055] In any training round of the training method, the first feature vector is iteratively updated once in the data processing step of each set of training data, wherein:

[0056] For the current training data, in the data processing step, the background region and foreground region of the initial first image feature are obtained based on the initial first image feature corresponding to the current training data and the mask image in the current training data, and the first feature vector corresponding to the current training data is obtained based on the background region and foreground region of the initial first image feature.

[0057] Based on the first feature vector corresponding to the current training data and the first feature vector after the last iteration, the first feature vector after the last iteration is obtained; where, if the current training data is the first set of training data in the training set, the first feature vector after the last iteration is a 0 vector.

[0058] In any training round of the second diffusion network, the first feature vector updated in the last iteration is used as the comprehensive feature vector.

[0059] In some embodiments, obtaining the first feature vector corresponding to the current training data based on the background region and foreground region of the initial first image features includes:

[0060] The first feature vector corresponding to the current training data includes four types of vectors: the background mean vector, background standard deviation vector, foreground mean vector, and foreground standard deviation vector of the initial first image feature. Specifically, the background mean vector of the initial first image feature is the mean vector of the high-dimensional feature vectors corresponding to all pixels in the background region of the initial first image feature; the background standard deviation vector of the initial first image feature is the standard deviation vector of the high-dimensional feature vectors corresponding to all pixels in the background region of the initial first image feature; the foreground mean vector of the initial first image feature is the mean vector of the high-dimensional feature vectors corresponding to all pixels in the foreground region of the initial first image feature; and the foreground standard deviation vector of the initial first image feature is the standard deviation vector of the high-dimensional feature vectors corresponding to all pixels in the foreground region of the initial first image feature.

[0061] In some embodiments, obtaining the first feature vector updated in the current iteration based on the first feature vector corresponding to the current training data and the first feature vector updated in the previous iteration includes:

[0062] In the g-th iteration update of a training round, for any first feature vector, a momentum coefficient is set, and the momentum coefficient is used as the reference weight of the first feature vector after the (g-1)-th iteration update. The difference between 1 and the momentum coefficient is used as the reference weight of the first feature vector corresponding to the g-th training data. The products of the first feature vector obtained after the (g-1)-th iteration update, the first feature vector corresponding to the g-th training data, and the corresponding reference weights are calculated respectively. The sum of the two products is used as the first feature vector after the g-th iteration update.

[0063] In some embodiments, when the pixel value of the corresponding pixel in the second mask feature is 0, obtaining the high-dimensional feature vector corresponding to any pixel in the new first image feature based on the comprehensive feature vector or by referring to the image feature includes:

[0064] For any pixel in the new first image feature, when the pixel value of the corresponding pixel in the second mask feature is 0, the background standard deviation vector in the comprehensive feature vector is multiplied by the element value of the corresponding dimension of the standard high-dimensional noise, and the sum of the resulting vector and the background mean vector is used as the high-dimensional feature vector corresponding to any pixel in the new first image feature; or, reference image features are obtained according to a preset image pair; the high-dimensional feature vector corresponding to the corresponding pixel in the reference image features is used as the high-dimensional feature vector corresponding to any pixel in the new first image feature.

[0065] In some embodiments, the denoising operation on the third image features based on the first mask features and the second diffusion network includes:

[0066] For the denoising operation t', the third image feature after the previous denoising operation and the first mask feature are input into the second diffusion network. The second diffusion network performs fusion processing on the third image feature after the previous denoising operation and the first mask feature respectively. The third high-dimensional noise and the fourth high-dimensional noise corresponding to the denoising operation t' are obtained according to the output of the second diffusion network. The fifth high-dimensional noise corresponding to the denoising operation t' is obtained according to the third high-dimensional noise and the fourth high-dimensional noise corresponding to the denoising operation t'. The third image feature after the previous denoising operation is denoised according to the fifth high-dimensional noise corresponding to the denoising operation t', to obtain the third image feature after the denoising operation t'.

[0067] In some embodiments, the fusion processing of the third image features after the previous denoising operation and the first mask features in the second diffusion network includes:

[0068] Obtain the noise step vector corresponding to the t'th denoising operation. For any network module in the second diffusion network, first perform a first fusion process with the noise step vector corresponding to the t'th denoising operation to obtain the first fusion result of the network module. Then, perform a second fusion process with the attention parameters and the second mask features to obtain the second fusion result of the network module. The third image feature after the previous denoising operation is the input content of the first encoding module of the first diffusion network in the second diffusion network, and the first mask feature is the input content of the first encoding module of the control branch network in the second diffusion network.

[0069] In some embodiments, obtaining the noise step vector corresponding to the t'th denoising operation includes: subtracting t' from the preset total noise step count and then adding 1 as the noise step count corresponding to the t'th denoising operation; and using the noise step vector corresponding to the noise step count as the noise step vector corresponding to the t'th denoising operation.

[0070] In some embodiments, obtaining the third high-dimensional noise and the fourth high-dimensional noise corresponding to the t'th denoising operation based on the output of the second diffusion network includes:

[0071] Obtain the fourth image feature and the first general feature; for the denoising operation t', set the attention parameter used by any network module in the second diffusion network when performing the second fusion process as the fourth image feature, input the third image feature after the previous denoising operation and the first mask feature into the second diffusion network, and record the high-dimensional noise predicted by the second diffusion network as the third high-dimensional noise corresponding to the denoising operation t'; then set the attention parameter used by any network module in the second diffusion network when performing the second fusion process as the first general feature, and input the third image feature after the previous denoising operation and the first mask feature into the second diffusion network again, and record the high-dimensional noise predicted by the second diffusion network as the fourth high-dimensional noise corresponding to the denoising operation t'.

[0072] According to the training method for noise prediction in the above embodiments, firstly, the grayscale image and height map are superimposed to extract the first image features. Then, the first mask features and second mask features are extracted based on the mask image. Finally, global features of different types of objects, i.e., the fourth image features, are extracted based on the object's point cloud, grayscale image, and natural language. Then, through contrastive learning, the global features include not only basic information about the objects themselves but also relevant information between objects. Finally, the second diffusion network, composed of the first diffusion network and the control branch network, is trained based on the first image features, the first mask features, the second mask features, and the fourth image features or the first general features. By fixing the parameters within the first diffusion network and updating only the parameters in the control branch network and some learnable parameters, the number of parameters within the network during training is reduced, lowering the training difficulty and achieving precise fine-tuning without changing the original diffusion model. Simultaneously, the control branch network provides additional information to the first diffusion network, thereby improving the first diffusion network's difficulty in achieving pixel-level image generation.

[0073] According to the noise prediction method of the above embodiment, firstly, a first mask feature and a first image feature are obtained; the obtained first image feature is gradually denoised to obtain a denoised first image feature; then, the denoised first image feature and the first mask feature are input into a second diffusion network, and the first mask feature and the denoised first image feature are fused in the second diffusion network respectively; finally, the output of the second diffusion network is used as the predicted high-dimensional noise; in this process, a second fusion process is performed with attention parameters in different network modules, and by setting the attention parameters, noise prediction results can be obtained under conditions (i.e., when additional information such as point clouds are involved) or unconditions (i.e., when additional information such as point clouds is not involved);

[0074] According to the defect image generation method of the above embodiments, the method can choose whether to require the defect sample data to have a specified background area to generate new first image features based on actual needs. This enriches the defect sample data and better meets user needs, making the final target image more diverse in form. In addition, since the second diffusion network used in the target image acquisition process can input additional information such as point clouds, and the mask image contains rich positional information, it can enhance the diversity of information while ensuring the integrity of the information, thereby enhancing the generation effect of the target image and achieving the generation of high-quality target images under pixel-level precision control. Attached Figure Description

[0075] Figure 1 This is a flowchart of a training method for noise prediction.

[0076] Figure 2 This is a flowchart of a noise prediction method;

[0077] Figure 3 This is a flowchart of a method for generating defect images;

[0078] Figure 4 The training flowchart for the second diffusion network is shown below.

[0079] Figure 5 This is a schematic diagram of the network structure of the second diffusion network. Detailed Implementation

[0080] The present invention will now be described in further detail with reference to specific embodiments and accompanying drawings. Similar elements in different embodiments are referred to by associated similar element reference numerals. In the following embodiments, many details are described to facilitate a better understanding of this application. However, those skilled in the art will readily recognize that some features may be omitted in different situations, or may be replaced by other elements, materials, or methods. In some cases, certain operations related to this application are not shown or described in the specification. This is to avoid obscuring the core parts of this application with excessive description. For those skilled in the art, detailed description of these related operations is not necessary; they can fully understand the related operations based on the description in the specification and general technical knowledge in the art.

[0081] Furthermore, the features, operations, or characteristics described in the specification can be combined in any suitable manner to form various embodiments. At the same time, the steps or actions in the method description can be rearranged or adjusted in a manner obvious to those skilled in the art. Therefore, the various orders in the specification and drawings are only for the clear description of a particular embodiment and do not imply a necessary order, unless otherwise stated that a particular order must be followed.

[0082] The serial numbers assigned to components in this document, such as "first" and "second," are used only to distinguish the described objects and have no sequential or technical meaning. The terms "connection" and "linkage" used in this application, unless otherwise specified, include both direct and indirect connections (linkages).

[0083] In this embodiment of the invention, the grayscale image and height map are first overlaid to extract the first image feature. Then, the first mask feature and the second mask feature are extracted based on the mask image. Finally, global features of different types of objects, i.e., the fourth image feature, are extracted based on the object's point cloud, grayscale image, and natural language. Then, through contrastive learning, the global features are made to include not only basic information about the objects themselves but also relevant information between objects. Finally, the second diffusion network, composed of the first diffusion network and the control branch network, is trained based on the first image feature, the first mask feature, the second mask feature, and the fourth image feature or the first general feature. The control branch network provides additional information to the first diffusion network. The first diffusion network can generate pixel-level images by adding information to improve the first diffusion network's inability to achieve this. Then, based on actual needs, it can choose whether to require the defect sample data to have a specified background area to generate new first image features. This enriches the defect sample data while better meeting user needs, resulting in more diverse forms of the final target image. Furthermore, since the second diffusion network used in the target image acquisition process can input additional information such as point clouds, and the mask image contains rich positional information, it can enhance the diversity of information while ensuring the integrity of the information, thereby enhancing the generation effect of the target image and achieving high-quality target image generation under pixel-level precision control.

[0084] Please refer to Figure 1 Some embodiments provide a training method for noise prediction, which includes the following steps:

[0085] Step S100: Train the feature extraction network to obtain the first image feature extraction network, the first feature decoding network, the first mask feature extraction network, and the second feature extraction network.

[0086] The feature extraction network involved in this embodiment includes a first image feature extraction network, a first feature decoding network, a first mask feature extraction network, and a second feature extraction network;

[0087] In some embodiments, the process of training the first image feature extraction network and the first feature decoding network includes: firstly training a variational autoencoder, which is called the first image variational autoencoder, and using the encoder in the first image variational autoencoder as the first image feature extraction network, and using the decoder in the first image variational autoencoder as the first feature decoding network.

[0088] The basic unit of the dataset used to train the first image variational autoencoder is a training image pair. Each training image pair contains a grayscale image and a height image. The result of channel superposition of the grayscale image and the height image in an image pair is denoted as δ. At this time, the grayscale information and the height information each correspond to a channel. Therefore, the result of channel superposition can be regarded as a two-channel image. Then, the result of channel superposition is input into the first image variational autoencoder. The first image variational autoencoder will output the reconstructed grayscale image and the height image. That is, the output of the first image variational autoencoder can also be regarded as a reconstructed two-channel image.

[0089] The loss function of the first image variational autoencoder is based on the distance function between its output and input. In this embodiment, the distance function used to train the first image variational autoencoder is Euclidean distance. θ It can be represented as:

[0090] loss θ =(θ vae (δ)-δ) 2

[0091] Where, θ vae δ is the result of channel overlay of the grayscale image and the height map, and it is used as the input of the first image variational autoencoder; θ vae (δ) represents the output of the first image variational autoencoder.

[0092] It should be noted that if a grayscale image lacks a pre-made corresponding height map, depth estimation algorithms such as depthanything can be used to process the grayscale image and then obtain the corresponding height map.

[0093] In some embodiments, the process of obtaining the first mask feature extraction network includes: retraining a new variational autoencoder and calling it the first mask variational autoencoder, and using the encoder of the first mask variational autoencoder as the first mask feature extraction network.

[0094] The dataset used to train the first mask variational autoencoder is a large dataset of mask images, where pixels with a value of 1 represent the foreground region and pixels with a value of 0 represent the background region. After inputting the mask images into the first mask variational autoencoder, it outputs a reconstructed mask image. The distance function between the output and input of the first mask variational autoencoder is used as the loss function to train it. In this embodiment, the distance function used to train the first mask variational autoencoder is Euclidean distance. Therefore, the loss function of the first mask variational autoencoder is... It can be represented as:

[0095]

[0096] in, γ is the first mask variational autoencoder, and γ is the mask image, which is the input of the first mask variational autoencoder; This represents the output of the first mask variational autoencoder.

[0097] In some embodiments, the process of obtaining the second feature extraction network includes:

[0098] The second feature extraction network includes a second point cloud feature extraction network, a second image feature extraction network, and a second text feature extraction network;

[0099] For the second point cloud feature extraction network, the preprocessed point cloud data is input into the point cloud neural network for feature extraction. In this embodiment, the point cloud neural network used is a pre-trained PointNet++ network. Let r be the number of feature vectors obtained. The r feature vectors are used as tokens and a trainable class representation is added. The r feature vectors and the trainable class representation are input into the encoder of the transformer model. The feature corresponding to the class representation in the output of the last layer of the encoder network is the global feature of the point cloud, thereby realizing feature extraction of point cloud data. The trained second point cloud feature extraction network can be considered to have a strong semantic information extraction capability for point clouds. Among them, the class representation is a basic concept of the transformer model, which will not be elaborated here.

[0100] For the second grayscale feature extraction network and the second text feature extraction network, pre-trained weights from image encoders such as Open-Clip and natural language encoders are generally used as pre-trained weights for the second grayscale feature extraction network and the second text feature extraction network. The weights of the second grayscale feature extraction network and the second text feature extraction network are fixed, which is equivalent to the second grayscale feature extraction network and the second text feature extraction network being fixed models, with only the second point cloud feature extraction network being trained. Alternatively, the second grayscale feature extraction network and the second text feature extraction network can be trained simultaneously. The dataset required for training the second grayscale feature extraction network is a dataset composed of grayscale images, and the dataset required for training the second text feature extraction network is a dataset composed of natural language descriptions. Thus, global features of the point cloud are obtained from the second point cloud feature extraction network, global features of the grayscale images are obtained from the second grayscale feature extraction network, and global features of the natural language are obtained from the second text feature extraction network.

[0101] Furthermore, to bring the representations of different modalities from the same object closer together while keeping the representations of different objects further apart, this embodiment constructs positive and negative sample pairs through contrastive learning. Positive sample pairs represent the representations of different modalities of the same object, specifically the global features of the point cloud, the global features of the grayscale image, and the global features of natural language in this embodiment. Negative sample pairs represent the representations of different modalities of different objects. The loss function of the contrastive learning model is... D It can be represented as:

[0102]

[0103] Where N represents the number of basic units used in training, and in this embodiment, the point cloud, grayscale image, and natural language description representing the same object constitute one basic unit; f αi f represents the global feature of the point cloud corresponding to the i-th basic unit; βi This represents the global feature of the natural language corresponding to the i-th basic unit; f represents the global feature of the grayscale image corresponding to the i-th basic unit; αj f represents the global feature of the point cloud corresponding to the j-th basic unit; βj f represents the global feature of the natural language corresponding to the j-th basic unit; χj Let represent the global feature of the grayscale image corresponding to the j-th basic unit; ι is a learnable parameter; S iu This represents the similarity of the u-th comparison corresponding to the i-th basic unit; exp() is the exponential function; log() is the logarithmic function; "·" represents the dot product symbol.

[0104] Among them, fαi ·f βi f is a similarity measure used to represent the positive sample pair consisting of the global features of the point cloud corresponding to the i-th basic unit and the global features of its corresponding natural language. αi ·f βj This represents the similarity measure between the global features of the point cloud corresponding to the i-th basic unit and the global features of the natural language corresponding to other basic units, forming a negative sample pair, and so on; different contrastive similarities are obtained for each basic unit based on the different global features used to form positive and negative sample pairs, and the loss function for contrastive learning is obtained based on all contrastive similarities corresponding to all basic units.

[0105] By contrastive learning, the representations of different global features in the feature space can be made richer and more discriminative. At this point, the representations of these global features not only contain the basic information of the objects, but also the similarity and difference information between the objects, thereby improving the performance of the model in subsequent tasks.

[0106] Step S110: Obtain the second diffusion network formed by adding the control branch network to the pre-trained first diffusion network.

[0107] In this embodiment, the first diffusion network is a preset diffusion network, which includes two parts: an encoding network and a decoding network. The encoding network contains multiple encoding modules, each of which consists of one downsampling operation, several convolutions, and several nonlinear activation functions. The downsampling operation can be pooling with a stride greater than 1, convolution with a stride greater than 1, or resolution upscaling based on interpolation, etc. The decoding network contains multiple decoding modules, each of which consists of one upsampling operation, several convolutions, and several nonlinear activation functions. The upsampling operation can be transposed convolution, resolution downscaling based on interpolation, etc.

[0108] All weight parameters of the pre-trained first diffusion network are "frozen," meaning no weight parameter updates are performed. Then, a network structure identical to the encoding network of the first diffusion network and an encoding network of the control branch network with identical parameters are constructed. The resulting network, along with its corresponding zero-initialized convolutional layers, constitutes the control branch network. The control branch network is then skip-connected to the pre-trained first diffusion network, thus forming the network structure of the second diffusion network, as shown below. Figure 5 As shown, this method greatly reduces the number of parameters within the network during training, thus lowering the training difficulty.

[0109] Step S120: Train the second diffusion network for one or more rounds based on the training set to obtain the second diffusion network.

[0110] In this embodiment, the training set used to train the second diffusion network includes one or more sets of training data. Any set of training data includes an image pair consisting of a set of grayscale images and height maps, a mask image, and point cloud data and natural language data of the object corresponding to the image pair. The mask image can be randomly selected from the mask image dataset. During any training epoch, the second diffusion network is trained based on any set of training data in the training set, such as... Figure 4 As shown, it includes:

[0111] In data processing step S121, in the current training round, an initial first image feature is obtained based on the image pairs in the current training data, and a first mask feature is obtained based on the mask image in the current training data; that is, the image pairs in the current training data are input into the first image feature extraction network, and the output of the first image feature extraction network is used as the initial first image feature corresponding to the current training data. Then, the mask image in the current training data is input into the first mask feature extraction network to obtain the first mask feature corresponding to the current training data.

[0112] In the noise addition step S122, the first high-dimensional noise ε1 is obtained, where the first high-dimensional noise is a standard high-dimensional noise, and each dimension of it follows a standard Gaussian distribution, i.e., ε1~N(0,I), where I represents the identity matrix, and each dimension of it follows a standard Gaussian distribution with a mean of 0 and a variance of 1.

[0113] First, set the total number of noise-adding steps for the current training round. The total number of noise-adding steps for each training round can be randomly set within a preset range. That is, the total number of noise-adding steps for each training round is randomly set within a range of 2 to a preset value. This preset value can be set according to the requirements. In this embodiment, the preset value is set to 30 during the training of the second diffusion network.

[0114] Then, if the current noise addition step is the first noise addition step, the first high-dimensional noise is used to add noise to the initial first image feature to obtain the first image feature after the noise addition operation corresponding to the first noise addition step; if the current noise addition step is not the first noise addition step, the first high-dimensional noise is used to continue adding noise to the first image feature after the noise addition operation corresponding to the previous noise addition step to obtain the first image feature after the noise addition operation corresponding to the current noise addition step.

[0115] For example, let the current noise addition step number be t. Based on the first high-dimensional noise and the first image features after the noise addition operation corresponding to the previous noise addition step number t-1, the first image features after the noise addition operation corresponding to the current noise addition step number are obtained. The specific process is as follows: the noise intensity of the first high-dimensional noise corresponding to the t-th noise addition step number is obtained based on the preset intensity of the first high-dimensional noise corresponding to the first noise addition step number and the preset intensity of the first high-dimensional noise corresponding to the last noise addition step number; then the noise intensity of the first high-dimensional noise corresponding to the t-th noise addition step number under the square root is taken as the first high-dimensional noise corresponding to the t-th noise addition step number. The reference weights for noise are calculated as follows: The difference between the noise intensity of step 1 and the noise intensity of the first high-dimensional noise corresponding to the t-th noise addition step is used as the reference weight of the first image feature after noise addition corresponding to the (t-1)-th noise addition step. The first image feature after noise addition corresponding to the (t-1)-th noise addition step and the first high-dimensional noise corresponding to the t-th noise addition step are multiplied by their respective reference weights, and the result of the multiplication is used as the first image feature after noise addition corresponding to the t-th noise addition step. Therefore, the first image feature after noise addition corresponding to the t-th noise addition step can be expressed as:

[0116]

[0117] Where, x t This indicates that the current noise-adding step corresponds to the first image feature after the noise-adding operation, that is, the t-th noise-adding step corresponds to the first image feature after the noise-adding operation; x t-1 This represents the first image feature after the noise addition operation corresponding to the previous noise addition step, that is, the first image feature after the noise addition operation corresponding to the (t-1)th noise addition step; ε1 represents the first high-dimensional noise; β t Let t be the noise intensity of the first high-dimensional noise during the noise addition operation corresponding to the current noise addition step number t;

[0118] The noise intensity of the first high-dimensional noise corresponding to the current t-th noise addition step is obtained based on the preset intensity of the first high-dimensional noise during the first noise addition and the preset intensity of the last noise addition. Specifically, the process is as follows: Calculate the squared difference between the preset intensity of the first high-dimensional noise corresponding to the first noise addition step and the preset intensity of the first high-dimensional noise corresponding to the last noise addition step; obtain the difference between the preset total number of noise addition steps in the current training round and 1; use the ratio of the obtained squared difference to the obtained difference as the noise intensity change for a single noise addition; multiply the difference between the current noise addition step and 1 by the noise intensity change for a single noise addition, and use the product as the cumulative noise intensity change corresponding to the current noise addition step; add the cumulative noise intensity change corresponding to the current noise addition step to the square of the preset intensity of the first high-dimensional noise corresponding to the first noise addition step, and then use the square root of the sum as the noise intensity of the first high-dimensional noise corresponding to the t-th noise addition step. Therefore, the noise intensity of the first high-dimensional noise corresponding to the t-th noise addition step can be expressed as:

[0119]

[0120] Where, β t Let β1 be the noise intensity of the first high-dimensional noise during the t-th noise addition step; β2 is the preset intensity of the first high-dimensional noise added during the 1st noise addition step, i.e., the preset intensity of the first high-dimensional noise during the first noise addition. T The preset intensity of the first high-dimensional noise added during the noise addition operation corresponding to the Tth noise addition step is the preset intensity of the first high-dimensional noise during the last noise addition; T is the total number of noise addition steps in the current training round. This represents the change in noise intensity during a single noise addition. This represents the cumulative change in noise intensity corresponding to the current number of noise addition steps;

[0121] In the fusion processing step S123, the first image features and the first mask features corresponding to the current noise addition step are input into the second diffusion network, and the first image features and the first mask features corresponding to the current noise addition step are fused in the second diffusion network respectively.

[0122] In some embodiments, the fusion process includes a first fusion process and a second fusion process, the specific process of which includes:

[0123] First, the current noise-adding step vector is obtained based on the current noise-adding step count. The method for obtaining the current noise-adding step vector is as follows: For the k-th element in the current noise-adding step vector, obtain the logarithmic function value corresponding to the current noise-adding step count; for the k-th element in the noise-adding step vector, calculate the ratio between k-1 and the preset length of the noise-adding step vector; calculate the product between the obtained logarithmic function value and the obtained ratio, and take the exponential function value corresponding to the opposite of the obtained product as the value of the k-th element in the current noise-adding step vector. Therefore, the k-th element in the current noise-adding step vector can be expressed as:

[0124]

[0125] Where, τ k represents the value of the k-th element in the current noise step vector; t is the current noise step number; L is the preset length of the noise step vector, which is also the number of elements in the noise step vector, and this value is equal to the dimension of the first image feature; log() is the logarithmic function, e is the natural constant, which is also the base of the exponential function.

[0126] Then, for any network module in the second diffusion network, the input content of the network module is first fused with the current noise step vector to obtain the first fusion result of the network module; that is, the obtained current noise step vector is copied and expanded into a feature map of the same size as the input content of the network module, and the result of adding each element in the obtained feature map with the corresponding element in the input content of the network module is used as the element value in the first fusion result of the network module; for example: suppose the current noise step vector is a 1×1×L vector, and the input of the network module... The input content has a size of H×W×C, where L=C. The current noise-adding step vector is copied and expanded in the H and W dimensions, that is, copied H×W times to obtain a feature map of H×W×C. The obtained feature map is added to the corresponding element of the input content of the network module, and the result is used as the element value of the corresponding position in the first fusion result of the network module. Each element in the first fusion result of the network module is processed to obtain the first fusion result of the network module. This completes the first fusion processing of the input content of the network module.

[0127] Then, the first fusion result of the network module is fused with the attention parameters and second mask features corresponding to the current training data to obtain the second fusion result of the network module. The output content of the network module is obtained based on the second fusion result of the network module. Since each encoding module in the diffusion network consists of one downsampling operation, several convolutions and several non-linear activation functions, and each decoding module consists of one upsampling operation, several convolutions and several non-linear activation functions, the corresponding network module will also perform corresponding upsampling or downsampling operations, convolution operations and non-linear activation function calculations on the second fusion result to obtain the output content of the corresponding network module. This will not be elaborated here.

[0128] The second mask feature is obtained by scaling the mask image using a scaling image function. In this embodiment, the scaling image function uses nearest neighbor interpolation and a scaling factor of 0.25, meaning that the length and width of the scaled mask image are both 0.25 times the original.

[0129] In some embodiments, the method for obtaining the attention parameter corresponding to the current training data includes: obtaining a fourth image feature corresponding to the current training data based on point cloud data, natural language data, or grayscale image in the current training data, that is, inputting the point cloud data, natural language data, or grayscale image in the current training data into the corresponding second feature extraction network, that is, inputting the point cloud data in the current training data into the second point cloud feature extraction network, or inputting the natural language data in the current training data into the second text feature extraction network, or inputting the grayscale image in the current training data into the second grayscale feature extraction network, and collectively referring to the global features of the point cloud, the global features of the grayscale image, or the global features of the natural language obtained therefrom as the fourth image feature; then obtaining a first general feature, which is a learnable parameter with the same length as the fourth image feature; and using either the obtained first general feature or the fourth image feature corresponding to the current training data as the attention parameter corresponding to the current training data.

[0130] It should be noted that the first image features obtained from the first image feature extraction network may not be complete. By combining additional information, namely the fourth image features, the integrity of the information can be guaranteed and the diversity of the information can be enhanced. For example, grayscale information can reflect texture information and guide the reconstruction of height information. The first general feature is a feature structure with strong feature extraction capabilities. Using it as an attention parameter can yield more stable and accurate noise prediction results.

[0131] Then, the process of performing a second fusion process on the first fusion result of the network module, the attention parameters corresponding to the current training data, and the second mask features includes: first, multiplying the attention parameters with the first fusion result of the network module to obtain an attention score result. For example, assuming the attention parameter is a 1×1×C vector, the size of the first fusion result of the network module is H×W×C, and the attention score result obtained after attention scoring is H×W×1; multiplying the attention parameters with the attention score result to obtain an attention convergence result, which is H×W×C; multiplying the second mask features with the attention convergence result to obtain a mask operation result, which is still H×W×C in size; adding the mask operation result to the corresponding elements of the first fusion result of the network module to obtain the second fusion result of the network module; this completes the second fusion process of the first fusion result of the network module, thus completing the feature fusion within a single network module.

[0132] Furthermore, since the first diffusion network and the control branch network are connected in a skip connection in this embodiment, the input content of different network modules in the second diffusion network is also different, such as... Figure 5 As shown, for any network module in the second diffusion network, if that network module is the first coding module in the first diffusion network, the corresponding... Figure 5 In the "Diffusion Coding Module_1", the input of this network module is the first image feature after the noise addition operation corresponding to the current noise addition step; if this network module is the coding module in the first diffusion network but not the first coding module, for example... Figure 5 In the context of "diffusion coding module_3", or if this network module is an intermediate module in the first diffusion network, the corresponding... Figure 5 In the example of "Diffusion Intermediate Module_4", the input content of this network module is the output content of the previous encoding module in the first diffusion network. That is to say, the input content of "Diffusion Encoding Module_3" is the output content corresponding to "Diffusion Encoding Module_2"; and the input content of "Diffusion Intermediate Module_4" is the output content corresponding to "Diffusion Encoding Module_4".

[0133] If this network module is the first encoding module in the control branch network, then... Figure 5 The “control coding module_1” in the network module is the first mask feature as its input. It should be noted that since there is a zero-initialized convolutional layer before the first coding module of the control branch network, the input of this network module actually refers to the first mask feature after convolution with the zero-initialized convolutional layer.

[0134] If the network module is an encoding module in the control branch network and is not the first encoding module, for example... Figure 5In the context of "control coding module_3", or if this network module is an intermediate module in the control branch network, the corresponding... Figure 5 If "Control Intermediate Module_4" is selected, then the input content of this network module is the output content of the previous encoding module in the control branch network; that is, the input content of "Control Encoding Module_3" is the output content corresponding to "Control Encoding Module_2"; and the input content of "Control Intermediate Module_4" is the output content corresponding to "Control Encoding Module_4".

[0135] If this network module is the first decoding module in the first diffusion network, corresponding to Figure 5 In the context of "diffusion decoding module_4", the previous decoding module of this network module refers to an intermediate module of the first diffusion network; if this network module is a decoding module of the first diffusion network but not the first decoding module, then... Figure 5 Taking "Diffusion Decoding Module_3" as an example, we obtain the previous decoding module of this network module in the first diffusion network, that is... Figure 5 The “diffusion decoding module_4” in the control branch network is convolved with a zero-initialized convolutional layer to obtain the branch fusion result. The obtained branch fusion result, the output of the previous decoding module, and the output of the encoding module with the same size as the previous decoding module in the first diffusion network are then fused together, and the obtained fusion result is used as the input of the network module.

[0136] The specific process for obtaining the input content of "Diffusion Decoding Module_3" is as follows: First, the output content of "Control Encoding Module_3" is convolved with a zero-initialized convolutional layer to obtain the branch fusion result. Then, the obtained branch fusion result is fused with the output content of "Diffusion Decoding Module_4" and the output content of "Diffusion Encoding Module_4". The fusion result obtained is used as the input content of "Diffusion Decoding Module_3", and so on.

[0137] It should be noted that when the network module is the first decoding module "Diffusion Decoding Module_4" in the first diffusion network, the previous decoding module of the network module refers to the intermediate module "Diffusion Intermediate Module_4" in the first diffusion network. Since the intermediate module does not have a corresponding encoding module, the input content acquisition process of the first decoding module in the first diffusion network does not require the "output content of the encoding module with the same size as the previous decoding module in the first diffusion network" mentioned above to participate in the fusion. That is, for the first decoding module in the first diffusion network, the input content acquisition process of the network module is as follows: first, the output content of "Control Intermediate Module_4" is convolved with a zero-initialized convolutional layer to obtain the branch fusion result, and then the obtained branch fusion result is fused with the output content of "Diffusion Intermediate Module_4". The fusion result obtained is used as the input content of "Diffusion Decoding Module_4".

[0138] The second diffusion network is used to predict noise from the input of the second diffusion network and output high-dimensional noise when the current noise addition step is equal to the preset total noise addition step in the current training round.

[0139] It should be noted that during the training of the second diffusion network, the attention parameter used by the first diffusion network is the fourth image feature. The attention parameter in the control branch network can be either the fourth image feature or the first general feature, and can be set according to the actual situation.

[0140] In training judgment step S124, if the current noise addition step is less than the preset total noise addition step for the current training round, then the current noise addition step is increased by 1 step to update the current noise addition step, and training continues from noise addition step S122; if the current noise addition step is equal to the preset total noise addition step for the current training round, then the high-dimensional noise output by the second diffusion network is used as the second high-dimensional noise; the loss function of the second diffusion network is obtained based on the first high-dimensional noise and the second high-dimensional noise, and backpropagation is performed based on the loss function to update the weight parameters in the second diffusion network. At this time, since the weight parameters in the first diffusion network are fixed, the weight parameters updated during training include the weight parameters in the control branch network and some learnable parameters, such as the first general feature;

[0141] The loss function of the second diffusion network can use loss functions that represent the similarity of distributions, such as cross-entropy and mean squared error. In this embodiment, the loss function used is the cross-entropy loss function, which is obtained as follows: For any element value of the second high-dimensional noise, calculate the logarithmic function value of that element value, multiply the obtained logarithmic function value by the element value of the first high-dimensional noise at the corresponding position, and obtain the multiplication result corresponding to that element value; obtain the sum of the multiplication results corresponding to all element values ​​of the second high-dimensional noise, and use the negative of the obtained sum as the loss function of the second diffusion network. Then, the loss function of the second diffusion network can be expressed as:

[0142]

[0143] Where, ε 1a ε represents the value of the a-th element of the first high-dimensional noise; 2a is the value of the a-th element of the second high-dimensional noise; m is the total number of elements contained in the first and second high-dimensional noises;

[0144] It should be noted that the initial first image feature is obtained by the first image feature extraction network after extracting features from the input data. Therefore, each pixel in the first image feature corresponds to a high-dimensional feature vector, and the first high-dimensional noise will apply a noise perturbation value to each element at each position in the first image feature. The noise perturbation values ​​corresponding to different positions follow a standard Gaussian distribution.

[0145] If the loss function converges, training stops, resulting in a trained second diffusion network. If the loss function does not converge, training continues from data processing step S121 using the next set of training data in the training set. If the current training data is the last set of training data in the training set, training for the current training round stops. After stopping the current training round, the second diffusion network used in the current training round is used as the second diffusion network for the next training round. Then, training for the next training round starts from data processing step S121 using the first set of training data in the training set. Furthermore, before stopping the current training round and starting the next training round, it is determined whether the preset maximum number of training rounds has been reached. If the maximum number of training rounds has been reached, training stops, and the obtained second diffusion network is adjusted. For example, if the second diffusion network fails to converge, it may be due to unreasonable network parameter settings. In this case, the network parameters of the second diffusion network need to be adjusted, and then the adjusted second diffusion network is retrained for multiple rounds based on the training set. The maximum number of training rounds can be set according to actual needs; in this embodiment, it can be set to 10,000 times.

[0146] This embodiment first overlays grayscale and height maps to extract the first image feature. Then, it extracts the first and second mask features based on the mask image. Finally, it extracts different types of global features (the fourth image feature) of the object based on the object's point cloud, grayscale image, and natural language. Through contrastive learning, the global features are made to include not only basic information about the object itself but also relevant information between objects. Finally, the second diffusion network, composed of the first diffusion network and the control branch network, is trained based on the first image feature, the first mask feature, the second mask feature, and the fourth image feature or the first general feature. This is achieved by fixing the parameters within the first diffusion network and training only the parameters in the control branch network and some learnable parameters. The training parameters are updated to reduce the number of parameters within the network during training, thereby reducing the training difficulty and enabling precise fine-tuning without changing the original diffusion model. At the same time, the control branch network provides additional information to the first diffusion network, thereby improving the phenomenon that the first diffusion network is unable to achieve pixel-level image generation. Furthermore, since attention parameters are fused in different network modules of the second diffusion network, by setting the attention parameters, the resulting second diffusion network can complete noise prediction tasks under conditions (i.e., with the participation of additional information such as point clouds) or without conditions (i.e., without the participation of additional information such as point clouds), so as to denoise the noisy image and generate a high-quality image.

[0147] Please refer to Figure 2 Some embodiments disclose a noise prediction method, which includes the following steps:

[0148] Step S200: Obtain the second diffusion network trained by the training method.

[0149] In some embodiments, the training method mentioned in step S200 may refer to a training method for noise prediction mentioned above, such as the second diffusion network obtained through steps S100 to S120.

[0150] Step S210: Obtain the first mask features and the first image features.

[0151] In some embodiments, the first mask feature can be obtained by inputting a mask image into a first mask feature extraction network. The mask image can be set according to actual needs or randomly selected from a mask image dataset.

[0152] In some embodiments, the first image feature can be obtained by inputting an image pair consisting of a grayscale image and a height map into a first image feature extraction network. The image pair can be set according to actual needs or randomly selected from a dataset consisting of training image pairs.

[0153] Step S220: Gradually add noise to the obtained first image features. When the number of noise addition steps reaches the preset total number of noise addition steps, the noise-added first image features are obtained.

[0154] The first image feature is progressively denoised based on random high-dimensional noise, where the random high-dimensional noise is standard high-dimensional noise. A preset total number of denoising steps is set, for example, 20 steps, which can be set as needed. When the preset total number of denoising steps is reached, denoising stops, indicating that the denoising process of the first image feature is complete, and the denoised first image feature is obtained. For example, the denoising process of progressively denoising the first image feature based on random high-dimensional noise is as follows:

[0155] If the current noise addition step is the first noise addition step, then random high-dimensional noise is used to add noise to the first image feature to obtain the first image feature after the noise addition operation corresponding to the first noise addition step; if the current noise addition step is not the first noise addition step, random high-dimensional noise is used to add noise to the first image feature after the noise addition operation corresponding to the previous noise addition step to obtain the first image feature after the noise addition operation corresponding to the current noise addition step.

[0156] For example, let the current noise addition step number be t. Based on the random high-dimensional noise and the first image feature after the noise addition operation corresponding to the previous noise addition step number t-1, the first image feature after the noise addition operation corresponding to the current noise addition step number is obtained. The specific process is as follows: the noise intensity of the random high-dimensional noise corresponding to the t-th noise addition step number is obtained based on the preset intensity of the random high-dimensional noise corresponding to the first noise addition step number and the preset intensity of the random high-dimensional noise corresponding to the last noise addition step number; then the noise intensity of the random high-dimensional noise corresponding to the t-th noise addition step number under the square root is taken as the random high-dimensional noise corresponding to the t-th noise addition step number. The reference weights for noise are calculated as follows: The difference between the noise intensity of step 1 and the random high-dimensional noise corresponding to the t-th noise addition step is used as the reference weight of the first image feature after noise addition at the (t-1)-th noise addition step. The first image feature after noise addition at the (t-1)-th noise addition step and the random high-dimensional noise corresponding to the t-th noise addition step are multiplied by their respective reference weights, and the result of the multiplication is used as the first image feature after noise addition at the t-th noise addition step. Therefore, the first image feature after noise addition at the t-th noise addition step can be expressed as:

[0157]

[0158] Among them, y t This indicates that the current noise-adding step corresponds to the first image feature after the noise-adding operation, that is, the t-th noise-adding step corresponds to the first image feature after the noise-adding operation; y t-1This represents the first image feature after the noise addition operation corresponding to the previous noise addition step, which is the first image feature after the noise addition operation corresponding to the (t-1)th noise addition step; ε0 represents random high-dimensional noise; ω t Let t be the noise intensity of the random high-dimensional noise during the noise addition operation corresponding to the current noise addition step number t;

[0159] The noise intensity of the random high-dimensional noise corresponding to the current t-th noise addition step is obtained based on the preset intensity of the random high-dimensional noise during the first noise addition and the preset intensity of the last noise addition. Specifically, the process is as follows: Calculate the squared difference between the preset intensity of the random high-dimensional noise corresponding to the first noise addition step and the preset intensity of the random high-dimensional noise corresponding to the last noise addition step, obtaining the difference between the preset total number of noise addition steps in the current training round and 1; use the ratio of the obtained squared difference to the obtained difference as the noise intensity change for a single noise addition; multiply the difference between the current noise addition step and 1 by the noise intensity change for a single noise addition, and use the product as the cumulative noise intensity change corresponding to the current noise addition step; add the cumulative noise intensity change corresponding to the current noise addition step to the square of the preset intensity of the random high-dimensional noise corresponding to the first noise addition step, and then use the square root of the sum as the noise intensity of the random high-dimensional noise corresponding to the t-th noise addition step. Therefore, the noise intensity of the random high-dimensional noise corresponding to the t-th noise addition step can be expressed as:

[0160]

[0161] Where, ω t Let ω be the noise intensity of the random high-dimensional noise corresponding to the current t-th noise addition step; ω1 is the preset intensity of the random high-dimensional noise added during the 1st noise addition step, i.e., the preset intensity of the random high-dimensional noise during the first noise addition; ω Q The preset intensity of the random high-dimensional noise added during the noise addition operation corresponding to the Qth noise addition step is the preset intensity of the random high-dimensional noise during the last noise addition; Q is the preset total number of noise addition steps.

[0162] Step S230: Input the first image features after noise addition and the first mask features into the second diffusion network, and perform fusion processing on the first mask features and the first image features after noise addition in the second diffusion network.

[0163] In some embodiments, the fusion process includes a first fusion process and a second fusion process, wherein:

[0164] First, the current noise step count vector is obtained based on the current noise step count. Since the first image feature after noise addition is the first image feature after the noise addition steps reach the preset total noise addition steps, the noise addition steps at this time are equal to the preset total noise addition steps. In addition, if the second diffusion network is used to perform denoising operation on the noise-added first image feature, the corresponding noise addition steps can be changed according to the change in the denoising steps, which will in turn change the current noise step count vector.

[0165] Then, for any network module in the second diffusion network, the input content of the network module is first fused with the current noise step vector to obtain the first fusion result of the network module; then the first fusion result of the network module is fused with the corresponding attention parameters and the second mask features to obtain the second fusion result of the network module, and the second fusion result of the network module is used as the output content of the network module.

[0166] In some embodiments, the second mask feature is obtained by scaling the mask image using a scaling image function;

[0167] In some embodiments, the fourth image feature is obtained based on the point cloud data, natural language data, or grayscale image of the corresponding object used when the first image feature was acquired. That is, the obtained point cloud data, natural language data, or grayscale image is input into the corresponding second feature extraction network, and the global feature output by the corresponding second feature extraction network is used as the fourth image feature.

[0168] Obtain a first general feature, for example, the first general feature may be obtained during the training of the second diffusion network; use the obtained first general feature and any one of the fourth image features as attention parameters; perform a second fusion process on the first fusion result of the network module, the attention parameters and the second mask features, and then obtain the output content of a single network module, thus completing the feature fusion within a single network module;

[0169] Furthermore, due to the skip connections between the first diffusion network and the control branch network, the outputs of different network modules in the second diffusion network will be merged, which will not be elaborated here.

[0170] It should be noted that since the noise step vector, attention parameters, and second mask features are used in the fusion process of each network module, the input of the second diffusion network as a whole includes the first mask features, the first image features after noise addition, the noise step vector, attention parameters, and the second mask features.

[0171] Step S240: Use the output of the second diffusion network as the predicted high-dimensional noise.

[0172] This embodiment first obtains a first mask feature and a first image feature; then, the obtained first image feature is progressively denoised to obtain a denoised first image feature; next, the denoised first image feature and the first mask feature are input into a second diffusion network, and the first mask feature and the denoised first image feature are fused in the second diffusion network; finally, the output of the second diffusion network is used as the predicted high-dimensional noise; in this process, a second fusion process is performed with attention parameters in different network modules, and by setting the attention parameters, noise prediction results can be obtained under conditions (i.e., when additional information such as point clouds are involved) or unconditions (i.e., when additional information such as point clouds is not involved).

[0173] Please refer to Figure 3 Some embodiments disclose a method for generating defect images, which includes the following steps:

[0174] Step S300: Obtain the second diffusion network trained by the training method.

[0175] In some embodiments, the training method mentioned in step S300 may refer to a training method for noise prediction mentioned above, such as the second diffusion network obtained through steps S100 to S120.

[0176] Step S310: Obtain the first mask features and the second mask features based on the mask image.

[0177] In some embodiments, a mask image is obtained, which can be set according to actual needs or randomly selected from a mask image dataset;

[0178] The mask image is input into the first mask feature extraction network to obtain the first mask feature; the mask image is scaled by the scaling image function to obtain the second mask feature.

[0179] Step S320: Obtain new first image features based on the second mask features.

[0180] First, the comprehensive feature vector is obtained, which is obtained during the training process of the second diffusion network through the training method.

[0181] In some embodiments, the training method mentioned in step S320 may refer to a training method for noise prediction mentioned above, such as the second diffusion network obtained through steps S100 to S120.

[0182] In some embodiments, in any training round of the second diffusion network, the first feature vector is iteratively updated once in the data processing step of each training data set. In any training round, for the current training data, the background region and foreground region of the initial first image feature are obtained in the data processing step based on the initial first image feature corresponding to the current training data and the mask image in the current training data.

[0183] In some embodiments, if the pixel value of a pixel in the mask image of the current training data is 1, then the corresponding pixel in the first image feature is considered to belong to the foreground region; if the pixel value of a pixel in the mask image of the current training data is 0, then the corresponding pixel in the first image feature is considered to belong to the background region.

[0184] The first feature vector corresponding to the current training data is obtained based on the background and foreground regions of the initial first image features;

[0185] In some embodiments, the first feature vector corresponding to the current training data includes four types of vectors: the background mean vector, the background standard deviation vector, the foreground mean vector, and the foreground standard deviation vector of the initial first image feature. Specifically, for the initial first image feature corresponding to the current training data, its background mean vector is the mean vector of the high-dimensional feature vectors corresponding to all pixels in the background region of the initial first image feature; its background standard deviation vector is the standard deviation vector of the high-dimensional feature vectors corresponding to all pixels in the background region of the initial first image feature; its foreground mean vector is the mean vector of the high-dimensional feature vectors corresponding to all pixels in the foreground region of the initial first image feature; and its foreground standard deviation vector is the standard deviation vector of the high-dimensional feature vectors corresponding to all pixels in the foreground region of the initial first image feature.

[0186] Based on the first feature vector corresponding to the current training data and the first feature vector after the last iteration, the first feature vector after the last iteration is obtained; where, if the current training data is the first set of training data in the training set, the first feature vector after the last iteration is a 0 vector.

[0187] In the g-th iteration update, for any first feature vector, a momentum coefficient is first preset and used as the reference weight for any first feature vector after the g-1-th iteration update. The difference between 1 and the momentum coefficient is used as the reference weight for the first feature vector corresponding to the g-th training data. The product between any first feature vector obtained after the g-1-th iteration update, the first feature vector corresponding to the g-th training data, and the corresponding reference weight is calculated respectively. The sum of the two products is used as any first feature vector after the g-th iteration update.

[0188] Taking the background mean vector as an example, let the momentum coefficient ρ. Use the momentum coefficient ρ as the reference weight for the background mean vector after the (g-1)th iteration update. Use the difference between 1 and the momentum coefficient, i.e., 1-ρ, as the reference weight for the background mean vector corresponding to the g-th training data. Calculate the product between the background mean vector obtained after the (g-1)th iteration update, the background mean vector corresponding to the g-th training data, and the corresponding reference weight. Sum the two products as the background mean vector after the g-th iteration update. Then, the background mean vector after the g-th iteration update can be expressed as:

[0189] μ g =ρ×μ g-1 +(1-ρ)×c g

[0190] Where, μ g μ represents the background mean vector after the g-th iteration update; g-1 Let ρ represent the background mean vector after the (g-1)th iteration update; ρ is the momentum coefficient; c g Let represent the background mean vector corresponding to the g-th training data set;

[0191] The same processing method is applied to the other three first eigenvectors, namely, the processing method for the background standard deviation vector, the foreground mean vector, and the foreground standard deviation vector is the same as the processing method for the background mean vector.

[0192] The first feature vector updated in the last iteration of a training round of the second diffusion network is used as the comprehensive feature vector.

[0193] It should be noted that, since all training data in the entire training set needs to be processed sequentially in any training round, and each processing of a set of training data will yield the corresponding first feature vector, and then the first feature vector will be updated, the comprehensive feature vector is actually only related to the training data in each update during the acquisition process. Therefore, the comprehensive feature vector obtained in different training rounds is the same.

[0194] Then, for any pixel in the new first image feature, when the pixel value of the corresponding pixel in the second mask feature is 1, the foreground standard deviation vector in the comprehensive feature vector is multiplied by the element value of the corresponding dimension of the standard high-dimensional noise. The sum of the resulting vector and the foreground mean vector is used as the high-dimensional feature vector corresponding to that pixel in the new first image feature. When the pixel value of the corresponding pixel in the second mask feature is 0, the high-dimensional feature vector corresponding to any pixel in the new first image feature is obtained according to the comprehensive feature vector or by referring to the image feature.

[0195] Specifically, when the pixel value of the corresponding pixel in the second mask feature is 0, the background standard deviation vector in the comprehensive feature vector is multiplied by the element value of the corresponding dimension of the standard high-dimensional noise. The sum of the resulting vector and the background mean vector is used as the high-dimensional feature vector corresponding to any pixel in the new first image feature. Alternatively, a preset image pair is obtained, which can be set by the user. Then, reference image features are obtained based on the preset image pair, that is, the preset image pair is input into the first image feature extraction network, and the output of the first image feature extraction network is called reference image features. Then, the high-dimensional feature vector corresponding to the corresponding pixel in the reference image features is used as the high-dimensional feature vector corresponding to any pixel in the new first image feature.

[0196] When a new first image feature is generated directly using a mask image and a first feature vector, the defect location can appear in different positions by selecting different mask images, thus greatly enriching the diversity of defect sample data. When a new first image feature is generated based on the preset image pair, mask image, and first feature vector, the final defect sample can have the background image that actually needs to be obtained. At this time, by selecting different mask images, the second mask feature can also be changed, thus changing the defect location. While enriching the defect sample data, it can also better meet the user's needs. In this way, different defect regions can be transferred to any background image and corresponding annotations can be generated, thereby synthesizing images and annotations for supervised learning. This greatly simplifies the process of collecting and annotating images, making deep learning projects easier to implement.

[0197] Step S330: Gradually add noise to the new first image feature according to the preset total number of noise addition steps to obtain the third image feature.

[0198] In some embodiments, the step of gradually adding noise to the new first image feature according to the preset total number of noise addition steps involved in step S330 can be referred to in step S220 above for obtaining the first image feature after noise addition, and the obtained first image feature after noise addition is called the third image feature. In this embodiment, the preset total number of noise addition steps is still denoted as T.

[0199] Step S340: Denoise the third image features based on the first mask features and the second diffusion network to obtain the first denoised feature map.

[0200] First, the first mask feature is input into the control branch network, and the third image feature is input into the first diffusion network. The second diffusion network then performs fusion processing on the input third image feature and the first mask feature. During this process, by changing the attention parameters used by each network module in the second diffusion network for the second fusion processing, high-dimensional noise is obtained for denoising the third image feature. The obtained high-dimensional noise is then used to denoise the third image feature. The first mask feature and the denoised third image feature are then re-input into the second diffusion network, and so on, to complete the gradual denoising of the third image feature until the number of denoising steps equals the preset total number of denoising steps in step S330. The data to be processed in the denoising operation corresponding to the first denoising step is the third image feature input into the first diffusion network, which is the third image feature obtained in step S330. The data to be processed in the denoising operations corresponding to other denoising steps is the third image feature after the previous denoising operation.

[0201] Taking the denoising operation corresponding to the t'th denoising step as an example, first obtain the denoising step vector corresponding to each denoising step. The denoising step vector corresponding to each denoising step is obtained by first obtaining the denoising step corresponding to that denoising step, and then obtaining the denoising step vector based on the obtained denoising step.

[0202] In some embodiments, the number of noise addition steps corresponding to the t'th denoising operation is equal to the total number of noise addition steps minus t' plus 1, that is, the number of noise addition steps q corresponding to the t'th denoising operation is T-t'+1; the corresponding noise addition step vector is obtained according to the number of noise addition steps corresponding to the t'th denoising operation.

[0203] The first mask feature and the third image feature after the previous denoising operation are input into the second diffusion network. The third image feature after the previous denoising operation and the first mask feature are fused in the second diffusion network. During the denoising process, for any network module in the second diffusion network, the input content of the network module is first fused with the denoising step vector corresponding to the t'th denoising operation to obtain the first fusion result of the network module. Then, the first fusion result of the network module is fused with the attention parameters and the second mask feature to obtain the second fusion result of the network module. The third image feature after the previous denoising operation is the input content of the first encoding module of the first diffusion network in the second diffusion network, and the first mask feature is the input content of the first encoding module of the control branch network in the second diffusion network. The input content of the first encoding module of the control branch network here also refers to the first mask feature after convolution with the zero-initialized convolutional layer.

[0204] Then, based on the output of the second diffusion network, the third and fourth high-dimensional noise corresponding to the denoising operation at the t'th time are obtained. First, the fourth image feature and the first general feature are obtained. The fourth image feature can be obtained from the point cloud data, natural language data, or grayscale image of the corresponding object used when obtaining the first image feature. That is, the obtained point cloud data, natural language data, or grayscale image is input into the corresponding second feature extraction network, and the global feature output by the corresponding second feature extraction network is used as the fourth image feature. The first general feature can be obtained during the training of the second diffusion network.

[0205] For the denoising operation t', the attention parameter used by any network module in the second diffusion network during the second fusion process is set to the fourth image feature. The third image feature and the first mask feature after the previous denoising operation are input into the second diffusion network. The high-dimensional noise predicted by the second diffusion network is recorded as the third high-dimensional noise corresponding to the denoising operation t'. Then, the attention parameter used by any network module in the second diffusion network during the second fusion process is set to the first general feature. The third image feature and the first mask feature after the previous denoising operation are input into the second diffusion network again. The high-dimensional noise predicted by the second diffusion network is recorded as the fourth high-dimensional noise corresponding to the denoising operation t'.

[0206] The fifth high-dimensional noise corresponding to the t'th denoising operation is obtained from the third high-dimensional noise and the fourth high-dimensional noise corresponding to the t'th denoising operation. The difference between the element values ​​at the same position of the third high-dimensional noise and the fourth high-dimensional noise is taken as the element value of the fifth high-dimensional noise at the corresponding position.

[0207] Then, based on the fifth high-dimensional noise corresponding to the denoising operation at the t'th denoising operation, the third image features after the previous denoising operation are denoised to obtain the third image features after the t'th denoising operation. The denoising operation can be viewed as solving a stochastic differential equation, and various stochastic differential equation solving methods can be used, such as Euler sampling and Heun sampling. In this embodiment, Euler sampling is used for denoising. Therefore, the method for obtaining the third image features after the t'th denoising operation is as follows:

[0208]

[0209] Where q is the noise addition step corresponding to the t'th denoising operation, i.e., q = T - t' + 1, meaning there is a one-to-one correspondence between q and t'; F represents the third image feature, F(q) represents the third image feature after the previous denoising operation, i.e., the third image feature after the t'-1th denoising operation; F(q-1) represents the third image feature after the t'th denoising operation; a qε is the denoising intensity of the t'th denoising operation; ε5(q) represents the fifth high-dimensional noise corresponding to the t'th denoising operation; ε(q) is the standard high-dimensional noise corresponding to the t'th denoising operation, i.e., ε(q) ~ N(0,I), where I represents the identity matrix, and each dimension of it follows a standard Gaussian distribution with a mean of 0 and a variance of 1; p is a preset value used to represent the randomness of the stochastic differential equation solution process. This value can be set according to actual needs, and in this embodiment it is set to 1. This is the product of the denoising intensities from the 0th denoising operation to the current t'th denoising operation, i.e.

[0210] When t' = T, that is, when the total number of denoising operations is equal to the preset total number of noise addition steps in step S330, denoising is stopped after the current denoising operation is completed, and the third image feature obtained after the last denoising operation is recorded as the first denoised image feature.

[0211] Step S350: Obtain the target image based on the first denoised feature map.

[0212] In some embodiments, the first denoised feature map is input into the first feature decoding network, and the output of the first feature decoding network is the target image.

[0213] In this embodiment, the system first selects whether to require the defect sample data to have a specified background area to generate new first image features based on actual needs. This enriches the defect sample data and better meets user needs, making the final target image more diverse. In addition, since the second diffusion network used in the target image acquisition process can input additional information such as point clouds, and the mask image contains rich positional information, it can enhance the diversity of information while ensuring the integrity of the information, thereby enhancing the generation effect of the target image and achieving the generation of high-quality target images under pixel-level precision control.

[0214] Those skilled in the art will understand that all or part of the functions of the various methods in the above embodiments can be implemented by hardware or by computer programs. When all or part of the functions in the above embodiments are implemented by computer programs, the program can be stored in a computer-readable storage medium, which may include: read-only memory, random access memory, disk, optical disk, hard disk, etc., and the program is executed by a computer to achieve the above functions. For example, the program can be stored in the memory of a device, and when the program in the memory is executed by the processor, all or part of the above functions can be achieved. In addition, when all or part of the functions in the above embodiments are implemented by computer programs, the program can also be stored in a server, another computer, disk, optical disk, flash drive, or external hard drive, etc., and can be downloaded or copied to the memory of a local device, or the system of the local device can be updated. When the program in the memory is executed by the processor, all or part of the functions in the above embodiments can be achieved.

[0215] The above examples illustrate the present invention only to aid in understanding it and are not intended to limit the scope of the invention. Those skilled in the art can make various simple deductions, modifications, or substitutions based on the principles of this invention.

Claims

1. A training method for noise prediction, characterized in that, include: Obtain a second diffusion network formed by adding a control branch network to a pre-trained first diffusion network, wherein the first diffusion network is a preset diffusion network; The second diffusion network is trained in one or more rounds based on the training set, wherein the training set includes one or more sets of training data, and any set of training data includes image pairs composed of grayscale images and height maps, and mask images; During any training epoch, the second diffusion network is trained using any set of training data from the training set, including: The data processing steps are as follows: In the current training round, the initial first image features are obtained based on the image pairs in the current training data, and the first mask features are obtained based on the mask image in the current training data. In the noise addition step, a first high-dimensional noise is obtained; if the current noise addition step is the first noise addition step, the initial first image features are noise-added according to the first high-dimensional noise to obtain the first image features after the noise addition operation corresponding to the first noise addition step; if the current noise addition step is not the first noise addition step, the first image features after the noise addition operation corresponding to the previous noise addition step are noise-added again according to the first high-dimensional noise to obtain the first image features after the noise addition operation corresponding to the current noise addition step; wherein, for the first... t Number of noise-adding steps, t The integer is a positive integer. The first high-dimensional noise level is obtained based on the preset intensity of the first high-dimensional noise corresponding to the first noise-adding step and the preset intensity of the first high-dimensional noise corresponding to the last noise-adding step. t Each noise addition step corresponds to the noise intensity of the first high-dimensional noise; The first under the square root sign t The noise intensity of the first high-dimensional noise corresponding to each noise addition step is used as the first... t Each noise addition step corresponds to a reference weight for the first high-dimensional noise; calculate the weights of step 1 and the first high-dimensional noise. t The difference between the noise intensities of the first high-dimensional noise corresponding to each noise addition step is taken as the second square root of the noise intensity. t -1 noise addition step corresponds to the reference weight of the first image feature after noise addition; the first step... t -1 noise addition step corresponds to the first image feature after noise addition operation, the first... t Each noise addition step corresponds to a first high-dimensional noise multiplied by its corresponding reference weight. The result of this multiplication is used as the first... t Each noise-adding step corresponds to the first image feature after the noise-adding operation; The fusion processing step involves inputting the first image feature after noise addition corresponding to the current noise addition step number and the first mask feature into the second diffusion network. In the second diffusion network, the first image feature after noise addition corresponding to the current noise addition step number and the first mask feature are fused respectively. The second diffusion network is used to perform noise prediction on the input of the second diffusion network to output high-dimensional noise when the current noise addition step number is equal to the preset total noise addition step number of the current training round. In the training judgment step, if the current noise addition step is less than the preset total noise addition step for the current training round, then the current noise addition step is increased by 1 step to update the current noise addition step, and training continues from the noise addition step; if the current noise addition step is equal to the preset total noise addition step for the current training round, then the high-dimensional noise output by the second diffusion network is used as the second high-dimensional noise; the loss function of the second diffusion network is obtained based on the first high-dimensional noise and the second high-dimensional noise; if the loss function converges, then training stops.

2. The method as described in claim 1, characterized in that, The training judgment step also includes: If the loss function does not converge, training continues from the data processing step based on the next set of training data in the training set; if the current training data is the last set of training data in the training set, training for the current training round is stopped.

3. The method as described in claim 2, characterized in that, The training judgment step also includes: After stopping the training of the current training round, the second diffusion network after the end of the current training round is used as the second diffusion network for the training process of the next training round, and the training of the next training round starts from the data processing step based on the first set of training data in the training set.

4. The method as described in claim 3, characterized in that, The training judgment step also includes: Before stopping the current training round and starting the next training round, it also checks whether the preset maximum number of training rounds has been reached. If it has, then training stops.

5. The method as described in claim 1, characterized in that, The first high-dimensional noise is obtained based on the preset intensity of the first high-dimensional noise corresponding to the first noise addition step and the preset intensity of the first high-dimensional noise corresponding to the last noise addition step. t Each noise addition step corresponds to the noise intensity of the first high-dimensional noise, including: Calculate the squared difference between the preset intensity of the first high-dimensional noise corresponding to the first noise addition step and the preset intensity of the first high-dimensional noise corresponding to the last noise addition step, and obtain the difference between the preset total number of noise addition steps in the current training round and 1; use the ratio between the obtained squared difference and the obtained difference as the noise intensity change of a single noise addition. Multiply the difference between the current noise addition step number and 1 by the noise intensity change of a single noise addition, and use the product as the cumulative noise intensity change corresponding to the current noise addition step number; add the cumulative noise intensity change corresponding to the current noise addition step number to the square of the preset intensity of the first high-dimensional noise corresponding to the first noise addition step number, and then use the square root of the result as the first... t Each noise addition step corresponds to the noise intensity of the first high-dimensional noise.

6. The method as described in claim 1, characterized in that, In the second diffusion network, the first image features after the noise addition operation corresponding to the current noise addition step and the first mask features are respectively fused, including: The fusion process includes a first fusion process and a second fusion process; The current noise step number vector is obtained based on the current noise step number; for any network module in the second diffusion network, the input content of the network module is first fused with the current noise step number vector to obtain the first fusion result of the network module; The training data set further includes point cloud data and / or natural language data of the corresponding objects of the image pair. The fourth image feature corresponding to the current training data is obtained based on the point cloud data, natural language data, or grayscale image in the current training data. The first general feature is obtained, and either the first general feature or the fourth image feature corresponding to the current training data is used as the attention parameter corresponding to the current training data. The second mask feature corresponding to the current training data can be obtained based on the mask image in the current training data. The first fusion result of any one of the network modules is combined with the attention parameters and second mask features corresponding to the current training data to perform a second fusion process, thereby obtaining the second fusion result of the network module. The output content of the network module is obtained based on the second fusion result of the network module.

7. The method as described in claim 6, characterized in that, The input content of any network module includes: If any of the network modules is the first encoding module in the first diffusion network, the input of any of the network modules is the first image feature after the noise addition operation corresponding to the current noise addition step; if any of the network modules is an encoding module in the first diffusion network but is not the first encoding module, or if any of the network modules is an intermediate module in the first diffusion network, the input of any of the network modules is the output of the previous encoding module in the first diffusion network; If any of the network modules is the first encoding module in the control branch network, the input content of the network module is the first mask feature; if any of the network modules is an encoding module in the control branch network but is not the first encoding module, or if any of the network modules is an intermediate module in the control branch network, the input content of the network module is the output content of the previous encoding module in the control branch network. If any of the network modules is the first decoding module in the first diffusion network, then the previous decoding module of any of the network modules is an intermediate module; if any of the network modules is a decoding module of the first diffusion network but is not the first decoding module, then the previous decoding module of any of the network modules in the first diffusion network is obtained; the output of the encoding module in the control branch network with the same size as the previous decoding module is convolved with a convolutional layer to obtain a branch fusion result; then the branch fusion result, the output of the previous decoding module, and the output of the encoding module in the first diffusion network with the same size as the previous decoding module are fused, and the obtained fusion result is used as the input of any of the network modules.

8. The method as described in claim 6, characterized in that, The step of obtaining the current noise step vector based on the current noise step number includes: Set the preset length of the noise-adding step vector, and obtain the logarithmic function value corresponding to the current noise-adding step; for the th step in the noise-adding step vector... k 1 element, calculate k The ratio between -1 and the preset length; calculate the product between the obtained logarithmic function value and the obtained ratio, and take the exponential function value corresponding to the negative of the obtained product as the number of steps in the current noise-adding vector. k Each element value.

9. The method as described in claim 1, characterized in that, The step of obtaining the loss function of the second diffusion network based on the first high-dimensional noise and the second high-dimensional noise includes: For any element value of the second high-dimensional noise, calculate the logarithmic function value of the arbitrary element value, multiply the obtained logarithmic function value by the element value of the first high-dimensional noise at the corresponding position, and obtain the multiplication result corresponding to the arbitrary element value; obtain the sum of the multiplication results corresponding to all element values ​​of the second high-dimensional noise, and use the negative of the obtained sum as the loss function of the second diffusion network.

10. A noise prediction method, characterized in that, include: Obtain a second diffusion network trained by the training method described in any one of claims 1 to 9; The predicted high-dimensional noise obtained from the second diffusion network includes: Obtain the first mask features and the first image features; The first image feature is gradually denoised. When the number of denoising steps reaches the preset total number of denoising steps, the denoised first image feature is obtained. The denoised first image feature and the first mask feature are input into the second diffusion network. In the second diffusion network, the first mask feature and the denoised first image feature are fused respectively. The output of the second diffusion network is used as the predicted high-dimensional noise.

11. A method for generating defect images, characterized in that, include: Obtain a second diffusion network trained by the training method described in any one of claims 1 to 9; Obtain a mask image, and obtain a first mask feature and a second mask feature based on the mask image; A new first image feature is obtained based on the second mask feature; The new first image feature is gradually denoised according to the preset total number of denoising steps, and the first image feature after the last denoising step is recorded as the third image feature. Denoising is performed on the third image features based on the first mask features and the second diffusion network. When the number of denoising steps is equal to the preset total number of noise addition steps, denoising is stopped, and the first denoised feature map is obtained. The target image is obtained based on the first denoised feature map.

12. The method as described in claim 11, characterized in that, The step of obtaining new first image features based on the second mask features includes: Obtain the comprehensive feature vector; For any pixel in the new first image feature, when the pixel value of the corresponding pixel in the second mask feature is 1, the foreground standard deviation vector in the comprehensive feature vector is multiplied by the element value of the corresponding dimension of the standard high-dimensional noise, and the sum of the resulting vector and the foreground mean vector is used as the high-dimensional feature vector corresponding to any pixel in the new first image feature; when the pixel value of the corresponding pixel in the second mask feature is 0, the high-dimensional feature vector corresponding to any pixel in the new first image feature is obtained according to the comprehensive feature vector or by referring to the image feature.

13. The method as described in claim 12, characterized in that, The process of obtaining the comprehensive feature vector includes: In the training rounds, the first feature vector is iteratively updated once in the data processing step of each set of training data, wherein: For the current training data, in the data processing step, the background region and foreground region of the initial first image feature are obtained based on the initial first image feature corresponding to the current training data and the mask image in the current training data, and the first feature vector corresponding to the current training data is obtained based on the background region and foreground region of the initial first image feature. Based on the first feature vector corresponding to the current training data and the first feature vector after the last iteration, the first feature vector after the last iteration is obtained; where, if the current training data is the first set of training data in the training set, the first feature vector after the last iteration is a 0 vector. In any training round of the second diffusion network, the first feature vector updated in the last iteration is used as the comprehensive feature vector.

14. The method as described in claim 13, characterized in that, The step of obtaining the first feature vector corresponding to the current training data based on the background region and foreground region of the initial first image features includes: The first feature vector corresponding to the current training data includes four types of vectors: the background mean vector, background standard deviation vector, foreground mean vector, and foreground standard deviation vector of the initial first image feature. Specifically, the background mean vector of the initial first image feature is the mean vector of the high-dimensional feature vectors corresponding to all pixels in the background region of the initial first image feature; the background standard deviation vector of the initial first image feature is the standard deviation vector of the high-dimensional feature vectors corresponding to all pixels in the background region of the initial first image feature; the foreground mean vector of the initial first image feature is the mean vector of the high-dimensional feature vectors corresponding to all pixels in the foreground region of the initial first image feature; and the foreground standard deviation vector of the initial first image feature is the standard deviation vector of the high-dimensional feature vectors corresponding to all pixels in the foreground region of the initial first image feature.

15. The method as described in claim 13, characterized in that, The step of obtaining the first feature vector updated in the current iteration based on the first feature vector corresponding to the current training data and the first feature vector updated in the previous iteration includes: In the g-th iteration update of a training epoch, for any first feature vector, a momentum coefficient is set, and the momentum coefficient is used as the g-th iteration update. - The reference weight of any first feature vector after the first iteration update is obtained by taking the difference between 1 and the momentum coefficient as the reference weight of the first feature vector corresponding to the g-th training data. The product between the first feature vector obtained after the (g-1)-th iteration update, the first feature vector corresponding to the g-th training data, and the corresponding reference weight is calculated respectively, and the sum of the two products is taken as the first feature vector after the g-th iteration update.

16. The method as described in claim 12, characterized in that, When the pixel value of the corresponding pixel in the second mask feature is 0, the high-dimensional feature vector corresponding to any pixel in the new first image feature is obtained based on the comprehensive feature vector or by referring to the image features, including: For any pixel in the new first image feature, when the pixel value of the corresponding pixel in the second mask feature is 0, the background standard deviation vector in the comprehensive feature vector is multiplied by the element value of the corresponding dimension of the standard high-dimensional noise, and the sum of the resulting vector and the background mean vector is used as the high-dimensional feature vector corresponding to any pixel in the new first image feature; or, reference image features are obtained according to a preset image pair; the high-dimensional feature vector corresponding to the corresponding pixel in the reference image features is used as the high-dimensional feature vector corresponding to any pixel in the new first image feature.

17. The method as described in claim 11, characterized in that, The denoising operation on the third image features based on the first mask features and the second diffusion network includes: For the denoising operation t', the third image feature after the previous denoising operation and the first mask feature are input into the second diffusion network. The second diffusion network performs fusion processing on the third image feature after the previous denoising operation and the first mask feature respectively. The third high-dimensional noise and the fourth high-dimensional noise corresponding to the denoising operation t' are obtained according to the output of the second diffusion network. The fifth high-dimensional noise corresponding to the denoising operation t' is obtained according to the third high-dimensional noise and the fourth high-dimensional noise corresponding to the denoising operation t'. The third image feature after the previous denoising operation is denoised according to the fifth high-dimensional noise corresponding to the denoising operation t', to obtain the third image feature after the denoising operation t'.

18. The method as described in claim 17, characterized in that, The second diffusion network performs fusion processing on the third image features after the previous denoising operation and the first mask features, including: Obtain the noise step vector corresponding to the t'th denoising operation. For any network module in the second diffusion network, first perform a first fusion process with the noise step vector corresponding to the t'th denoising operation to obtain the first fusion result of the network module. Then, perform a second fusion process with the attention parameters and the second mask features to obtain the second fusion result of the network module. The third image feature after the previous denoising operation is the input content of the first encoding module of the first diffusion network in the second diffusion network, and the first mask feature is the input content of the first encoding module of the control branch network in the second diffusion network.

19. The method as described in claim 18, characterized in that, The step of obtaining the noise step vector corresponding to the t'th denoising operation includes: subtracting t' from the preset total noise step and then adding 1 as the noise step number corresponding to the t'th denoising operation; and using the noise step vector corresponding to the noise step number as the noise step vector corresponding to the t'th denoising operation.

20. The method as described in claim 17, characterized in that, The step of obtaining the third and fourth high-dimensional noise corresponding to the denoising operation at the t'th time based on the output of the second diffusion network includes: Obtain the fourth image feature and the first general feature; for the denoising operation t', set the attention parameter used by any network module in the second diffusion network when performing the second fusion process as the fourth image feature, input the third image feature after the previous denoising operation and the first mask feature into the second diffusion network, and record the high-dimensional noise predicted by the second diffusion network as the third high-dimensional noise corresponding to the denoising operation t'; then set the attention parameter used by any network module in the second diffusion network when performing the second fusion process as the first general feature, and input the third image feature after the previous denoising operation and the first mask feature into the second diffusion network again, and record the high-dimensional noise predicted by the second diffusion network as the fourth high-dimensional noise corresponding to the denoising operation t'.

21. A computer-readable storage medium, characterized in that, The medium stores a computer program that can be executed by a processor to implement the method as described in any one of claims 1-20.

Images