Data denoising and enhancement collaborative representation learning method

Abstract

The application provides a data denoising and enhancement collaborative representation learning method, relates to the technical field of data processing, and aims to collaboratively represent natural noise and adversarial noise to improve the accuracy of a task model. The method comprises the following steps: performing natural noise denoising processing on an original image sample to obtain a denoised image sample, wherein the natural noise represents noise generated by natural factors that affects the accuracy of a task model; adding adversarial noise to the denoised image sample to generate an adversarial noise image sample, wherein the adversarial noise represents noise used to deceive a task model to output an incorrect result; purifying noise in the adversarial noise image sample based on a diffusion model to obtain a data enhancement image sample; and training a task model based on the data enhancement image sample and the denoised image sample until the task model converges.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a data denoising and enhancement collaborative representation learning method. BACKGROUND

[0002] In real life, the digitization and transmission of images are often affected by imaging devices and external environmental noise interference, resulting in a decline in image quality, and a large amount of natural noise and adversarial noise in the image, thereby affecting the accuracy of the task result (for example, a target detection task).

[0003] In related technologies, only one aspect of natural noise and adversarial noise is often optimized. Although the existing method for removing natural noise can effectively remove natural noise, it performs poorly when facing adversarial noise, and the effect decreases a lot when disturbed or attacked. The existing method for adversarial noise only focuses on learning adversarial noise, although it performs better when facing attacks, it sacrifices a certain task accuracy, resulting in an increase in error rate and reducing the overall performance. SUMMARY

[0004] In view of the above problems, the embodiments of the present application provide a data denoising and enhancement collaborative representation learning method in order to overcome the above problems or at least partially solve the above problems.

[0005] The first aspect of the embodiments of the present application discloses a data denoising and enhancement collaborative representation learning method, which comprises:

[0006] Performing natural noise denoising processing on the original image sample to obtain a denoised image sample, wherein the natural noise represents noise that affects the accuracy of a task model due to natural factors;

[0007] Adding adversarial noise to the denoised image sample to generate an adversarial noise image sample, wherein the adversarial noise represents noise used to deceive the task model to output an incorrect result;

[0008] Purifying the noise in the adversarial noise image sample based on a diffusion model to obtain a data enhancement image sample;

[0009] Training the task model based on the data enhancement image sample and the denoised image sample until the task model converges.

[0010] Optionally, training the task model based on the data enhancement image sample and the denoised image sample until the task model converges, comprising:

[0011] Fusing the data enhancement image sample and the denoised image sample to obtain a fused image sample;

[0012] inputting the fusion image sample into the task model to obtain a task processing result;

[0013] obtaining a first loss according to a difference between the task processing result and a true label corresponding to the original image sample;

[0014] obtaining a second loss according to a difference between the denoised image sample and the fusion image sample, and obtaining a third loss according to a difference between the data enhanced image sample and the fusion image sample;

[0015] updating model parameters of the task model according to a total loss obtained according to the first loss, the second loss and the third loss.

[0016] Optionally, the original image sample is subjected to natural noise denoising processing to obtain a denoised image sample, including:

[0017] obtaining input representations of a plurality of attention heads according to the original image sample;

[0018] performing multi-head squeezing operation on the input representations of the plurality of attention heads to obtain channel features of the plurality of attention heads;

[0019] performing activation operation on the channel features of the plurality of attention heads to obtain activation operation results of the plurality of attention heads, the activation operation being to express and extract nonlinear relationships of the channel features of the plurality of attention heads;

[0020] fusing the channel features of the plurality of attention heads and the activation operation results of the plurality of attention heads, and obtaining natural noise according to a fusion result;

[0021] removing the natural noise from the original image sample to obtain a denoised sample.

[0022] Optionally, the noise in the adversarial noise image sample is purified based on a diffusion model to obtain a data enhanced image sample, including:

[0023] at each time step, performing noise adding processing on the adversarial noise image sample according to randomly sampled Gaussian noise to obtain a noise added image sample, and predicting an input image sample of a current time step according to the noise added image sample of each time step, repeating a plurality of time steps until an input image sample of a last time step is obtained;

[0024] iterating a plurality of times in the above manner to obtain a trained diffusion model;

[0025] The trained diffusion model is used to perform noise adding and noise removing on the adversarial noise image sample to obtain a data enhanced image sample.

[0026] Optionally, at each time step, the adversarial noise image sample is subjected to noise adding according to randomly sampled Gaussian noise to obtain a noise added image sample, and an input image sample of the current time step is predicted according to the noise added image sample of each time step, and the multiple time steps are repeated until an input image sample of the last time step is obtained, comprising:

[0027] Gaussian noise is randomly sampled from a Gaussian distribution;

[0028] The Gaussian noise is added to the input image sample of the previous time step and the adversarial noise image sample of the previous time step respectively to obtain an input image sample of the current time step and an adversarial noise image sample of the current time step;

[0029] The adversarial noise image sample of the current time step is input into a denoising module of the diffusion model to obtain a model denoised image sample;

[0030] A fourth loss is constructed according to the model denoised image sample and the input image sample of the current time step;

[0031] Based on the fourth loss, a gradient of the input image sample of the current time step is calculated, and the diffusion model is updated based on the gradient;

[0032] The above steps are repeated for multiple time steps until an input image sample of the last time step is obtained.

[0033] In a second aspect, an image processing method is disclosed, and the method comprises:

[0034] An image to be processed is obtained;

[0035] The image to be processed is input into a task model for processing to obtain a task processing result, and the task model is trained by the data denoising and enhancement collaborative representation learning method of the first aspect.

[0036] In a third aspect, a data denoising and enhancement collaborative representation learning device is disclosed, and the device comprises:

[0037] A denoising processing module is configured to perform natural noise denoising processing on an original image sample to obtain a denoised image sample, and the natural noise represents noise generated by natural factors that affects the accuracy of a task model;

[0038] a noise adding module, configured to add adversarial noise to the denoised image sample to generate an adversarial noise image sample, the adversarial noise representing noise used to deceive the task model to output an incorrect result;

[0039] a noise purification module, configured to purify the noise in the adversarial noise image sample based on a diffusion model to obtain a data enhanced image sample;

[0040] a model training module, configured to train the task model based on the data enhanced image sample and the denoised image sample until the task model converges.

[0041] In a fourth aspect, an electronic device is disclosed, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the data denoising and enhancement collaborative representation learning method according to the first aspect of the embodiments of the present application when executing the computer program.

[0042] In a fifth aspect, a computer readable storage medium is disclosed, which stores a computer program, and the computer program implements the steps of the data denoising and enhancement collaborative representation learning method according to the first aspect of the embodiments of the present application when executed by a processor.

[0043] In a sixth aspect, a computer program product is disclosed, which includes a computer program, and the computer program implements the steps of the data denoising and enhancement collaborative representation learning method according to the first aspect of the embodiments of the present application when executed by a processor.

[0044] The embodiments of the present application have the following advantages:

[0045] In the embodiments of the present application, the original image sample is subjected to natural noise denoising processing to obtain a denoised image sample, the natural noise representing noise affecting the accuracy of the task model caused by natural factors; the adversarial noise is added to the denoised image sample to generate an adversarial noise image sample, the adversarial noise representing noise used to deceive the task model to output an incorrect result; and the noise in the adversarial noise image sample is purified based on a diffusion model to obtain a data enhanced image sample; finally, the task model is trained based on the data enhanced image sample and the denoised image sample until the task model converges.

[0046] The method uses a denoised image sample to learn the adversarial noise, removes the influence of natural noise in the original image sample on the overall learning effect of the adversarial noise, thereby improving the learning effect of the adversarial noise; and uses a purification-based adversarial noise learning method to directly predict and remove the noise, thereby avoiding the occurrence of overfitting caused by excessive learning of the adversarial noise image sample, thereby reducing the reduction of the overall performance caused by the adversarial noise. In this way, an effective robustness-enhanced data representation is realized, and the data denoising and enhancement collaborative representation learning method can better preserve the original performance and accuracy of the task model. BRIEF DESCRIPTION OF DRAWINGS

[0047] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the description of the embodiments of the present application will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0048] Figure 1 is a step flow chart of a data denoising and enhancement collaborative representation learning method provided by the embodiments of the present application;

[0049] Figure 2 is a structural schematic diagram of a data denoising and enhancement collaborative representation learning model provided by the embodiments of the present application;

[0050] Figure 3 is a step flow chart of an image processing method provided by the embodiments of the present application;

[0051] Figure 4 is a structural schematic diagram of a data denoising and enhancement collaborative representation learning device provided by the embodiments of the present application;

[0052] Figure 5 is a structural schematic diagram of an electronic device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0053] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0054] There are a large amount of natural noise and adversarial noise in the image, where the natural noise refers to the noise generated by natural factors that affect the accuracy of the task model, and the climate factors such as fog, rain, waves and lighting conditions can significantly affect the performance of the sensor, leading to a decrease in the quality of image and radar data, and introducing a large amount of natural noise, which can mask the features of the target, making the target detection and classification more difficult. Adversarial noise represents noise used to deceive the task model to output incorrect results. With the widespread use of various anti-detection technologies, including stealth materials, electronic interference and camouflage means, these technologies can effectively interfere with and deceive the task model to produce adversarial noise; adversarial noise not only increases the complexity of the target detection and classification task, but also misleads the classification result, leading to incorrect judgments and responses.

[0055] In related technologies, only one aspect of natural noise and adversarial noise is often optimized. In terms of natural noise removal, common algorithms mainly include artificial feature-based denoising methods and deep learning-based methods; 1) artificial feature-based denoising methods treat noisy images as adding a layer of white noise to clean images. The Gaussian smoothing method uses Gaussian smoothing to remove high-frequency interference components by forming a low-pass filter, and many filter-based noise removal methods are derived therefrom. The variational method minimizes the difference between the denoised image and the original image, which is equivalent to solving the Euler-Lagrange equation to solve this problem. The structural texture restoration method treats denoising as removing redundant text information in the image and reconstructing the image structure by finding a lower bound of the image feature information bottleneck in a certain norm space. The wavelet transform method adds noise to the image by using wavelet transform, and removes noise by inverse wavelet transform, and removes noise by different methods of soft and hard threshold values. 2) Deep learning-based methods mainly include convolutional neural network-based and generative adversarial network-based denoising methods. The convolutional neural network-based method uses self-supervised learning to find the mapping relationship between the target pixel and the input pixel to achieve the denoising task, and extracts the noise distribution after training the supervised learning network and performs difference operation with the original input image to complete the task. The generative adversarial network-based method uses a generator and a discriminator to perform generative adversarial training, which can alleviate the problem of insufficient training image samples and enhance the learning effect.

[0056] In terms of adversarial noise learning, it is mainly divided into methods based on adversarial learning and methods based on purification; 1) the method based on adversarial learning mainly generates adversarial noise through the method of attack and adds it to the original image, and through the training of the model, the classification and detection of the noise image can be correctly completed to realize the adaptation to the noise. At present, the generation of attack image samples is mainly divided into gradient-based attack, heuristic-based attack and generation-based attack. The gradient-based attack interferes with the task result by using gradient information, the heuristic-based attack tampers with the task result by using heuristic rules and strategies, and the generation-based attack is realized by using a generation adversarial network. The generator learns the feature distribution of the image to generate a false attack image, and the discriminator judges the generated result to generate an image with sufficient attack. 2) The method based on purification generates the image with added adversarial noise through a generative model. Compared with the method based on adversarial learning, it mainly removes the added adversarial noise in the image through an additional purification module, predicts the expression after removing the adversarial noise, and trains it to realize effective defense against attack.

[0057] From the above analysis, it can be seen that the existing natural noise removal cannot effectively overcome the influence of adversarial noise. Although it can effectively enhance the model performance, when facing malicious adversarial attack, the model effect will be greatly reduced and greatly affected. The adversarial noise learning focuses on correctly classifying the image samples with adversarial noise, but it over-learns the adversarial image samples, resulting in overfitting of the noise distribution. Although it shows good stability in the face of attack, the correct rate of classification is greatly reduced, and the overall performance is greatly reduced. At the same time, the image samples contain natural noise before adding adversarial noise, which will have a negative impact on subsequent training.

[0058] In order to overcome the limitations of related technologies, the embodiment of the present application provides a data denoising and enhancement collaborative representation learning method. The method uses denoising image samples to learn adversarial noise in order to overcome the problem that the overall adversarial training does not fit the adversarial noise. The natural noise in the original image sample is removed to affect the overall learning effect of the adversarial noise, thereby improving the learning effect of the adversarial noise. In order to reduce the problem that the adversarial noise reduces the overall performance, the method uses a purification-based adversarial noise learning method to directly predict and remove the noise, thereby avoiding the occurrence of overfitting problem caused by over-learning of adversarial noise image samples.

[0059] Referring to Figure 1 , Figure 1 is a step flowchart of a data denoising and enhancement collaborative representation learning method provided by the embodiment of the present application. As Figure 1As shown, the data denoising and enhancement collaborative representation learning method provided by the embodiment of the present application can include steps S110 to S140.

[0060] Step S110: performing natural noise denoising processing on the original image sample to obtain a denoised image sample, wherein the natural noise represents noise generated by natural factors that affects the accuracy of the task model.

[0061] The original image sample can be an image obtained by a sensing device and an image expression obtained after preprocessing (for example, cropping and image conversion). The original image sample contains natural noise, which affects the accuracy of the task model. In order to remove the influence of the natural noise in the original image sample on the subsequent adversarial noise learning effect, the original image sample is subjected to natural noise denoising processing to obtain a denoised image sample that does not contain natural noise.

[0062] For each original image sample, the channel dimension-based attention mechanism is used to process the original image sample to obtain natural noise, and the natural noise is removed from the original image sample to obtain a denoised image sample. In some embodiments, the channel dimension-based attention mechanism is implemented by using a squeeze-activation neural network. Specifically, for an original image sample with a dimension of , the squeeze module of the squeeze-activation neural network aggregates the feature mapping across the spatial dimension to generate channel dimension information with a dimension of . Then, in the activation stage, each channel learns the activation of a specific image sample by using a channel-dependent self-selection gate mechanism, so as to learn to use global information to obtain the output result of the activation operation. Finally, the output result of the activation operation is weighted by using the channel information obtained by the squeeze module, so as to realize the channel dimension-based attention mechanism and obtain a denoised image sample.

[0063] Step S120: generating an adversarial noise image sample by adding adversarial noise to the denoised image sample, wherein the adversarial noise represents noise used to deceive the task model to output an incorrect result.

[0064] In the embodiment of the present application, a purification-based adversarial noise learning method is used to directly predict and remove noise. After obtaining the denoised image sample, the adversarial noise image sample is obtained by adding adversarial noise to the denoised image sample.

[0065] Specifically, the adversarial noise image sample is generated by adding the adversarial noise to the denoised image sample, including: constructing a loss function according to the denoised image sample and the real label corresponding to the original image sample; obtaining a gradient by taking a partial derivative of the loss function, and adding a disturbance according to a change direction of the gradient to obtain the adversarial noise; and generating the adversarial noise image sample by adding the adversarial noise to the denoised image sample.

[0066] For example, the adversarial noise can be represented as:

[0067] ,

[0068] wherein, is a parameter of the task model, is the denoised image sample, is the real label corresponding to the original image sample, is the loss function, and the error is increased by keeping the change amount consistent with the change direction of the gradient, the function ensures that the change direction is consistent with the gradient direction, and the change of the task result is maximized by taking a partial derivative of the loss function, is represented as a disturbance as small as possible, so as to ensure that the image change amplitude is small before and after adding the adversarial noise.

[0069] Further, the adversarial noise is added to the denoised image sample to obtain the adversarial noise image sample:

[0070] ,

[0071] wherein, denotes the adversarial noise image sample, denotes the denoised image sample.

[0072] Step S130: purifying the noise in the adversarial noise image sample based on a diffusion model to obtain a data enhanced image sample.

[0073] In the embodiment of the application, after obtaining the adversarial noise image sample, the diffusion model is used to purify the noise in the adversarial noise image sample. The diffusion model is mainly used to effectively drown the malicious adversarial noise by adding Gaussian noise in the diffusion process of the diffusion model, and to remove the noise by denoising when the distribution before and after adding a certain amount of Gaussian noise is consistent, so as to realize the common removal of the adversarial noise.

[0074] Step S140: training the task model based on the data enhanced image sample and the denoised image sample until the task model converges.

[0075] In the embodiment of the present application, the denoised image sample does not contain natural noise, the data enhanced image sample is obtained by purifying the adversarial noise image sample based on the diffusion process of the diffusion module, and the data enhanced image sample does not contain adversarial noise on the basis of not containing natural noise. Training the task model based on the data enhanced image sample and the denoised image sample enables the task model to learn the characteristics of different image samples.

[0076] The task model refers to a model used to perform a specific task, and can be a target detection model, a target classification model, etc. Training the task model based on the data enhanced image sample and the denoised image sample means that the data enhanced image sample and the denoised image sample construct a loss value, and the model parameters of the task model are updated based on the loss value. In the case of convergence of the task model, the trained task model is obtained.

[0077] In the above process, the denoised image sample without natural noise is used for adversarial noise learning, which removes the influence of natural noise in the original image sample on the overall learning effect of adversarial noise, thereby improving the adversarial noise learning effect. And using the purifying-based adversarial noise learning method, the noise is directly predicted and removed, avoiding the occurrence of overfitting problem caused by excessive learning of adversarial noise image sample, thereby reducing the reduction of overall performance caused by adversarial noise. In this way, an effective data denoising and enhancement collaborative representation learning method is realized, which can effectively enhance the robustness of data representation and better preserve the original performance and accuracy of the task model.

[0078] The data denoising and enhancement collaborative representation learning method provided by the embodiments of the present application is described in the following sections 1.1, 1.2 and 1.3.

[0079] 1.1 Training process of task model

[0080] In combination with the above embodiments, in an embodiment, the data denoising and enhancement collaborative representation learning method provided by the embodiments of the present application is also provided. In this method, the "training the task model based on the data enhanced image sample and the denoised image sample until the task model converges" in the above step S140 can specifically include steps S140-1 to S140-5:

[0081] Step S140-1: fuse the data enhanced image sample and the denoised image sample to obtain a fused image sample.

[0082] Step S140-2: input the fused image sample into the task model to obtain a task processing result.

[0083] Step S140-3: obtaining a first loss according to a difference between the task processing result and a true label corresponding to the original image sample.

[0084] Step S140-4: obtaining a second loss according to a difference between the denoised image sample and the fusion image sample, and obtaining a third loss according to a difference between the data enhanced image sample and the fusion image sample.

[0085] Step S140-5: updating model parameters of the task model according to a total loss obtained according to the first loss, the second loss and the third loss.

[0086] In the embodiments of the present application, the task processing result is determined according to a downstream task of the task model, for example, the task result can be a target detection result, or a target classification result, etc.

[0087] For example, the total loss Ltotal is represented as:

[0088]

[0089] wherein, L1 represents the first loss, the task processing accuracy of the task model is optimized through the first loss; L2 represents the second loss, the consistency between the denoised image sample and the fusion image sample is ensured through the second loss; L3 represents the third loss, the consistency between the data enhanced image sample and the fusion image sample is ensured through the third loss; is an adjustable hyperparameter, used for controlling the weight of the loss.

[0090] In this way, the task model is trained based on the fusion image sample, the denoised image sample and the adversarial noise image sample. Since the denoised image sample does not contain natural noise, the adversarial noise learning is performed by using the denoised image sample from which the natural noise is removed, the influence of the natural noise in the original image sample on the overall learning effect of the adversarial noise is removed, and the purified adversarial noise learning method is used to directly predict and remove the noise, avoiding the occurrence of overfitting problem caused by excessive learning of the adversarial noise image sample, thereby reducing the reduction of the overall performance caused by the adversarial noise.

[0091] 1.2 Natural noise denoising processing on the original image sample:

[0092] ​​​In combination with the above embodiments, in an embodiment, the application further provides a data denoising and enhancement collaborative representation learning method. In the method, the "performing natural noise denoising processing on the original image sample to obtain a denoised image sample" in the step S110 can specifically include steps S110-1 to S110-5.

[0093] Step S110-1: obtaining input representations of a plurality of attention heads according to the original image sample.

[0094] Step S110-2: performing multi-head squeezing operation on the input representations of the plurality of attention heads to obtain channel features of the plurality of attention heads.

[0095] Step S110-3: performing activation operation on the channel features of the plurality of attention heads to obtain activation operation results of the plurality of attention heads, the activation operation being an operation of expressing and extracting nonlinear relationships of the channel features of the plurality of attention heads.

[0096] Step S110-4: fusing the channel features of the plurality of attention heads and the activation operation results of the plurality of attention heads, and obtaining natural noise according to the fusion results.

[0097] Step S110-5: removing the natural noise from the original image sample to obtain a denoised sample.

[0098] In the embodiments of the application, in order to obtain more accurate denoised image samples, a multi-head squeezing-activation neural network model is proposed to implement a more general and efficient channel dimension attention mechanism. The original image sample is processed by the multi-head squeezing-activation neural network model to obtain more accurate natural noise, thereby obtaining more accurate denoised image samples.

[0099] Specifically, for each original image sample, step S110-1 is first performed to obtain input representations of a plurality of attention heads. For example, for an original image sample with a dimension of C x H x W, the input representations of the plurality of attention heads are obtained by transforming the original image sample. The input representations of the plurality of attention heads can be represented as:

[0100] ,

[0101] wherein, represents the input representation of the first attention head, represents the input representation of the second attention head, represents the input representation of the nth attention head.

[0102] ​Then, step S110-2 is performed, and the input representations of the plurality of attention heads are subjected to multi-head squeezing operation in the squeezing section of the multi-head squeezing-activation neural network model to obtain channel features of the plurality of attention heads. The channel features of the plurality of attention heads can be represented as:

[0103] ,

[0104] wherein c represents the dimension of the input channel; n represents the number of attention heads; represents a pixel point in the i-th row and the j-th column of the input representation; represents a channel feature in one channel dimension; represents the channel features of the plurality of attention heads, and the dimension is That is, the multi-head squeezing operation is performed on the input representations of the plurality of attention heads, and the compressed features of different channel information (i.e., the channel features of the plurality of attention heads) can be obtained for n groups of inputs.

[0105] Then, step S110-3 is performed to obtain the activation operation results of the plurality of attention heads. Specifically, the channel features of the plurality of attention heads are compressed by using a first fully connected layer to obtain channel features in a target number of channel dimensions; the number of channels in the target number of channel dimensions is recovered by using a nonlinear activation layer and a second fully connected layer to obtain recovered channel features; and the recovered channel features are processed by using a target activation function to obtain the activation operation results of the plurality of attention heads.

[0106] In the embodiments of the present application, the activation module of the multi-head squeezing-activation neural network model is used to realize comprehensive expression of the channel features of the plurality of attention heads and extraction of nonlinear relationships and non-exclusive relationships. For example, the activation operation results of the plurality of attention heads are represented as:

[0107] ,

[0108] wherein, is a target activation function, and in some embodiments, the target activation function can be a sigmord function; is a nonlinear activation layer, and in some embodiments, the activation function can be a LeakyReLU activation function; is a weight matrix of the first fully connected layer; is a weight matrix of the second fully connected layer.

[0109] The channel features of the plurality of attention heads of are converted into channel features in a target number of channel dimensions of by using the first fully connected layer, and after the nonlinear activation layer and the second fully connected layer, they are recovered again into the recovery channel features. Finally, the activation operation results of the plurality of attention heads are obtained through a target activation function

[0110] The channel features of the plurality of attention heads and the activation operation results of the plurality of attention heads are fused by performing step S110-4, and natural noise is obtained according to the fusion result. Exemplarily, the fusion of the channel features of the plurality of attention heads and the activation operation results of the plurality of attention heads can be represented as:

[0111]

[0112] wherein, represents a fusion operation, represents a fusion result.

[0113] Further, according to the fusion result and the original image sample, natural noise is obtained, and the natural noise may be represented as:

[0114]

[0115] Finally, the natural noise is removed from the original image sample to obtain a denoised image sample, which can be represented as:

[0116]

[0117] wherein, represents a denoised image sample.

[0118] In the above process, a more general and efficient channel dimension attention mechanism is realized based on the multi-head squeeze-activation neural network model, which solves the problems of too rough channel granularity and poor cross-effect in related denoising schemes, thereby obtaining a more accurate denoised image sample.

[0119] 1.3 Purifying noise in the adversarial noise image sample based on a diffusion model:

[0120] In the embodiments of the present application, there are problems such as overfitting of adversarial image samples, generalization bias, poor performance on normal data sets, etc. in adversarial training, and an adversarial purification method is used. Adversarial purification aims to remove adversarial perturbations from adversarial image samples to protect classifiers from attacks by unseen threats. This method does not depend on the specific assumptions of attack forms and classification models, does not need to retrain the classifier, only needs to apply the purification step before the classifier, and therefore can be used to protect existing classifiers against unknown attacks and maintain performance on normal image samples as much as possible.

[0121] ​​​​The embodiment of the application purifies the noise in the adversarial noise image sample based on a diffusion model, and can be expressed as follows for the time step of the noise adding process:

[0122] ,

[0123] That is, in the forward process, the distribution of clean data is closest to the distribution of adversarial perturbation data , that is, the derivative of the KL divergence of the two with respect to t is less than 0, and the problem of adversarial purification can be formalized as follows:

[0124] ,

[0125] wherein, represents the optimal reconstructed image sample, the adversarial noise image sample is regarded as a perturbation measurement of the original clean image sample, and it is assumed that the clean image sample is generated by a prior probability distribution, that is, . Given the adversarial noise image sample , the target is to restore the original image sample , so that the posterior distribution is maximized. The defense of the adversarial image sample is solved by optimizing the problem, that is, finding the original image sample that can maximize the posterior distribution, and the maximization of the log-likelihood of the posterior distribution is realized. Therefore, a variational posterior distribution is introduced to approximate the true posterior distribution in the original optimization target, wherein the variational upper bound is:

[0126] ,

[0127] wherein, represents the clean data distribution reconstructed using the parameters .

[0128] After simplification, the loss function can be expressed as follows:

[0129] ,

[0130] wherein, is the output of the input image sample after adding Gaussian noise and passing through the diffusion model, and the loss function is obtained by calculating the loss of and the input image sample , so that the diffusion model is trained using the loss function to obtain the data enhancement image sample.

[0131] In combination with the above embodiments, in an embodiment, the application further provides a data denoising and enhancement collaborative representation learning method. In the method, the step S130 of "purifying the noise in the adversarial noise image sample based on the diffusion model to obtain a data enhancement image sample" can include steps S130-1 to S130-3.

[0132] Step S130-1: At each time step, the adversarial noise image sample is added with noise according to the randomly sampled Gaussian noise to obtain a noise-added image sample, and the input image sample at the current time step is predicted according to the noise-added image sample at each time step, and the process is repeated for multiple time steps until the input image sample at the last time step is obtained.

[0133] Step S130-2: The above process is iterated multiple times to obtain a trained diffusion model.

[0134] Step S130-3: The adversarial noise image sample is added with noise and denoised using the trained diffusion model to obtain a data enhancement image sample.

[0135] In the embodiments of the application, the training of the diffusion model is simplified to the form of predicting the loss function of the noise added at each time step and the real added noise, so as to train the diffusion model. In this way, a smaller time step is selected to ensure that the added noise is as small as possible but sufficient to overwhelm the adversarial noise, so that it is removed together with the adversarial noise in the denoising process.

[0136] Finally, the trained diffusion model is used to add noise and denoise the adversarial noise image sample to obtain a data enhancement image sample.

[0137] In this way, the diffusion model predicts the added noise corresponding to each time step based on the noise-added image sample at each time step, and then obtains the clean image sample corresponding to the last time step through multiple time step prediction processes, i.e., obtains a data enhancement image sample.

[0138] Further, the step S130-1 of "at each time step, the adversarial noise image sample is added with noise according to the randomly sampled Gaussian noise to obtain a noise-added image sample, and the input image sample at the current time step is predicted according to the noise-added image sample at each time step, and the process is repeated for multiple time steps until the input image sample at the last time step is obtained" includes steps B1 to B7.

[0139] Step B1: Randomly sample Gaussian noise from a Gaussian distribution.

[0140] For example, the Gaussian distribution can be represented as Randomly sample the Gaussian distribution to obtain Gaussian noise .

[0141] Step B2: add the Gaussian noise to the input image sample of the previous time step and the adversarial noise image sample of the previous time step respectively to obtain the input image sample of the current time step and the adversarial noise image sample of the current time step.

[0142] For example, the input image sample of the current time step can be represented as:

[0143] ,

[0144] The Gaussian noise-added image sample can be represented as:

[0145] ,

[0146] wherein, denotes the input image sample of the previous time step, denotes the input image sample of the current time step, denotes the adversarial noise image sample of the current time step, denotes the adversarial noise image sample of the previous time step, wherein the adversarial noise image sample of the first time is the adversarial noise image sample generated in step S120 , denotes the correlation coefficient of time t.

[0147] Step B3: input the adversarial noise image sample of the current time step to the denoising module of the diffusion model to obtain a model denoised image sample.

[0148] Specifically, the denoising module of the diffusion model predicts the added noise of the adversarial noise image sample of the current time step, and removes the predicted added noise from the adversarial noise image sample of the current time step, thereby obtaining a model denoised image sample.

[0149] Step B4: construct a fourth loss according to the model denoised image sample and the input image sample of the current time step.

[0150] For example, the fourth loss is represented as wherein, is the model denoised image sample.

[0151] Step B5: calculate the gradient of the input image sample of the current time step based on the fourth loss, and update the diffusion model based on the gradient.

[0152] For example, the gradient of the input image sample of the current time step is represented as:

[0153] ,

[0154] Step B6: Repeat the above steps for multiple time steps until the input image sample of the last time step is obtained.

[0155] In the embodiments of the present application, according to the method of steps B1 to B6 and the calculation of the input image sample of each time step, when the input image sample of the last time step is obtained, it is indicated that the iteration of the current round is completed. Then the iteration of the next round can be started until the training end condition is met, and the trained diffusion model is obtained.

[0156] The data denoising and enhancement collaborative representation learning method provided in the embodiments of the present application is implemented based on a data denoising and enhancement collaborative representation learning model. For example, Figure 2 is a structural schematic diagram of a data denoising and enhancement collaborative representation learning model provided in the embodiments of the present application, wherein the data denoising and enhancement collaborative representation learning model includes a denoising module and a robust enhancement module; the denoising module is used to obtain a denoised image sample by performing natural noise denoising processing on an original image sample; and the denoising module is based on a multi-head squeeze-activation neural network model to process the original image sample to obtain natural noise; and the natural noise is removed from the original image sample to obtain the denoised image sample.

[0157] The robust enhancement module is used to add adversarial noise to the denoised image sample to generate an adversarial noise image sample, and based on a diffusion model, the noise in the adversarial noise image sample is purified to obtain a data enhancement image sample. Finally, the data enhancement image sample and the denoised image sample are fused to obtain a fused image sample. Finally, the fused image sample output by the robust enhancement module can be used to train the task model.

[0158] It can be understood that the data denoising and enhancement collaborative representation learning method provided in the embodiments of the present application can be applied to the fields of target detection and target classification. For example, for ship target detection and classification, since the images collected by the sensing device contain a large amount of natural noise and adversarial noise, the method can improve the accuracy of ship target detection and classification.

[0159] Furthermore, to better illustrate the performance of the representation learning method that combines data denoising and enhancement in the embodiments of this application, robustness improvements were achieved on multiple tasks in adversarial noise scenarios through trials on multiple datasets. Compared to traditional methods for removing natural noise, this method achieves a significant improvement in robustness; and compared to other adversarial noise learning methods, this method achieves an effective performance improvement, reducing the loss of model performance due to learning adversarial noise. Extensive experiments were conducted on ship target detection and classification tasks, and the experimental results demonstrate the effectiveness of the technology presented in this application.

[0160] Specifically, the test results for the ship classification FGSC 23 dataset and the ship target detection HRSC 2016 dataset are shown in Table 1. Four task models (VGG16, ResNet50, MobileNet, and Xception) were trained using the proposed method, FGSM Adv Train, and C&W Adv Train, respectively. As shown in Table 1, the task model trained using the proposed method exhibits better performance.

[0161] Table 1 Test results of ship target detection and classification tasks

[0162]

[0163] Reference Figure 3 As shown, Figure 3 This is a flowchart illustrating the steps of an image processing method provided in an embodiment of this application. Figure 3 As shown, an image processing method provided in this application embodiment may include steps S310 to S320:

[0164] Step S310: Obtain the image to be processed.

[0165] Step S320: Input the image to be processed into the task model for processing to obtain the task processing result. The task model is trained by the representation learning method of data denoising and enhancement collaboration described in the embodiments of this application.

[0166] In the embodiment of the present application, the task model is used to process the image to be processed. Since the task model is trained by the data denoising and enhancement collaborative representation learning method described in the embodiment of the present application, the data denoising and enhancement collaborative representation learning method uses the denoising image sample to remove natural noise for adversarial noise learning, removes the influence of natural noise in the original image sample on the overall learning effect of the adversarial noise, thereby improving the adversarial noise learning effect. Moreover, the adversarial noise learning method based on purification directly predicts and removes the noise, avoids the occurrence of overfitting problem caused by excessive learning of the adversarial noise image sample, thereby reducing the overall performance reduction of the adversarial noise. Therefore, based on the task model, an accurate task processing result can be obtained.

[0167] The embodiment of the present application also provides a data denoising and enhancement collaborative representation learning device, as shown in Figure 4 Figure 4 The device provided by the embodiment of the present application includes:

[0168] The denoising processing module 410 is configured to perform natural noise denoising processing on the original image sample to obtain a denoising image sample, wherein the natural noise represents noise generated by natural factors that affects the accuracy of the task model.

[0169] The noise adding module 420 is configured to generate an adversarial noise image sample by adding adversarial noise to the denoising image sample, wherein the adversarial noise represents noise used to deceive the task model to output an incorrect result.

[0170] The noise purification module 430 is configured to purify the noise in the adversarial noise image sample based on a diffusion model to obtain a data enhancement image sample.

[0171] The model training module 440 is configured to train the task model based on the data enhancement image sample and the denoising image sample until the task model converges.

[0172] In an optional embodiment, the model training module includes:

[0173] The sample fusion module is configured to fuse the data enhancement image sample and the denoising image sample to obtain a fused image sample.

[0174] The task processing result module is configured to input the fused image sample into the task model to obtain a task processing result.

[0175] The result loss module is configured to obtain a first loss according to the difference between the task processing result and the real label corresponding to the original image sample.

[0176] ​an image sample loss module configured to obtain a second loss according to a difference between the denoised image sample and the fused image sample, and obtain a third loss according to a difference between the data enhanced image sample and the fused image sample;

[0177] a parameter updating module configured to update model parameters of the task model according to a total loss obtained by the first loss, the second loss and the third loss.

[0178] In an optional embodiment, the denoising processing module comprises:

[0179] an obtaining module configured to obtain input representations of a plurality of attention heads according to the original image sample;

[0180] a squeezing operation module configured to perform multi-head squeezing operation on the input representations of the plurality of attention heads to obtain channel features of the plurality of attention heads;

[0181] an activation operation module configured to perform activation operation on the channel features of the plurality of attention heads to obtain activation operation results of the plurality of attention heads, the activation operation being to express and extract nonlinear relationships of the channel features of the plurality of attention heads;

[0182] a result fusion module configured to fuse the channel features of the plurality of attention heads and the activation operation results of the plurality of attention heads, and obtain natural noise according to a fusion result;

[0183] a first removing module configured to remove the natural noise from the original image sample to obtain a denoised image sample.

[0184] In an optional embodiment, the noise purification module comprises:

[0185] a prediction module configured to, at each time step, perform noise adding processing on an adversarial noise image sample according to randomly sampled Gaussian noise to obtain a noise added image sample, and predict an input image sample of a current time step according to the noise added image sample of each time step, and repeat a plurality of time steps until an input image sample of a last time step is obtained;

[0186] an iteration module configured to perform a plurality of iterations in the above manner to obtain a trained diffusion model;

[0187] a second processing module configured to perform noise adding processing and denoising processing on the adversarial noise image sample by using the trained diffusion model to obtain a data enhanced image sample.

[0188] In an optional embodiment, the prediction module comprises:

[0189] a sampling module configured to randomly sample from a Gaussian distribution to obtain Gaussian noise;

[0190] a noise adding module configured to add the Gaussian noise to an input image sample of a previous time step and an adversarial noise image sample of the previous time step respectively to obtain an input image sample of a current time step and an adversarial noise image sample of the current time step;

[0191] a first input module configured to input the adversarial noise image sample of the current time step to a denoising module of the diffusion model to obtain a model denoised image sample;

[0192] a second loss constructing module configured to calculate a gradient of the input image sample of the current time step based on the fourth loss and update the diffusion model based on the gradient;

[0193] a repeating module configured to repeat the above steps for multiple time steps until an input image sample of a last time step is obtained.

[0194] The embodiment of the present application further provides an electronic device, referring to Figure 5 , Figure 5 is a structural schematic diagram of an electronic device provided by the embodiment of the present application. As shown in Figure 5 , the electronic device 500 comprises a memory 510 and a processor 520, the memory 510 and the processor 520 are in communication connection through a bus, the memory 510 stores a computer program, the computer program can run on the processor 520, and then the steps of the data denoising and enhancement collaborative representation learning method described in the embodiment of the present application are realized.

[0195] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize the steps of the data denoising and enhancement collaborative representation learning method described in the embodiment of the present application.

[0196] The embodiment of the present application further provides a computer program product, comprising a computer program, and the computer program is executed by a processor to realize the steps of the data denoising and enhancement collaborative representation learning method described in the embodiment of the present application.

[0197] Each embodiment in the specification is described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same and similar parts between each embodiment can be referred to each other.

[0198] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, apparatuses (systems) according to the embodiments of the present application. It is understood that each flow and / or block in the flowcharts and / or block diagrams, and a combination of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminals to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminals generate a device implemented in the flowcharts and / or block diagrams of the methods, apparatuses (systems) according to the embodiments of the present application. Figure 1 The device implemented in one flow or multiple flows and / or one block or multiple blocks. Figure 1 The device implemented in one flow or multiple flows and / or one block or multiple blocks.

[0199] These computer program instructions can also be stored in a computer readable memory capable of directing the computer or other programmable data processing terminal to work in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including instruction devices, which implement the functions specified in the flowcharts and / or block diagrams of the methods, apparatuses (systems) according to the embodiments of the present application. Figure 1 The device implemented in one flow or multiple flows and / or one block or multiple blocks. Figure 1 The device implemented in one flow or multiple flows and / or one block or multiple blocks.

[0200] These computer program instructions can also be loaded into a computer or other programmable data processing terminal, so that a series of operation steps are performed on the computer or other programmable terminal to produce a computer implemented process, so that the instructions executed on the computer or other programmable terminal provide steps for implementing the functions specified in the flowcharts and / or block diagrams of the methods, apparatuses (systems) according to the embodiments of the present application. Figure 1 The device implemented in one flow or multiple flows and / or one block or multiple blocks. Figure 1 The device implemented in one flow or multiple flows and / or one block or multiple blocks.

[0201] Although the preferred embodiments of the present application have been described, those skilled in the art can make additional changes and modifications to the embodiments once they know the basic inventive concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the embodiments of the present application.

[0202] Finally, it is to be understood that the phraseology or terminology such as "first" and "second" etc. used herein is merely intended to differentiate one entity or operation from another entity or operation, without necessarily requiring or implying any actual such relationship or order between such entities or operations. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0203] The above describes in detail the data denoising and enhancement collaborative representation learning method provided by the present application. The principles and implementation manners of the present application are described by using specific examples. The above description of the examples is only used to help understand the method of the present application and its core idea. Meanwhile, for those skilled in the art, the specific implementation manners and application ranges can be changed according to the idea of the present application. In summary, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A data denoising and enhancement collaborative representation learning method, characterized in that, The method includes: The method involves denoising the original image samples using natural noise to obtain denoised image samples. This includes: acquiring input representations of multiple attention heads based on the original image samples; performing multi-head compression on the input representations of the multiple attention heads to obtain channel features of the multiple attention heads; performing activation operations on the channel features of the multiple attention heads to obtain activation operation results, where the activation operation refers to expressing and extracting the nonlinear relationship of the channel features of the multiple attention heads; fusing the channel features of the multiple attention heads and the activation operation results of the multiple attention heads, and obtaining natural noise based on the fusion result; and removing the natural noise from the original image samples to obtain denoised samples. The natural noise represents noise generated by natural factors that affects the accuracy of the task model. By adding adversarial noise to the denoised image samples, adversarial noise is generated, which represents noise used to deceive the task model into outputting incorrect results; The process of purifying the adversarial noise image samples using a diffusion model to obtain data-enhanced image samples includes: at each time step, adding noise to the adversarial noise image samples based on randomly sampled Gaussian noise to obtain noise-added image samples; predicting the input image sample for the current time step based on the noise-added image samples at each time step; repeating this process multiple times until the input image sample for the last time step is obtained; performing multiple iterations in the above manner to obtain a trained diffusion model; and using the trained diffusion model to perform noise addition and denoising on the adversarial noise image samples to obtain data-enhanced image samples. Within each time step, the adversarial image sample is denoised using randomly sampled Gaussian noise to obtain a noise-added image sample. Based on the noise-added image sample from each time step, the input image sample for the current time step is predicted. This process is repeated for multiple time steps until the input image sample for the last time step is obtained. This includes: randomly sampling from a Gaussian distribution to obtain Gaussian noise; adding the Gaussian noise to the input image sample from the previous time step and the adversarial image sample from the previous time step to obtain the input image sample and the adversarial image sample for the current time step; inputting the adversarial image sample for the current time step into the denoising module of the diffusion model to obtain a model-denoised image sample; constructing a fourth loss based on the model-denoised image sample and the input image sample for the current time step; calculating the gradient of the input image sample for the current time step based on the fourth loss, and updating the diffusion model based on the gradient; repeating the above steps for multiple time steps until the input image sample for the last time step is obtained. The task model is trained based on the data-enhanced image samples and the denoised image samples until the task model converges.

2. The method of claim 1, wherein, Based on the data-enhanced image samples and the denoised image samples, the task model is trained until the task model converges, including: fuse the data enhanced image sample and the denoised image sample to obtain a fused image sample; input the fused image sample into the task model to obtain a task processing result; obtain a first loss according to a difference between the task processing result and a true label corresponding to the original image sample; obtain a second loss according to a difference between the denoised image sample and the fused image sample, and obtain a third loss according to a difference between the data enhanced image sample and the fused image sample; update model parameters of the task model according to a total loss obtained according to the first loss, the second loss and the third loss.

3. An image processing method characterized by, The method comprises: obtaining a to-be-processed image; inputting the to-be-processed image into a task model for processing to obtain a task processing result, wherein the task model is trained by the data denoising and enhancement collaborative representation learning method of claim 1 or 2.

4. A data denoising and enhancement collaborative representation learning apparatus, characterized in that, The device comprises: a denoising processing module configured to perform natural noise denoising processing on an original image sample to obtain a denoised image sample, comprising: obtaining input representations of a plurality of attention heads according to the original image sample; performing multi-head squeezing operation on the input representations of the plurality of attention heads to obtain channel features of the plurality of attention heads; performing activation operation on the channel features of the plurality of attention heads to obtain activation operation results of the plurality of attention heads, wherein the activation operation refers to expressing and extracting nonlinear relationships of the channel features of the plurality of attention heads; fusing the channel features of the plurality of attention heads and the activation operation results of the plurality of attention heads, and obtaining natural noise according to a fusion result; removing the natural noise from the original image sample to obtain a denoised sample, wherein the natural noise represents noise generated by natural factors that affects the accuracy of a task model; a noise adding module configured to generate an adversarial noise image sample by adding adversarial noise to the denoised image sample, wherein the adversarial noise represents noise used to deceive a task model to output an incorrect result; a noise purification module configured to purify noise in the adversarial noise image sample based on a diffusion model to obtain a data enhanced image sample, comprising: at each time step, performing noise adding processing on the adversarial noise image sample according to randomly sampled Gaussian noise to obtain a noise added image sample, and predicting an input image sample at a current time step according to the noise added image sample at each time step, and repeating a plurality of time steps until an input image sample at a last time step is obtained; performing multiple iterations in the above manner to obtain a trained diffusion model; and performing noise adding processing and denoising processing on the adversarial noise image sample by using the trained diffusion model to obtain a data enhanced image sample. The method comprises: in each time step, adding noise to an adversarial noise image sample according to randomly sampled Gaussian noise to obtain a noise-added image sample, and predicting an input image sample of the current time step according to the noise-added image sample of each time step, repeating a plurality of time steps until an input image sample of the last time step is obtained, comprising: randomly sampling Gaussian noise from a Gaussian distribution; adding the Gaussian noise to the input image sample of the previous time step and the adversarial noise image sample of the previous time step to obtain the input image sample of the current time step and the adversarial noise image sample of the current time step; inputting the adversarial noise image sample of the current time step into a denoising module of the diffusion model to obtain a model denoised image sample; constructing a fourth loss according to the model denoised image sample and the input image sample of the current time step; calculating the gradient of the input image sample of the current time step based on the fourth loss, and updating the diffusion model based on the gradient; repeating a plurality of time steps according to the above steps until the input image sample of the last time step is obtained; The model training module is configured to train a task model based on the data enhanced image sample and the denoised image sample until the task model converges.

5. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the steps of the data denoising and enhancement collaborative representation learning method of claim 1 or 2.

6. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the data denoising and enhancement collaborative representation learning method of claim 1 or 2.

7. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the data denoising and enhancement collaborative representation learning method of claim 1 or 2. The computer program is executed by the processor to implement the steps of the data denoising and enhancement collaborative representation learning method of claim 1 or 2.

Images