Data denoising and enhancement collaborative representation learning method

Abstract

The invention provides a data denoising and enhancement collaborative representation learning method, relates to the technical field of data processing, and aims to perform collaborative representation learning on natural noise and adversarial noise so as to improve the precision of a task model.The method comprises the steps that natural noise denoising processing is performed on an original image sample to obtain a denoised image sample, and the denoised image sample is subjected to collaborative representation learning; the natural noise represents noise which is generated by natural factors and affects the accuracy of the task model; adversarial noise is added to the de-noised image sample, an adversarial noise image sample is generated, and the adversarial noise represents noise used for outputting an error result by a deception task model; noise in the anti-noise image sample is purified based on a diffusion model, and a data enhancement image sample is obtained; and training a task model based on the data enhanced image sample and the de-noised image sample until the task model converges.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a representation learning method for collaborative data denoising and enhancement. Background Art

[0002] In real life, the digitization and transmission of images are often affected by the interference of imaging equipment and external environmental noise, resulting in a decrease in the quality of the acquired images. There is a large amount of natural noise and adversarial noise in the images, which affects the accuracy of the task results (for example, target detection tasks).

[0003] In related technologies, optimization is often performed on only one aspect, natural noise and adversarial noise. Although existing methods for removing natural noise can effectively remove natural noise, they are vulnerable to adversarial noise and their effectiveness decreases significantly when interfered with or attacked. Existing methods for adversarial noise only focus on learning adversarial noise. Although they show good stability when facing attacks, they sacrifice certain task accuracy, resulting in increased error rates and reduced overall performance. Summary of the invention

[0004] In view of the above problems, an embodiment of the present application provides a representation learning method for collaborative data denoising and enhancement to overcome the above problems or at least partially solve the above problems.

[0005] In a first aspect of an embodiment of the present application, a data denoising and enhancement collaborative representation learning method is disclosed, the method comprising: Performing natural noise denoising on the original image sample to obtain a denoised image sample, wherein the natural noise represents the noise generated by natural factors that affects the accuracy of the task model; Generate an adversarial noise image sample by adding adversarial noise to the denoised image sample, wherein the adversarial noise represents noise used to deceive the task model to output an erroneous result; Purifying the noise in the anti-noise image sample based on a diffusion model to obtain a data enhanced image sample; The task model is trained based on the data enhanced image samples and the denoised image samples until the task model converges.

[0006] Optionally, training the task model based on the data augmented image samples and the denoised image samples until the task model converges includes: Fusing the data enhanced image sample and the denoised image sample to obtain a fused image sample; Inputting the fused image sample into the task model to obtain a task processing result; Obtaining a first loss according to a difference between the task processing result and a true label corresponding to the original image sample; Obtaining a second loss according to a difference between the denoised image sample and the fused image sample, and obtaining a third loss according to a difference between the data enhanced image sample and the fused image sample; The model parameters of the task model are updated according to the total loss obtained by the first loss, the second loss and the third loss.

[0007] Optionally, performing natural noise denoising on the original image sample to obtain a denoised image sample includes: According to the original image sample, obtaining input representations of multiple attention heads; Performing a multi-head squeezing operation on the input representations of the multiple attention heads to obtain channel features of the multiple attention heads; Performing an activation operation on the channel features of the multiple attention heads to obtain activation operation results of the multiple attention heads, wherein the activation operation refers to expressing and extracting a nonlinear relationship between the channel features of the multiple attention heads; Fusing the channel features of the multiple attention heads and the activation operation results of the multiple attention heads, and obtaining natural noise according to the fusion result; The natural noise is removed from the original image sample to obtain a denoised sample.

[0008] Optionally, purifying the noise in the anti-noise image sample based on a diffusion model to obtain a data enhanced image sample includes: In each time step, the adversarial noise image samples are denoised according to the randomly sampled Gaussian noise to obtain noise-added image samples, and the input image samples of the current time step are predicted according to the noise-added image samples of each time step, and the process is repeated for multiple time steps until the input image samples of the last time step are obtained; Perform multiple iterations in the above manner to obtain a trained diffusion model; The trained diffusion model is used to perform noise addition and denoising on the anti-noise image samples to obtain data enhanced image samples.

[0009] Optionally, in each time step, the adversarial noise image samples are subjected to noise addition processing according to the randomly sampled Gaussian noise to obtain noise-added image samples, and the input image samples of the current time step are predicted according to the noise-added image samples of each time step, and the process is repeated for multiple time steps until the input image samples of the last time step are obtained, including: Randomly sample from a Gaussian distribution to obtain Gaussian noise; Adding the Gaussian noise to the input image sample of the previous time step and the adversarial noise image sample of the previous time step respectively, to obtain the input image sample of the current time step and the adversarial noise image sample of the current time step; Inputting the anti-noise image sample of the current time step into the denoising module of the diffusion model to obtain a model denoised image sample; Constructing a fourth loss according to the model denoised image samples and the input image samples at the current time step; Based on the fourth loss, calculating the gradient of the input image sample at the current time step, and updating the diffusion model based on the gradient; Repeat the above steps for multiple time steps until the input image sample of the last time step is obtained.

[0010] A second aspect of the embodiments of the present application discloses an image processing method, the method comprising: Get the image to be processed; The image to be processed is input into a task model for processing to obtain a task processing result, and the task model is trained by the data denoising and enhancement collaborative representation learning method described in the first aspect of the embodiment of the present application.

[0011] In a third aspect of the embodiments of the present application, a data denoising and enhancement collaborative representation learning device is disclosed, the device comprising: A denoising processing module is used to perform natural noise denoising on the original image sample to obtain a denoised image sample, wherein the natural noise represents the noise generated by natural factors that affects the accuracy of the task model; A noise adding module, used for generating an adversarial noise image sample by adding adversarial noise to the denoised image sample, wherein the adversarial noise represents noise used to deceive the task model to output an erroneous result; A noise purification module, used for purifying the noise in the anti-noise image sample based on a diffusion model to obtain a data enhanced image sample; The model training module is used to train the task model based on the data enhanced image samples and the denoised image samples until the task model converges.

[0012] The fourth aspect of the embodiments of the present application discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the data denoising and enhancement collaborative representation learning method described in the first aspect of the embodiments of the present application are implemented.

[0013] The fifth aspect of the embodiments of the present application discloses a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the data denoising and enhancement collaborative representation learning method described in the first aspect of the embodiments of the present application are implemented.

[0014] The sixth aspect of the embodiments of the present application discloses a computer program product, including a computer program, which, when executed by a processor, implements the steps of the data denoising and enhancement collaborative representation learning method described in the first aspect of the embodiments of the present application.

[0015] The embodiments of the present application include the following advantages: In an embodiment of the present application, a denoised image sample is obtained by performing natural noise denoising on the original image sample, wherein the natural noise represents the noise generated by natural factors that affects the accuracy of the task model; an adversarial noise image sample is generated by adding adversarial noise to the denoised image sample, wherein the adversarial noise represents the noise used to deceive the task model into outputting an erroneous result; and the noise in the adversarial noise image sample is purified based on a diffusion model to obtain a data enhanced image sample; finally, based on the data enhanced image sample and the denoised image sample, the task model is trained until the task model converges.

[0016] This method uses denoised image samples with natural noise removed for adversarial noise learning, removing the influence of natural noise in the original image samples on the overall learning effect of adversarial noise, thereby improving the learning effect of adversarial noise; and, using a purification-based adversarial noise learning method, directly predicts and removes noise, avoiding the occurrence of overfitting problems caused by over-learning of adversarial noise image samples, thereby reducing the overall performance degradation caused by adversarial noise. In this way, a data denoising and enhancement collaborative representation learning method is achieved that effectively enhances the robustness of data expression and can better retain the original performance and accuracy of the task model. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0018] Figure 1 It is a flowchart of the steps of a data denoising and enhancement collaborative representation learning method provided in an embodiment of the present application; Figure 2 It is a structural diagram of a data denoising and enhancement collaborative representation learning model provided in an embodiment of the present application; Figure 3 is a flowchart of the steps of an image processing method provided by an embodiment of the present application; Figure 4 It is a structural schematic diagram of a data denoising and enhancement collaborative representation learning device provided in an embodiment of the present application; Figure 5 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0019] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and understandable, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0020] There is a large amount of natural noise and adversarial noise in the image. Natural noise refers to the noise generated by natural factors that affects the accuracy of the task model. Climate factors such as fog, rain, waves and lighting conditions will have a significant impact on the performance of the sensor, resulting in a decrease in the quality of image and radar data. A large amount of natural noise is introduced, which may mask the characteristics of the target, making target detection and classification more difficult. Adversarial noise represents the noise used to deceive the task model to output wrong results. With the widespread adoption of various anti-detection technologies, these technologies include stealth materials, electronic interference and camouflage methods, which can effectively interfere with and deceive the task model and generate adversarial noise; adversarial noise not only increases the complexity of target detection and classification tasks, but also misleads the classification results, leading to wrong judgments and responses.

[0021] In the related technologies, optimization is often only performed on one aspect, natural noise and anti-noise. In terms of natural noise removal, common algorithms mainly include methods based on artificial feature denoising and methods based on deep learning; 1) The method based on artificial feature denoising regards the noisy image as adding a layer of white noise on the clean image. The Gaussian smoothing method uses Gaussian smoothing to denoise and forms a low-pass filter to filter out high-frequency interference components. On this basis, many methods of using filters to filter out noise are derived; while the variational law minimizes the difference between the image after denoising and the original image, which is equivalent to solving the Euler-Lagrange equation to solve this problem; the structural texture restoration method regards denoising as removing redundant text information in the image and reconstructing the image structure, and denoising is performed in a certain norm space by finding a lower bound of the bottleneck of image feature information; the wavelet transform method uses wavelet transform to add noise to the image, removes noise through inverse wavelet transform, and removes noise through different methods of soft and hard thresholds. 2) The deep learning-based methods are mainly divided into denoising methods based on convolutional neural networks and generative adversarial networks. The convolutional neural network-based methods, on the one hand, use the independence of pixels through self-supervised learning methods to find the mapping relationship between target pixels and input pixels in a self-supervised manner to achieve the denoising task. On the other hand, they extract the noise distribution after supervised learning network training and perform difference operations with the original input image to complete the task. The generative adversarial network-based methods use generators and discriminators to perform generative adversarial training, which on the one hand alleviates the problem of insufficient training image samples, and on the other hand enhances the learning effect.

[0022] In terms of adversarial noise learning, it is mainly divided into adversarial learning-based methods and purification-based methods; 1) The adversarial learning-based method mainly generates adversarial noise through attack methods and adds it to the original image. The training model can correctly complete tasks such as classification and detection of noisy images to achieve adaptation to noise. At present, the generation of attack image samples is mainly divided into gradient-based attacks, heuristic-based attacks and generative adversarial attacks. Gradient-based attacks use gradient information to interfere with task results in a targeted manner. Heuristic-based attacks use heuristic rules and strategies to tamper with task results. Generative adversarial attacks are achieved through generative adversarial networks. The generator learns the feature distribution of the image and generates false attack images. The discriminator judges the aggressiveness of the generated results, thereby generating sufficiently aggressive images. 2) The purification-based method uses a generative model to purify images with adversarial noise added. Compared with the adversarial learning-based method, it mainly removes the adversarial noise added to the image through an additional purification module, and predicts the expression after removing the adversarial noise for training, thereby achieving effective defense against attacks.

[0023] From the above analysis, it can be seen that the existing natural noise removal cannot effectively overcome the impact of adversarial noise. Although it can effectively enhance model performance, the model effect will be greatly attenuated and greatly affected when facing malicious adversarial attacks. Adversarial noise learning focuses on correctly classifying image samples with adversarial noise, but due to over-learning of adversarial image samples, it overfits the noise distribution. Although it shows good stability in the face of attacks, it suffers from a large loss in classification accuracy and significantly reduces overall performance. At the same time, the image samples themselves contain natural noise before adding adversarial noise, which will have a negative impact on subsequent training.

[0024] In order to overcome the limitations of related technologies, an embodiment of the present application provides a representation learning method that coordinates data denoising and enhancement. This method addresses the problem that the current adversarial training is generally lacking in adversarial noise fitting. It uses denoised image samples with natural noise removed to perform adversarial noise learning, thereby removing the influence of natural noise in the original image samples on the overall learning effect of adversarial noise, thereby improving the adversarial noise learning effect. In order to address the problem of reducing the overall performance due to adversarial noise, a purification-based adversarial noise learning method is used to directly predict and remove noise, thereby avoiding the occurrence of overfitting problems caused by excessive learning of adversarial noise image samples.

[0025] Reference Figure 1 As shown, Figure 1 1 is a flowchart of a method for characterizing learning data denoising and enhancement synergy provided in an embodiment of the present application. Figure 1 As shown, a data denoising and enhancement collaborative representation learning method provided in an embodiment of the present application may include steps S110 to S140: Step S110: performing natural noise denoising processing on the original image sample to obtain a denoised image sample, wherein the natural noise represents the noise generated by natural factors that affects the accuracy of the task model.

[0026] The original image sample may be an image obtained by a sensor device and obtained through preprocessing (e.g., cropping, image conversion) to obtain an image expression. There is natural noise in the original image sample, which will affect the accuracy of the task model. In order to remove the influence of the natural noise in the original image sample on the subsequent anti-noise learning effect, the original image sample is subjected to natural noise denoising to obtain a denoised image sample, which does not contain natural noise.

[0027] For each original image sample, the original image sample is processed based on the attention mechanism of the channel dimension to obtain natural noise, and the natural noise is removed from the original image sample to obtain a denoised image sample. In some embodiments, the processing of the original image sample based on the attention mechanism of the channel dimension is implemented by a squeeze-activation neural network. Specifically, for a dimension of The original image sample, in the squeeze module of the squeeze-activation neural network, will cross the spatial dimension The feature maps are aggregated to generate dimensions of channel dimension information; then in the activation link, each channel learns the activation of a specific image sample through a self-selected gate mechanism based on channel dependency, so that it learns to use global information and obtains the output result of the activation operation; finally, the output result of the activation operation is weighted using the channel information obtained by the squeezing module, thereby realizing the attention mechanism of the channel dimension and obtaining the denoised image sample.

[0028] Step S120: generating an adversarial noise image sample by adding adversarial noise to the denoised image sample, wherein the adversarial noise represents noise used to deceive the task model to output an erroneous result.

[0029] In an embodiment of the present application, a purification-based anti-noise learning method is used to directly predict and remove noise. After obtaining a denoised image sample, an anti-noise image sample is obtained by adding anti-noise to the denoised image sample.

[0030] Specifically, adversarial noise is added to the denoised image samples to generate adversarial noise image samples, including: constructing a loss function according to the true labels corresponding to the denoised image samples and the original image samples; taking the partial derivative of the loss function to obtain the gradient, and adding disturbance according to the changing direction of the gradient to obtain adversarial noise; adding the adversarial noise to the denoised image samples to generate adversarial noise image samples.

[0031] For example, the adversarial noise can be expressed as: , in, are the parameters of the task model, is a denoised image sample, is the true label corresponding to the original image sample, is the loss function, which increases the error by making the change consistent with the direction of the gradient. The function ensures that the direction of change is consistent with the direction of the gradient. By taking the partial derivative of the loss function, the change to the task result is maximized. It is represented as a perturbation as small as possible to ensure that the image changes slightly before and after adding adversarial noise.

[0032] Then add the adversarial noise to the denoised image sample to obtain the adversarial noise image sample: , in, represents the adversarial noise image sample, Represents a denoised image sample.

[0033] Step S130: purifying the noise in the anti-noise image sample based on the diffusion model to obtain a data enhanced image sample.

[0034] In an embodiment of the present application, after obtaining an adversarial noise image sample, a diffusion model is used to purify the noise in the adversarial noise image sample. The use of the diffusion model mainly benefits from the fact that the noise adding process in the diffusion process of the diffusion model can effectively submerge the malicious adversarial noise through Gaussian noise. At the same time, when a certain amount of Gaussian noise is added to make the distribution before and after the noise addition consistent, it is removed through the denoising process, thereby achieving the joint removal of the adversarial noise.

[0035] Step S140: training the task model based on the data enhanced image samples and the denoised image samples until the task model converges.

[0036] In an embodiment of the present application, the denoised image samples do not contain natural noise, the data enhanced image samples are obtained by purifying the adversarial noise image samples based on the diffusion process by the diffusion module, and the data enhanced image samples do not contain adversarial noise in addition to containing natural noise. The task model is trained based on the data enhanced image samples and the denoised image samples, so that the task model can learn the characteristics of different image samples.

[0037] The task model refers to a model used to perform a specific task, and the task model may be a target detection model, a target classification model, etc. Based on the data-enhanced image samples and the denoised image samples, training the task model means: constructing a loss value based on the data-enhanced image samples and the denoised image samples, and updating the model parameters of the task model based on the loss value, and obtaining a trained task module when the task model converges.

[0038] In the above process, the denoised image samples with natural noise removed are used for adversarial noise learning, which removes the influence of natural noise in the original image samples on the overall learning effect of adversarial noise, thereby improving the learning effect of adversarial noise; and, using the purification-based adversarial noise learning method, the noise is directly predicted and removed, avoiding the occurrence of overfitting problems caused by over-learning of adversarial noise image samples, thereby reducing the overall performance degradation caused by adversarial noise. In this way, a data denoising and enhancement collaborative representation learning method is implemented that effectively enhances the robustness of data expression and can better retain the original performance and accuracy of the task model.

[0039] The data denoising and enhancement collaborative representation learning method provided in the embodiment of the present application is described in Sections 1.1, 1.2 and 1.3 respectively.

[0040] 1.1 Task model training process: In combination with the above embodiments, in one implementation, the embodiment of the present application further provides a data denoising and enhancement collaborative representation learning method. In this method, the above step S140 of "training the task model based on the data enhanced image sample and the denoised image sample until the task model converges" may specifically include steps S140-1 to S140-5: Step S140 - 1 : Fusing the data enhanced image sample and the denoised image sample to obtain a fused image sample.

[0041] Step S140 - 2 : Input the fused image sample into the task model to obtain a task processing result.

[0042] Step S140-3: Obtain a first loss based on the difference between the task processing result and the true label corresponding to the original image sample.

[0043] Step S140 - 4 : obtaining a second loss according to the difference between the denoised image sample and the fused image sample, and obtaining a third loss according to the difference between the data enhanced image sample and the fused image sample.

[0044] Step S140 - 5 : updating the model parameters of the task model according to the total loss obtained by the first loss, the second loss and the third loss.

[0045] In an embodiment of the present application, the task processing result is determined based on the downstream tasks of the task model. For example, the task result may be a target detection result, or a target classification result, etc.

[0046] For example, the total loss It is expressed as: , in, represents the first loss, through which the task model is optimized for accurate task processing; represents the second loss, through which the consistency between the denoised image samples and the fused image samples is ensured; represents the third loss, through which the consistency between the data enhanced image samples and the fused image samples is ensured; is an adjustable hyperparameter used to control the weight of the loss.

[0047] In this way, the task model is trained based on fused image samples, denoised image samples and adversarial noise image samples. Since the denoised image samples do not contain natural noise, the denoised image samples with natural noise removed are used for adversarial noise learning, which removes the influence of natural noise in the original image samples on the overall learning effect of adversarial noise. In addition, the purification-based adversarial noise learning method is used to directly predict and remove the noise, avoiding the occurrence of overfitting problems caused by over-learning of adversarial noise image samples, thereby reducing the overall performance degradation caused by adversarial noise.

[0048] 1.2 Perform natural noise denoising on the original image samples: In combination with the above embodiments, in one implementation, the embodiment of the present application further provides a data denoising and enhancement collaborative representation learning method. In this method, the "performing natural noise denoising on the original image sample to obtain a denoised image sample" in the above step S110 may specifically include steps S110-1 to S110-5: Step S110 - 1 : Obtain input representations of multiple attention heads based on the original image sample.

[0049] Step S110 - 2 : performing a multi-head squeezing operation on the input representations of the multiple attention heads to obtain channel features of the multiple attention heads.

[0050] Step S110-3: performing an activation operation on the channel features of the multiple attention heads to obtain activation operation results of the multiple attention heads, wherein the activation operation refers to expressing and extracting a nonlinear relationship between the channel features of the multiple attention heads.

[0051] Step S110-4: Fuse the channel features of the multiple attention heads and the activation operation results of the multiple attention heads, and obtain natural noise based on the fusion results.

[0052] Step S110 - 5 : removing the natural noise from the original image sample to obtain a denoised sample.

[0053] In an embodiment of the present application, in order to obtain more accurate denoised image samples, a multi-head squeeze-activation neural network model is proposed to implement a more general and efficient channel dimension attention mechanism. The original image samples are processed by the multi-head squeeze-activation neural network model to obtain more accurate natural noise, thereby obtaining more accurate denoised image samples.

[0054] Specifically, for each original image sample, step S110-1 is first performed to obtain input representations of multiple attention heads. For example, for a dimension of The original image samples are transformed to obtain the input representation of each attention head; the input representation of multiple attention heads It can be expressed as: , in, represents the input representation of the first attention head, represents the input representation of the second attention head, Represents the input representation of the nth attention head.

[0055] Next, step S110-2 is executed. In the squeezing phase of the multi-head squeezing-activation neural network model, the input representations of multiple attention heads are subjected to multi-head squeezing operations to obtain the channel features of multiple attention heads. The channel features of multiple attention heads can be expressed as: , Where c represents the dimension of the input channel; n represents the number of attention heads; represents the pixel at the i-th row and j-th column in the input representation; Represents the channel features on a channel dimension; Represents the channel features of multiple attention heads, and its dimension is That is to say, a multi-head squeezing operation is performed on the input representations of multiple attention heads, and the compressed features of different channel information (i.e., the channel features of multiple attention heads) can be obtained for n groups of inputs.

[0056] Then, execute step S110-3 to obtain the activation operation results of multiple attention heads. Specifically, use the first fully connected layer to compress the channel features of the multiple attention heads to obtain channel features of target channel dimensions; use the nonlinear activation layer and the second fully connected layer to restore the number of channels of the target channel dimensions to obtain restored channel features; use the target activation function to process the restored channel features to obtain the activation operation results of multiple attention heads.

[0057] In the embodiment of the present application, the activation module of the multi-head squeeze-activation neural network model is used to realize the comprehensive expression of the channel features of multiple attention heads and extract non-linear relationships and non-mutually exclusive relationships. For example, the activation operation results of multiple attention heads are expressed as: , in, is a target activation function. In some embodiments, the target activation function may be a sigmoid function; is a nonlinear activation layer. In some embodiments, the activation function may be a LeakyReLU activation function; is the weight matrix of the first fully connected layer; is the weight matrix of the second fully connected layer.

[0058] Through the first fully connected layer The channel features of multiple attention heads are transformed into The channel features of the target channel dimension are restored to The restored channel features. Finally, through the target activation function , get the activation operation results of multiple attention heads

[0059] By executing step S110-4, the channel features of the multiple attention heads and the activation operation results of the multiple attention heads are fused, and natural noise is obtained according to the fusion result. For example, the fusion of the channel features of the multiple attention heads and the activation operation results of the multiple attention heads can be expressed as: , in, represents the fusion operation, Indicates the fusion result.

[0060] Then, according to the fusion result and the original image sample, the natural noise is obtained. It can be expressed as: , Finally, the natural noise is removed from the original image sample to obtain a denoised image sample, which can be expressed as: , in, Represents a denoised image sample.

[0061] In the above process, a more general and efficient channel dimension attention mechanism is implemented based on a multi-head squeeze-activation neural network model, which solves the problems of coarse channel granularity and poor cross-effect in related denoising schemes, thereby obtaining more accurate denoised image samples.

[0062] 1.3 Purifying the noise in the adversarial noise image sample based on the diffusion model: In the embodiment of the present application, in order to solve the problems of overfitting adversarial image samples, generalization bias, and poor performance on normal data sets in adversarial training, an adversarial purification method is used. Adversarial purification aims to remove adversarial perturbations from adversarial image samples to protect the classifier from attacks by unseen threats. This method does not rely on the specific assumptions of the attack form and classification model, does not require retraining the classifier, and only requires applying a purification step before the classifier. Therefore, it can be used to protect existing classifiers against unknown attacks and maintain performance on normal image samples as much as possible.

[0063] In this embodiment of the present application, the noise in the anti-noise image sample is purified based on a diffusion model, and the time step of the noise adding process can be expressed as: , That is, in the forward process, the distribution of clean data With adversarial perturbation data distribution When they are closest, that is, the derivative of the KL divergence with respect to t is less than 0, the problem of adversarial purification can be formalized as follows: , in, Represents the optimal reconstructed image sample, and regards the adversarial noise image sample as a perturbation measurement of the original clean image sample. It is assumed that the clean image sample is generated by a prior probability distribution, that is, Given an adversarial noise image sample , the goal is to restore the original image sample , so that the posterior distribution Maximization. The defense against adversarial image samples is solved by optimizing the problem, that is, finding the original image sample that can maximize the posterior distribution, which is achieved by maximizing the log-likelihood of the posterior distribution. Therefore, a variational posterior distribution is introduced To approximate the true posterior distribution in the original optimization objective , where the variational upper bound is: , in, Indicates the use of parameters Reconstructed clean data distribution.

[0064] After simplification, the loss function is expressed as follows: , in, The output of the input image sample after adding Gaussian noise and passing through the diffusion model is calculated by With the input image sample The loss is thus obtained, and the diffusion model is trained using this loss function to obtain data-enhanced image samples.

[0065] In combination with the above embodiments, in one implementation, the embodiment of the present application further provides a data denoising and enhancement collaborative representation learning method. In this method, the above step S130 of "purifying the noise in the anti-noise image sample based on the diffusion model to obtain a data enhanced image sample" may include steps S130-1 to S130-3: Step S130-1: In each time step, the anti-noise image sample is subjected to noise adding processing according to the randomly sampled Gaussian noise to obtain the noise added image sample, and the input image sample of the current time step is predicted according to the noise added image sample of each time step, and the process is repeated for multiple time steps until the input image sample of the last time step is obtained; Step S130-2: performing multiple iterations in the above manner to obtain a trained diffusion model; Step S130 - 3 : using the trained diffusion model, performing noise addition and denoising processing on the anti-noise image sample to obtain a data enhanced image sample.

[0066] In the embodiment of the present application, the training of the diffusion model is simplified to the form of a loss function of the noise added at each time step and the actual added noise, so as to train the diffusion model. In this way, a smaller time step is selected to ensure that the added noise is as small as possible but sufficient to drown out the adversarial noise, so that it can be removed together with the adversarial noise during the denoising process.

[0067] Finally, the trained diffusion model is used to perform noise addition and denoising processing on the anti-noise image samples, thereby obtaining data enhanced image samples.

[0068] In this way, the diffusion model predicts the added noise corresponding to each time step based on the noise-added image sample in each time step, and then obtains the clean image sample corresponding to the last time step through the prediction processing of multiple time steps, that is, the data enhanced image sample.

[0069] Furthermore, in step S130-1, "in each time step, according to the randomly sampled Gaussian noise, the anti-noise image sample is subjected to noise adding processing to obtain the noise added image sample, and according to the noise added image sample of each time step, the input image sample of the current time step is predicted, and the process is repeated for multiple time steps until the input image sample of the last time step is obtained", includes steps B1 to B7: Step B1: Randomly sample from Gaussian distribution to obtain Gaussian noise.

[0070] For example, the Gaussian distribution can be expressed as , randomly sample the Gaussian distribution to obtain Gaussian noise .

[0071] Step B2: Add the Gaussian noise to the input image samples of the previous time step and the adversarial noise image samples of the previous time step respectively to obtain the input image samples of the current time step and the adversarial noise image samples of the current time step.

[0072] For example, the input pattern sample at the current time step can be expressed as: , Gaussian noise added image samples can be expressed as: , in, represents the input image sample at the previous time step, represents the input image sample at the current time step, represents the adversarial noise image sample at the current time step, Represents the anti-noise image sample of the previous time step, where the anti-noise image sample of the first time step is the anti-noise image sample generated in step S120 , represents the correlation coefficient at time t.

[0073] Step B3: Input the adversarial noise image sample of the current time step into the denoising module of the diffusion model to obtain a model denoised image sample.

[0074] Specifically, the denoising module of the diffusion model predicts the added noise of the adversarial noise image sample at the current time step, and removes the predicted added noise from the adversarial noise image sample at the current time step, thereby obtaining the model denoised image sample.

[0075] Step B4: construct a fourth loss based on the model denoising image samples and the input image samples of the current time step.

[0076] For example, the fourth loss is expressed as ,in, Denoise image samples for the model.

[0077] Step B5: Based on the fourth loss, calculate the gradient of the input image sample at the current time step, and update the diffusion model based on the gradient.

[0078] For example, the gradient of the input image sample at the current time step is It is expressed as: , Step B6: Repeat the above steps for multiple time steps until the input image sample of the last time step is obtained.

[0079] In the embodiment of the present application, the method of step B1 to step B6 is followed and the input image samples of each time step are calculated. When the input image samples of the last time step are obtained, it indicates that the current round of iteration is completed. Then the next round of iteration can be started until the training end condition is met, and a trained diffusion model is obtained.

[0080] The data denoising and enhancement collaborative representation learning method provided in the embodiment of the present application is implemented based on the data denoising and enhancement collaborative representation learning model. For example, Figure 2 It is a structural schematic diagram of a data denoising and enhancement collaborative representation learning model provided in an embodiment of the present application, wherein the data denoising and enhancement collaborative representation learning model includes a denoising module and a robust enhancement module; the denoising module is used to obtain denoised image samples by performing natural noise denoising on original image samples; and the denoising module processes the original image samples based on a multi-head squeeze-activation neural network model to obtain natural noise; the natural noise is removed from the original image samples to obtain denoised image samples.

[0081] The robust enhancement module is used to add adversarial noise to the denoised image samples to generate adversarial noise image samples, and purify the noise in the adversarial noise image samples based on the diffusion model to obtain data enhanced image samples, and finally fuse the data enhanced image samples and the denoised image samples to obtain fused image samples. Finally, the fused image samples output by the robust enhancement module can be used to train the task model.

[0082] It is understandable that the data denoising and enhancement collaborative representation learning method provided in the embodiment of the present application can be applied to target detection, target classification and other fields. For example, for ship target detection and classification, since the images collected by the sensor equipment contain a large amount of natural noise and adversarial noise, the accuracy of ship target detection and classification can be improved based on this method.

[0083] Furthermore, in order to better illustrate the performance of the data denoising and enhancement collaborative representation learning method in the embodiment of the present application, by trying on multiple data sets, the robustness of multiple tasks in the noise-adverse scenario is improved. Compared with the traditional method of removing natural noise, this method can achieve a significant improvement in robustness; and compared with other noise-adverse learning methods, this method achieves an effective improvement in performance and reduces the loss of model performance caused by learning to combat noise. A large number of experiments have been conducted on ship target detection and classification tasks, and the experimental results have proved the effectiveness of the technology of the present application.

[0084] Specifically, for the ship classification FGSC 23 dataset and the ship target detection HRSC 2016 dataset, the test results are shown in Table 1. The four task models (VGG16, ResNet50, MobileNet, Xception) are trained using the proposed method, FGSM Adv Train, and C&W Adv Train, respectively. It can be seen from Table 1 that the task model trained based on the proposed method has better performance.

[0085] Table 1 Test results of ship target detection and classification tasks

[0086] Reference Figure 3 As shown, Figure 3 1 is a flowchart of the steps of an image processing method provided by an embodiment of the present application. Figure 3 As shown, an image processing method provided in an embodiment of the present application may include steps S310 to S320: Step S310: Acquire the image to be processed.

[0087] Step S320: input the image to be processed into the task model for processing to obtain a task processing result, wherein the task model is trained by the data denoising and enhancement collaborative representation learning method described in an embodiment of the present application.

[0088] In the embodiment of the present application, the image to be processed is processed using a task model. Since the task model is trained by the data denoising and enhancement collaborative representation learning method described in the embodiment of the present application, the data denoising and enhancement collaborative representation learning method uses denoised image samples with natural noise removed for noise-adverse learning, which removes the influence of natural noise in the original image samples on the overall learning effect of noise-adverse learning, thereby improving the noise-adverse learning effect; and, using a purification-based noise-adverse learning method, the noise is directly predicted and removed, avoiding the occurrence of overfitting problems caused by over-learning of noise-adverse image samples, thereby reducing the overall performance degradation caused by noise-adverse learning. Therefore, accurate task processing results can be obtained based on the task model.

[0089] The present application also provides a data denoising and enhancement collaborative representation learning device, referring to Figure 4 As shown, Figure 4 : is a schematic diagram of a data denoising and enhancement collaborative representation learning device provided in an embodiment of the present application, the device comprising: A denoising processing module 410 is used to perform natural noise denoising on the original image sample to obtain a denoised image sample, wherein the natural noise represents the noise generated by natural factors that affects the accuracy of the task model; A noise adding module 420, configured to generate an adversarial noise image sample by adding adversarial noise to the denoised image sample, wherein the adversarial noise represents noise used to deceive the task model to output an erroneous result; A noise purification module 430, configured to purify the noise in the anti-noise image sample based on a diffusion model to obtain a data enhanced image sample; The model training module 440 is used to train the task model based on the data enhanced image samples and the denoised image samples until the task model converges.

[0090] In an optional embodiment, the model training module includes: A sample fusion module, used for fusing the data enhanced image sample and the denoised image sample to obtain a fused image sample; A task processing result module, used for inputting the fused image sample into the task model to obtain a task processing result; A result loss module, used for obtaining a first loss according to a difference between the task processing result and a true label corresponding to the original image sample; An image sample loss module, configured to obtain a second loss according to a difference between the denoised image sample and the fused image sample, and to obtain a third loss according to a difference between the data enhanced image sample and the fused image sample; A parameter updating module is used to update the model parameters of the task model according to the total loss obtained by the first loss, the second loss and the third loss.

[0091] In an optional embodiment, the denoising processing module includes: An acquisition module, used for acquiring input representations of multiple attention heads according to the original image sample; A squeezing operation module, performing a multi-head squeezing operation on the input representations of the multiple attention heads to obtain channel features of the multiple attention heads; An activation operation module, configured to perform an activation operation on the channel features of the multiple attention heads to obtain activation operation results of the multiple attention heads, wherein the activation operation refers to expressing and extracting a nonlinear relationship between the channel features of the multiple attention heads; A result fusion module, used for fusing the channel features of the multiple attention heads and the activation operation results of the multiple attention heads, and obtaining natural noise according to the fusion result; The first removal module is used to remove the natural noise from the original image sample to obtain a denoised image sample.

[0092] In an optional embodiment, the noise purification module includes: A prediction module is used to add noise to the anti-noise image samples according to the randomly sampled Gaussian noise in each time step to obtain noise-added image samples, and predict the input image samples of the current time step according to the noise-added image samples of each time step, and repeat for multiple time steps until the input image samples of the last time step are obtained; An iteration module, used to perform multiple iterations in the above manner to obtain a trained diffusion model; The second processing module is used to use the trained diffusion model to perform noise addition and denoising on the anti-noise image sample to obtain a data enhanced image sample.

[0093] In an optional embodiment, the prediction module includes: A sampling module is used to randomly sample from a Gaussian distribution to obtain Gaussian noise; A noise adding module, used to add the Gaussian noise to the input image samples of the previous time step and the adversarial noise image samples of the previous time step, respectively, to obtain the input image samples of the current time step and the adversarial noise image samples of the current time step; A first input module, used for inputting the anti-noise image sample of the current time step into the denoising module of the diffusion model to obtain a model denoised image sample; a second loss construction module, which calculates the gradient of the input image sample at the current time step based on the fourth loss, and updates the diffusion model based on the gradient; The repetition module is used to repeat the above steps for multiple time steps until the input image sample of the last time step is obtained.

[0094] The present application also provides an electronic device, referring to Figure 5 , Figure 5 Schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 5 As shown, the electronic device 500 includes: a memory 510 and a processor 520. The memory 510 and the processor 520 are connected via a bus communication. A computer program is stored in the memory 510. The computer program can be run on the processor 520 to implement the steps of the data denoising and enhancement collaborative representation learning method described in the embodiment of the present application.

[0095] The embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the data denoising and enhancement collaborative representation learning method described in the embodiment of the present application are implemented.

[0096] The embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the data denoising and enhancement collaborative representation learning method described in the embodiment of the present application.

[0097] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0098] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods and devices according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0099] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0100] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0101] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0102] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0103] The above is a detailed introduction to a data denoising and enhancement collaborative representation learning method provided by the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A data denoising and enhancement collaborative representation learning method, characterized in that: The method comprises: Performing natural noise denoising on the original image sample to obtain a denoised image sample, wherein the natural noise represents the noise generated by natural factors that affects the accuracy of the task model; Generate an adversarial noise image sample by adding adversarial noise to the denoised image sample, wherein the adversarial noise represents noise used to deceive the task model to output an erroneous result; Purifying the noise in the anti-noise image sample based on a diffusion model to obtain a data enhanced image sample; The task model is trained based on the data enhanced image samples and the denoised image samples until the task model converges.

2. The method according to claim 1, characterized in that Training a task model based on the data-enhanced image samples and the denoised image samples until the task model converges, including: Fusing the data enhanced image sample and the denoised image sample to obtain a fused image sample; Inputting the fused image sample into the task model to obtain a task processing result; Obtaining a first loss according to a difference between the task processing result and a true label corresponding to the original image sample; Obtaining a second loss based on a difference between the denoised image sample and the fused image sample, and obtaining a third loss based on a difference between the data enhanced image sample and the fused image sample; The model parameters of the task model are updated according to the total loss obtained by the first loss, the second loss and the third loss.

3. The method according to claim 1, characterized in that The original image samples are subjected to natural noise denoising to obtain denoised image samples, including: According to the original image sample, obtaining input representations of multiple attention heads; Performing a multi-head squeezing operation on the input representations of the multiple attention heads to obtain channel features of the multiple attention heads; Performing an activation operation on the channel features of the multiple attention heads to obtain activation operation results of the multiple attention heads, wherein the activation operation refers to expressing and extracting a nonlinear relationship between the channel features of the multiple attention heads; Fusing the channel features of the multiple attention heads and the activation operation results of the multiple attention heads, and obtaining natural noise according to the fusion result; The natural noise is removed from the original image sample to obtain a denoised sample.

4. The method according to claim 1, characterized in that: Purifying the noise in the anti-noise image sample based on the diffusion model to obtain a data enhanced image sample, including: In each time step, the adversarial noise image samples are denoised according to the randomly sampled Gaussian noise to obtain noise-added image samples, and the input image samples of the current time step are predicted according to the noise-added image samples of each time step, and the process is repeated for multiple time steps until the input image samples of the last time step are obtained; Perform multiple iterations in the above manner to obtain a trained diffusion model; The trained diffusion model is used to perform noise addition and denoising on the anti-noise image samples to obtain data enhanced image samples.

5. The method according to claim 4, characterized in that In each time step, the adversarial noise image samples are denoised according to the randomly sampled Gaussian noise to obtain noise-added image samples, and the input image samples of the current time step are predicted according to the noise-added image samples of each time step. This is repeated for multiple time steps until the input image samples of the last time step are obtained, including: Randomly sample from a Gaussian distribution to obtain Gaussian noise; Adding the Gaussian noise to the input image sample of the previous time step and the adversarial noise image sample of the previous time step respectively, to obtain the input image sample of the current time step and the adversarial noise image sample of the current time step; Inputting the anti-noise image sample of the current time step into the denoising module of the diffusion model to obtain a model denoised image sample; Constructing a fourth loss according to the model denoised image samples and the input image samples at the current time step; Based on the fourth loss, calculating the gradient of the input image sample at the current time step, and updating the diffusion model based on the gradient; Repeat the above steps for multiple time steps until the input image sample of the last time step is obtained.

6. An image processing method, characterized in that: The method comprises: Get the image to be processed; The image to be processed is input into a task model for processing to obtain a task processing result, and the task model is trained by the data denoising and enhancement collaborative representation learning method described in any one of claims 1-5.

7. A data denoising and enhancement collaborative representation learning device, characterized in that: The device comprises: A denoising processing module is used to perform natural noise denoising on the original image sample to obtain a denoised image sample, wherein the natural noise represents the noise generated by natural factors that affects the accuracy of the task model; A noise adding module, used for generating an adversarial noise image sample by adding adversarial noise to the denoised image sample, wherein the adversarial noise represents noise used to deceive the task model to output an erroneous result; A noise purification module, used for purifying the noise in the anti-noise image sample based on a diffusion model to obtain a data enhanced image sample; The model training module is used to train the task model based on the data enhanced image samples and the denoised image samples until the task model converges.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the data denoising and enhancement collaborative representation learning method described in any one of claims 1 to 5 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the data denoising and enhancement collaborative representation learning method described in any one of claims 1 to 5 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the data denoising and enhancement collaborative representation learning method as described in any one of claims 1 to 5 are implemented.

Images