Unmanned system image multi-noise interference suppression method based on self-supervised continuous learning

Abstract

The application provides an unmanned system image multi-noise interference suppression method based on self-supervised continuous learning, comprising: estimating the noise type and / or noise intensity of a target image to obtain an estimated noise type and / or an estimated noise intensity; selecting a corresponding self-supervised continuous learning neural network branch from the self-supervised continuous learning neural network to perform denoising processing on the target image; the training process of the network comprises: dividing multi-task for noise images of at least one type and / or intensity, obtaining a noise image dataset, with the same type and / or intensity of noise image corresponding to one task; performing a first process on the noise image dataset of at least one task respectively, and taking the self-supervised continuous learning neural network updated in the last first process as the self-supervised continuous learning neural network completed training; and enabling the network to process noise images of different intensities or different types, and improving the accuracy of image denoising.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method for suppressing multiple noise interferences in images of unmanned systems based on self-supervised continuous learning. Background Technology

[0002] In practical applications, unmanned platforms often carry multiple types of sensors, and images acquired by different types of sensors contain different types and intensities of noise. For example, images captured by optical sensors are typically affected by additive noise, while images acquired by synthetic aperture radar are severely affected by multiplicative noise. Since the presence of noise degrades image quality and adversely affects subsequent tasks such as target detection and recognition, image denoising is a crucial task in fields such as artificial intelligence.

[0003] Traditional image denoising methods, including spatial filtering, multi-scale analysis, nonlocal mean filtering, and variational methods, are simple to implement, easy to understand, and highly interpretable. However, they all have some drawbacks, such as poor preservation of image edges and details, boundary oscillations, speckle effects, artifacts, or loss of weak targets. To overcome the shortcomings of traditional denoising methods, deep learning-based image denoising methods have developed rapidly and achieved better denoising results. Currently, most deep neural network-based denoising methods employ supervised learning, requiring pairs of clean and noisy images to train the network. The network parameters are optimized by minimizing the loss function between the denoised image and the corresponding clean image. However, in practical applications, clean images corresponding to noisy images are often difficult to obtain, resulting in lower accuracy of deep neural network-based denoising. Summary of the Invention

[0004] This invention provides a method for suppressing multiple noise interferences in images of unmanned systems based on self-supervised continuous learning, in order to improve the accuracy of image denoising.

[0005] This invention provides a method for suppressing multiple noise interferences in images of unmanned systems based on self-supervised continuous learning, comprising:

[0006] By fitting a density function, the noise type and / or noise intensity of the target image of the unmanned system are estimated, and the estimated noise type and / or estimated noise intensity of the target image are obtained.

[0007] From the self-supervised continuous learning neural network that has been trained, select the branch of the self-supervised continuous learning neural network that corresponds to the estimated noise type and / or estimated noise intensity.

[0008] Based on the self-supervised continuous learning neural network branch, the target image is denoised.

[0009] The training process of the self-supervised continuous learning neural network includes:

[0010] A task is assigned to a noise image of the same noise type and / or noise intensity. The noise images of at least one noise type and / or noise intensity are divided into multiple tasks to obtain a noise image dataset for at least one task.

[0011] The first process is performed on the noisy image dataset for at least one task, and the self-supervised continuous learning neural network updated in the last first process is used as the self-supervised continuous learning neural network that has completed training.

[0012] The first process includes:

[0013] Based on the noisy image dataset corresponding to the task of the current first process and the current self-supervised continuous learning neural network, training is performed, and the first part layer and the second part layer in the trained self-supervised continuous learning neural network are determined. The correlation scores of the combined gating and convolutional kernel elements corresponding to the first part layer are lower than the correlation scores of the combined gating and convolutional kernel elements corresponding to the second part layer. The first part layer does not include the output layer of the self-supervised continuous learning neural network, and the second part layer is the layer in the self-supervised continuous learning neural network other than the first part layer and the output layer.

[0014] Set the convolution kernels of the first layer in the trained self-supervised continuous learning neural network to zero to obtain the pruned self-supervised continuous learning neural network.

[0015] Based on the noisy image dataset corresponding to the task in the current first process, the parameters of the second layer in the pruned self-supervised continuous learning neural network are adjusted to obtain the parameter-adjusted self-supervised continuous learning neural network.

[0016] The parameters of the convolutional kernels and the output layer of the second part of the self-supervised continuous learning neural network are frozen after parameter adjustment. A mask is added to the parameters of the convolutional kernels and the output layer of the second part of the self-supervised continuous learning neural network after parameter adjustment to obtain an updated self-supervised continuous learning neural network. The updated self-supervised continuous learning neural network is used for training on the noisy image dataset of the next task in the next first process.

[0017] According to the present invention, a method for suppressing multiple noise interferences in images of unmanned systems based on self-supervised continuous learning further includes, before the first execution of the first process:

[0018] Adjust the structure of the U-Net network and initialize the adjusted U-Net network to obtain an initialized self-supervised continuous learning neural network;

[0019] The adjusted U-Net network includes an encoder, a decoder, and skip connections. The encoder consists of convolutional layers, a LeakyReLU activation function, and a max-pooling downsampling module. The decoder consists of bilinear interpolation upsampling, convolution, and activation function modules. The skip connections are used to connect some features of the encoder and the decoder together.

[0020] According to the present invention, a method for suppressing multiple noise interference in images of unmanned systems based on self-supervised continuous learning is provided. The initialized self-supervised continuous learning neural network includes: 20 convolutional layers, 5 downsampling layers, and 5 upsampling layers, wherein the last 5 convolutional layers of the 20 convolutional layers are the output layers; the structure of the gating module of the self-supervised continuous learning neural network is represented as follows:

[0021]

[0022] in This represents the gating module at layer l corresponding to task t in at least one task, ⊙ represents channel multiplication, and h l h represents the input feature map of the l-th layer. l+1 This represents the output feature map of the l-th layer. This represents the sparse feature map output by the l-th layer. This represents the gating value of the i-th channel in the l-th layer. 0 indicates that the channel parameter is masked, and 1 indicates that the channel parameter is selected.

[0023] According to the present invention, a method for suppressing multiple noise interferences in images of unmanned systems based on self-supervised continuous learning is provided, the method further comprising:

[0024] Before executing the first process corresponding to any one of the at least one tasks, the noisy image dataset of the task is preprocessed to obtain multiple sets of similar noisy images after random neighborhood downsampling, which are used as inputs for the training and parameter adjustment in the first process corresponding to the task.

[0025] According to the present invention, a method for suppressing multiple noise interferences in images of unmanned systems based on self-supervised continuous learning preprocesses the noisy image dataset of the task to obtain multiple sets of similar noisy images after random neighborhood downsampling, including:

[0026] The pixel value of the target pixel in the noisy image with multiplicative noise in the noisy image dataset is set to 1, and the noisy image with multiplicative noise is normalized to obtain the processed noisy image dataset, wherein the target pixel is a pixel with a pixel value less than 1.

[0027] Random neighborhood downsampling is performed on the noisy images in the processed noisy image dataset to obtain multiple sets of similar noisy images after random neighborhood downsampling.

[0028] According to the self-supervised continuous learning-based image multi-noise interference suppression method for unmanned systems provided by the present invention, in the first processes other than the last first process, the training loss function is expressed as:

[0029]

[0030] In the last first process, the training loss function is expressed as:

[0031]

[0032] in, To reconstruct the loss, For consistency regularization loss, The loss is sparse regularization loss, where γ1 is a hyperparameter controlling the degree of regularization and γ2 is a hyperparameter controlling the degree of sparsity.

[0033] The present invention also provides a device for suppressing multiple noise interferences in images of unmanned systems based on self-supervised continuous learning, comprising:

[0034] The noise estimation module is used to estimate the noise type and / or noise intensity of the target image of the unmanned system by fitting a density function, so as to obtain the estimated noise type and / or estimated noise intensity of the target image.

[0035] The denoising network branch selection module is used to select a branch of the self-supervised continuous learning neural network that corresponds to the estimated noise type and / or estimated noise intensity from the self-supervised continuous learning neural network that has completed training.

[0036] A denoising module is used to perform denoising processing on the target image based on the self-supervised continuous learning neural network branch.

[0037] It also includes a self-supervised continuous learning neural network training module, used for:

[0038] A task is assigned to a noise image of the same noise type and / or noise intensity. The noise images of at least one noise type and / or noise intensity are divided into multiple tasks to obtain a noise image dataset for at least one task.

[0039] The first process is performed on the noisy image dataset for at least one task, and the self-supervised continuous learning neural network updated in the last first process is used as the self-supervised continuous learning neural network that has completed training.

[0040] The first process includes:

[0041] Based on the noisy image dataset corresponding to the task of the current first process and the current self-supervised continuous learning neural network, training is performed, and the first part layer and the second part layer in the trained self-supervised continuous learning neural network are determined. The correlation scores of the combined gating and convolutional kernel elements corresponding to the first part layer are lower than the correlation scores of the combined gating and convolutional kernel elements corresponding to the second part layer. The first part layer does not include the output layer of the self-supervised continuous learning neural network, and the second part layer is the layer in the self-supervised continuous learning neural network other than the first part layer and the output layer.

[0042] Set the convolution kernels of the first layer in the trained self-supervised continuous learning neural network to zero to obtain the pruned self-supervised continuous learning neural network.

[0043] Based on the noisy image dataset corresponding to the task in the current first process, the parameters of the second layer in the pruned self-supervised continuous learning neural network are adjusted to obtain the parameter-adjusted self-supervised continuous learning neural network.

[0044] The parameters of the convolutional kernels and the output layer of the second part of the self-supervised continuous learning neural network are frozen after parameter adjustment. A mask is added to the parameters of the convolutional kernels and the output layer of the second part of the self-supervised continuous learning neural network after parameter adjustment to obtain an updated self-supervised continuous learning neural network. The updated self-supervised continuous learning neural network is used for training on the noisy image dataset of the next task in the next first process.

[0045] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the image multi-noise interference suppression method for unmanned systems based on self-supervised continuous learning as described above.

[0046] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image multi-noise interference suppression method for unmanned systems based on self-supervised continuous learning as described above.

[0047] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the image multi-noise interference suppression method for unmanned systems based on self-supervised continuous learning as described above.

[0048] The present invention provides a method and apparatus for suppressing multiple noise interferences in images of unmanned systems based on self-supervised continuous learning. During the model training phase, different types and / or different intensities of noise images acquired by the unmanned platform are divided into multiple tasks. Each noise type and / or noise intensity is set as a task and used as input for training the denoising model. Multiple tasks are learned sequentially on the same network, enabling the network to process noise images of different intensities or types. Furthermore, through self-supervised continuous learning, the network continuously learns from newly acquired data and avoids forgetting previously learned tasks, thereby improving the accuracy of image denoising. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0050] Figure 1 This is a flowchart illustrating the method for suppressing multiple noise interferences in images of unmanned systems based on self-supervised continuous learning provided by the present invention.

[0051] Figure 2 This is a schematic diagram of the original image and the image after adding noise used in the test of the multi-noise interference suppression method for unmanned platforms based on self-supervised continuous learning provided by the present invention.

[0052] Figure 3 This is a schematic diagram of the denoising results of the self-supervised continuous learning-based multi-noise interference suppression method for unmanned platforms provided by the present invention;

[0053] Figure 4 This is a flowchart illustrating the multi-noise interference suppression method for unmanned platforms based on self-supervised continuous learning provided by the present invention.

[0054] Figure 5 This is a flowchart illustrating the image multi-noise interference suppression device for unmanned systems based on self-supervised continuous learning provided by the present invention.

[0055] Figure 6 A schematic diagram of the physical structure of an electronic device is provided. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0057] Existing deep neural network-based image denoising methods require supervised learning using pairs of clean and noisy images during model training, but clean images are difficult to obtain. Therefore, self-supervised deep network denoising methods have been proposed. These methods aim to learn a network model from noisy training data using image pairs with identical content but independent noise, thus transforming the requirement for clean images into the requirement for two independent noisy images of the same scene. Furthermore, deep network denoising methods require separate model learning for specific types or intensities of noise; that is, a single network model can only handle a single type or intensity of noise, and it is not possible to use the same network to handle multiple types or intensities of noise.

[0058] In summary, existing image denoising methods based on deep neural networks have two problems: First, training the model requires supervised learning with pairs of clean and noisy images, but clean images are difficult to obtain in practical applications; second, related technologies can only use a single network to process image noise of a single intensity or type, and cannot continuously learn different intensities or types of noise on the same network.

[0059] The following description, in conjunction with the accompanying drawings, describes the method and apparatus for suppressing multiple noise interferences in images of unmanned systems based on self-supervised continuous learning.

[0060] Figure 1 This is a flowchart illustrating the image multi-noise interference suppression method for unmanned systems based on self-supervised continuous learning provided by the present invention, as shown below. Figure 1 As shown, the method includes the following steps:

[0061] Step 100: Estimate the noise type and / or noise intensity of the target image of the unmanned system by fitting a density function, and obtain the estimated noise type and / or estimated noise intensity of the target image.

[0062] Step 110: From the self-supervised continuous learning neural network that has completed training, select the self-supervised continuous learning neural network branch corresponding to the estimated noise type and / or estimated noise intensity, and identify the recognition noise type and / or recognition noise intensity of the target image.

[0063] Step 120: Based on the self-supervised continuous learning neural network branch, perform noise reduction processing on the target image;

[0064] The training process of the self-supervised continuous learning neural network includes:

[0065] A task is assigned to a noise image of the same noise type and / or noise intensity. The noise images of at least one noise type and / or noise intensity are divided into multiple tasks to obtain a noise image dataset for at least one task.

[0066] The first process is performed on the noisy image dataset for at least one task, and the self-supervised continuous learning neural network updated in the last first process is used as the self-supervised continuous learning neural network that has completed training.

[0067] The first process includes:

[0068] Based on the noisy image dataset corresponding to the task of the current first process and the current self-supervised continuous learning neural network, training is performed, and the first part layer and the second part layer in the trained self-supervised continuous learning neural network are determined. The correlation scores of the combined gating and convolutional kernel elements corresponding to the first part layer are lower than the correlation scores of the combined gating and convolutional kernel elements corresponding to the second part layer. The first part layer does not include the output layer of the self-supervised continuous learning neural network, and the second part layer is the layer in the self-supervised continuous learning neural network other than the first part layer and the output layer.

[0069] Set the convolution kernels of the first layer in the trained self-supervised continuous learning neural network to zero to obtain the pruned self-supervised continuous learning neural network.

[0070] Based on the noisy image dataset corresponding to the task in the current first process, the parameters of the second layer in the pruned self-supervised continuous learning neural network are adjusted to obtain the parameter-adjusted self-supervised continuous learning neural network.

[0071] The parameters of the convolutional kernels and the output layer of the second part of the self-supervised continuous learning neural network are frozen after parameter adjustment. A mask is added to the parameters of the convolutional kernels and the output layer of the second part of the self-supervised continuous learning neural network after parameter adjustment to obtain an updated self-supervised continuous learning neural network. The updated self-supervised continuous learning neural network is used for training on the noisy image dataset of the next task in the next first process.

[0072] Specifically, given that images acquired by multiple sensors on unmanned platforms contain different types and intensities of noise, and to address the shortcomings of existing methods that require paired clean-noise images for supervised learning when training deep learning denoising models (which are difficult to obtain in practice) and the inability to continuously learn different intensities or types of noise on the same network, this invention proposes a multi-noise interference suppression method for unmanned platforms based on self-supervised continuous learning. This method includes iteratively executing a first process to obtain a trained self-supervised continuous learning neural network, and then using the trained self-supervised continuous learning neural network to execute steps 100-120 to achieve incremental removal of multi-noise in the image.

[0073] It should be noted that unmanned platforms (unmanned systems) are often equipped with multiple types of sensors. Images acquired by different types of sensors contain different types and intensities of noise. This invention uses a unified lightweight model to process various types or intensities of noise, which can effectively suppress various interferences and noises in target images acquired by unmanned systems equipped with multiple sensors.

[0074] Figure 4 This is a flowchart illustrating the multi-noise interference suppression method for unmanned platforms based on self-supervised continuous learning provided by the present invention, as shown below. Figure 4 As shown, it specifically includes:

[0075] Step S10: During the training phase, the noise images of different types and / or different intensities acquired by the unmanned platform (also known as the unmanned system) are divided into multiple tasks. Each noise type and / or noise intensity is set as a task. For example, noise images of the same noise type are assigned to the same task, or noise images of the same noise intensity are assigned to the same task, or noise images of the same noise intensity and the same noise type are assigned to the same task. The noise image dataset of at least one task after division is used as the input data of the first process corresponding to multiple tasks.

[0076] Specifically, in practical applications, unmanned platforms often carry multiple types of sensors, and images acquired by different types of sensors contain different types and intensities of noise. For example, images captured by optical sensors are typically affected by additive Gaussian noise, while images acquired by synthetic aperture radar are severely affected by multiplicative gamma noise. As the observation time and space of the unmanned platform change, it continuously acquires streaming data. Faced with different types of noisy images or data streams of noise with varying intensities, a denoising model is needed to continuously learn from newly acquired data and avoid forgetting previously learned tasks.

[0077] Specifically, this invention divides the acquired data into multiple tasks based on different noise types and intensities, with the network model learning one noise task at a time. By sequentially learning multiple tasks on the same network, the network can handle noise images of different intensities or types.

[0078] Optionally, to facilitate the evaluation of denoising effects, this invention obtains input data for multiple tasks by adding Gaussian additive noise of different intensities, as well as adding different types of Gaussian additive noise and multiplicative gamma noise, to clean images. For multiplicative noise images, preprocessing is required: pixel values ​​less than 1 are set to 1, followed by a logarithmic transformation and division by log(255) to normalize the image to between 0 and 1. For additive noise images, the above preprocessing is unnecessary. This invention proposes a self-supervised continuous learning-based multi-noise interference suppression method for unmanned platforms. It can continuously learn additive noise images of various intensities, multiplicative noise images of various intensities, and both additive and multiplicative noise images using the same deep model. This effectively avoids catastrophic forgetting of previously learned tasks while learning new tasks.

[0079] Specifically, since this invention employs a self-supervised learning-based denoising method, for each task's noisy image y, a pair of downsampled noise images are obtained through random neighborhood downsampling, and the following settings are made:

[0080] G = (g1(y), g2(y))

[0081] Where G represents the random neighborhood downsampling method, and g1(y) and g2(y) represent two sample noise images obtained by downsampling, which are used as the input and output of the network, respectively.

[0082] This invention proposes a method for suppressing multiple noise interferences on unmanned platforms based on self-supervised continuous learning. By dividing the noise type and intensity of the images acquired by the unmanned platform into task segments, preprocessing and downsampling, a pair of noisy images is obtained for self-supervised learning, thereby avoiding dependence on clean images.

[0083] Step S20: Use the noisy image dataset of the first task (which can be a randomly selected task as the first task) to perform sparse training of the self-supervised continuous learning neural network. After training, prune the network.

[0084] Specifically, when pruning the network, except for the network layers in the output header, each other layer selects convolutional kernels with high relevance scores according to a certain proportion, and sets the parameters of unselected convolutional kernels to zero.

[0085] Optionally, since this invention relates to a multi-noise interference suppression method based on self-supervised continuous learning, it is necessary to continuously learn multiple noisy tasks on the same network. Therefore, it is essential to ensure that sufficient network capacity is retained for learning new tasks while fully memorizing the important parameters of the previous task. The training process for the current task uses all parameters of the learned task and unused parameters in the network (i.e., the self-supervised continuous learning neural network). Among the parameters specific to the current task, i.e., the parameters not used in the previous network, an independent gating module is added to each convolutional kernel for kernel selection, retaining the important convolutional kernels for the current task and reinitializing the unimportant convolutional kernels.

[0086] Optionally, the structure of the self-supervised continuous learning neural network used is as follows:

[0087] The improved U-Net network is used, which includes an encoder, a decoder, and skip connections. The encoder consists of convolutional layers, a LeakyReLU activation function, and a max-pooling downsampling module; the decoder consists of bilinear interpolation upsampling, convolution, and activation function modules; and skip connections connect partial features from the encoder and decoder together, making the model more accurate and robust.

[0088] The self-supervised continuous learning neural network model consists of 20 convolutional layers, 5 downsampling layers, and 5 upsampling layers. The last 5 convolutional layers of the network are set as multi-head output layers.

[0089] The gating module structure of a self-supervised continuous learning neural network is as follows:

[0090]

[0091] in This represents the gating module at level l of task t, ⊙ represents channel multiplication, and h l and h l+1 These represent the input and output feature maps of the l-th layer, respectively. This represents the sparse feature map output by the l-th layer. This represents the gating value of the i-th channel in the l-th layer. 0 indicates that the channel parameter is masked, and 1 indicates that the channel parameter is selected.

[0092] The gating module mainly consists of a multilayer perceptron and a ReLU activation function. The last linear layer provides a log probability for each output channel of the convolutional kernel. The gating module learns along with the network during training until optimization for a task is completed. To allow sufficient space for subsequent task learning, the number of convolutional kernels should be minimized during each task learning phase. This is achieved by adding sparsity regularization to the loss function to control the sparsity of the network parameters.

[0093]

[0094] Where L represents the total number of gate layers, λ s This represents a coefficient that controls the level of sparsity. This represents the number of channels in the output feature map of the l-th layer.

[0095] When training the Gaussian additive noise removal branch network and not for the last task, use the following regularized loss function to train the denoising branch network:

[0096]

[0097] Among them, f θ Let γ1 represent the denoising branch network, where γ1 is a hyperparameter controlling the degree of regularization and γ2 is a hyperparameter controlling the sparsity coefficient.

[0098] In the case of training the Gaussian additive noise-removed branch network, which is the last task, sparsity training is not performed again, so it is no longer used.

[0099] In the training of the multiplicative noise removal branch network, which is not the last task, the image is first logarithmically transformed to convert the multiplicative noise into additive noise, which is then input into the branch network. The denoising branch network is trained using the following regularized loss function:

[0100]

[0101] Among them, f θ Let γ1 represent the denoising branch network, where γ1 is a hyperparameter controlling the degree of regularization and γ2 is a hyperparameter controlling the sparsity coefficient.

[0102] In particular, when training the multiplicative noise-removed branch network and not on the last task, sparsity training is no longer performed, so it is no longer used.

[0103] Optionally, after training the multiplicative noise, it is necessary to map the image back to the original domain, i.e. Where x represents the logarithmic domain image restored by the neural network, and exp represents the exponential operation.

[0104] After task t is optimized, the self-supervised continuous learning neural network is pruned by adjusting the training set. The gating activation probability is calculated, and the correlation score of each combination element of gating and convolutional kernel in layer l is estimated. The higher the correlation score of a convolutional kernel, the more important the kernel is. When calculating the correlation score, convolutional kernels frozen for tasks that have already been learned should be excluded. The correlation scores of all unfrozen convolutional kernels should be calculated, sorted, and the convolutional kernel parameters with the highest percentage of ranking should be frozen.

[0105] The method for calculating the relevant scores is as follows:

[0106]

[0107] in, Let p(·) represent an index function, and let p(·) represent a probability distribution.

[0108] Based on the calculated relevance scores, except for the network layers in the output header and the convolutional kernels that have been frozen for previously learned tasks, each other layer selects convolutional kernels with higher relevance scores according to a certain proportion, freezes the selected convolutional kernels for use in subsequent tasks, and reinitializes the parameters of unfrozen kernels with lower relevance or unrelated kernels to zero, leaving room for learning in subsequent tasks.

[0109] Step S30: Use the noisy image dataset of the first task to fine-tune the selected convolutional kernels in the network (i.e., based on the noisy image dataset of the task corresponding to the current first process, adjust the parameters of the second layer in the pruned self-supervised continuous learning neural network to obtain the parameter-adjusted self-supervised continuous learning neural network). After training, freeze all selected convolutional kernel parameters and the corresponding output head of the network, and add task masks to these parameters.

[0110] Specifically, fine-tuning training is performed only on the convolutional kernels selected for the current task. Convolutional kernels frozen for previously learned tasks remain unchanged, and the parameters of all unselected convolutional kernels for all tasks are set to zero. Fine-tuning further optimizes the parameters of the selected convolutional kernels, resulting in better denoising performance. After fine-tuning, all selected convolutional kernel parameters and the corresponding network output headers are frozen, and task masks are added to these parameters so that the parameters for the corresponding task can be used during subsequent inference.

[0111] Step S40: Perform sparse training of the self-supervised continuous learning neural network using the noisy image dataset of the next new task. During training, use the frozen convolutional kernel parameters from the previous task (i.e., one or more previous tasks), as well as the unfrozen convolutional kernels and new output heads in the self-supervised continuous learning neural network to learn new parameters. After training, select convolutional kernels with high relevance scores for each layer according to a certain proportion, and set the parameters of the unselected convolutional kernels to zero. Fine-tune the selected convolutional kernels and freeze all selected convolutional kernels and their corresponding output heads; that is, repeat the convolutional kernel selection and zeroing in step S20, and the convolutional kernel parameter fine-tuning and freezing in step S30.

[0112] Step S50: Repeat step S40 to process subsequent learning tasks. If the current task is the last task, train the network using all network parameters and the new output head. After training, no further fine-tuning is performed. Instead, freeze the convolutional kernels and the corresponding output heads, and the training process ends.

[0113] This invention proposes a method for suppressing multiple noise interference on unmanned platforms based on self-supervised continuous learning. It has the ability to incrementally process multi-task noise under limited network capacity, and does not require maintaining the data of learned tasks, which greatly saves the platform's computing and storage resources.

[0114] Step S60: Perform steps 110-130 above, that is, in the inference stage, for target images with different types of noise, estimate the noise type by fitting the statistical distribution of the density function, and then select the corresponding branch of the trained self-supervised continuous learning neural network for denoising processing; for target images with different intensities of noise, select a part of the uniform region in the target image, estimate the noise intensity by fitting the density function distribution, and then select the corresponding network model branch for denoising processing to obtain the denoised image and restore it back to the original image domain.

[0115] In this invention, during the inference stage, the noise type and intensity on the target image may be unknown and need to be estimated in order to select a suitable model branch for denoising. When judging the noise type, different noise distribution functions are used to fit the data of the whole image to estimate the parameters of the probability distribution. The degree of fit of various distributions is calculated by KL divergence, and the noise type corresponding to the smallest KL divergence value is obtained.

[0116] Specifically, to distinguish between additive Gaussian noise and multiplicative gamma noise, the parameter values ​​are estimated separately: Gaussian noise (mean μ, standard deviation σ), gamma noise (distribution parameter...). The estimated parameters are substituted into the probability density function and the histogram of the image data to calculate the discrete KL divergence value. The smaller value indicates the noise type of the image.

[0117] Optionally, to estimate the noise intensity, the flattest region with a 50x50 window can be selected from the target image, which can be calculated using a sliding window. The region with the smallest value is identified, and then the standard deviation of the data within this window is calculated for Gaussian noise, which represents the intensity of the additive Gaussian noise. For gamma noise, the standard deviation is calculated... The value of is the intensity of the multiplicative gamma noise.

[0118] Based on the estimated noise type and intensity, the appropriate branch of the self-supervised continuous learning neural network is selected for denoising.

[0119] Optionally, the target image can be an image to be noise-suppressed, or a test image used to test the denoising effect of a self-supervised continuous learning neural network.

[0120] This invention tests its method by adding Gaussian additive noise of different intensities, multiplicative noise of different intensities, and adding both Gaussian additive and multiplicative noise separately to an image. When testing the denoising effect of the model, this invention adds noise to a single image 20 times, calculates the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) for each denoising step, and then calculates the average peak signal-to-noise ratio (PSNR) and structural similarity (SSIM).

[0121] This invention determines the type of noise and estimates its intensity by fitting statistical distribution data. It can effectively distinguish whether the noise in the image is additive or multiplicative and estimate the level of noise. Therefore, when performing denoising inference, it is not necessary to know the type and intensity of the noise in the input image in advance.

[0122] Figure 2 This is a schematic diagram showing the original image and the image after adding noise used in testing the multi-noise interference suppression method for unmanned platforms based on self-supervised continuous learning provided by this invention. Figure 3 This is a schematic diagram illustrating the denoising results of the self-supervised continuous learning-based multi-noise interference suppression method for unmanned platforms provided by this invention, as shown below. Figure 3 As shown, it can suppress various intensities of additive noise, various intensities of multiplicative noise, and both additive and multiplicative noise.

[0123] In this invention, continuous learning effectively addresses the catastrophic forgetting problem in networks. Continuous learning incrementally learns new tasks on the same network while overcoming the forgetting of previously learned tasks. Continuous learning methods are mainly divided into three categories: sample replay-based methods, regularization-based methods, and parameter isolation-based methods. In practical applications, unmanned platforms equipped with sensors continuously acquire streaming data as observation time and space change, and images may be affected by noise of varying intensities or types. This invention introduces the concept of continuous learning based on self-supervised learning, continuously learning to denoise images of different intensities and types, ultimately achieving effective suppression of noise of varying intensities or types using the same network.

[0124] Optionally, before executing the first process for the first time, the method further includes:

[0125] Adjust the structure of the U-Net network and initialize the adjusted U-Net network to obtain an initialized self-supervised continuous learning neural network;

[0126] The adjusted U-Net network includes an encoder, a decoder, and skip connections. The encoder consists of convolutional layers, a LeakyReLU activation function, and a max-pooling downsampling module. The decoder consists of bilinear interpolation upsampling, convolution, and activation function modules. The skip connections are used to connect some features of the encoder and the decoder together.

[0127] This invention uses an improved U-Net network, including an encoder, a decoder, and skip connections. The encoder consists of convolutional layers, a LeakyReLU activation function, and a max-pooling downsampling module; the decoder consists of bilinear interpolation upsampling, convolution, and activation function modules; and skip connections connect some features of the encoder and decoder together, making the model more accurate and robust.

[0128] Optionally, the initialized self-supervised continuous learning neural network includes: 20 convolutional layers, 5 downsampling layers, and 5 upsampling layers, wherein the last 5 convolutional layers of the 20 convolutional layers are the output layers; the structure of the gating module of the self-supervised continuous learning neural network is represented as follows:

[0129]

[0130] in This represents the gating module at layer l corresponding to task t in at least one task, ⊙ represents channel multiplication, and h l h represents the input feature map of the l-th layer. l+1 This represents the output feature map of the l-th layer. This represents the sparse feature map output by the l-th layer. This represents the gating value of the i-th channel in the l-th layer. 0 indicates that the channel parameter is masked, and 1 indicates that the channel parameter is selected.

[0131] Optionally, the method further includes:

[0132] Before executing the first process corresponding to any one of the at least one tasks, the noisy image dataset of the task is preprocessed to obtain multiple sets of similar noisy images after random neighborhood downsampling, which are used as inputs for the training and parameter adjustment in the first process corresponding to the task.

[0133] Optionally, the noisy image dataset for the task is preprocessed to obtain multiple sets of similar noisy images after random neighborhood downsampling, including:

[0134] The pixel value of the target pixel in the noisy image with multiplicative noise in the noisy image dataset is set to 1, and the noisy image with multiplicative noise is normalized to obtain the processed noisy image dataset, wherein the target pixel is a pixel with a pixel value less than 1.

[0135] Random neighborhood downsampling is performed on the noisy images in the processed noisy image dataset to obtain multiple sets of similar noisy images after random neighborhood downsampling.

[0136] Specifically, the preprocessing of noisy images in the dataset involves using random neighborhood downsampling to obtain two similar downsampled noisy images. This can include: if the acquired images contain multiplicative noise, preprocessing is required by setting all pixel values ​​less than 1 in the noise image to 1, then performing a logarithmic transformation on the noise image and dividing by log(255) to normalize the noise image to a range of 0 to 1. If the acquired data contains additive noise, the above preprocessing is not necessary.

[0137] For a noisy image sample y, a pair of downsampled noise images are obtained through random neighborhood downsampling, and the following settings are made:

[0138] G = (g1(y), g2(y))

[0139] Where G represents the random neighborhood downsampling method, and g1(y) and g2(y) represent two noisy images obtained by downsampling, which are used as the input and output of the self-supervised continuous learning neural network, respectively.

[0140] Optionally, in the first processes other than the last first process, the training loss function is expressed as:

[0141]

[0142] In the last first process, the training loss function is expressed as:

[0143]

[0144] in, To reconstruct the loss, For consistency regularization loss, The loss is sparse regularization loss, where γ1 is a hyperparameter controlling the degree of regularization and γ2 is a hyperparameter controlling the degree of sparsity.

[0145] Optionally, by minimizing the loss function Guide network parameter learning and optimization, among which To reconstruct the loss, For consistency regularization loss, The loss is sparse regularization loss, where γ1 is a hyperparameter controlling the degree of regularization and γ2 is a hyperparameter controlling the degree of sparsity.

[0146] Specifically, Where f θ For a deep neural network model, p is a distance metric function;

[0147] q is the distance metric function;

[0148] Where L represents the total number of gate layers, λ s This represents a coefficient that controls the level of sparsity. The channels of the output feature map of layer l

[0149] When training the last task, a new initial output head is set, and training is performed using all the parameters frozen from the learned tasks and the remaining parameters of the network. The loss function no longer uses sparse regularization constraints, i.e., it is trained through the loss function. Optimize parameters.

[0150] In one embodiment, the present invention can be implemented in software, which can run on a Linux system or a Windows system, and the computer configuration can be two Intel Xeon Silver 4210 CPUs, 64GB of memory, and one RTX 3090.

[0151] The experimental content and results analysis in this embodiment are as follows:

[0152] To evaluate the effectiveness of the self-supervised continuous learning-based multi-noise interference suppression method for unmanned platforms of this invention, the experimental comparison methods include: individual training (training a model separately for each denoising task using existing self-supervised denoising methods), fine-tuning (training the model for the next task based on the model learned from the previous task, until all tasks are learned, and then testing the model on all tasks), and the method of this invention.

[0153] like Figure 2 The image shown is the original image and the image with added noise used in testing the multi-noise interference suppression method for unmanned platforms based on self-supervised continuous learning according to the present invention. Figure 2 The image with subscript A is the original image used in the test. Figure 2 The images labeled B to E are the noise maps after adding Gaussian additive noise of 25, Gaussian additive noise of 50, multiplicative gamma noise with an equivalent number of views of 4, and multiplicative gamma noise with an equivalent number of views of 1, respectively, to the original image. Figure 2 The image with subscript A and Figure 3The image with subscript D is a denoising result image of a test image containing Gaussian 50 additive noise and a test image containing Gaussian 25 additive noise, respectively, using a model that first learns Gaussian 50 additive noise and then learns Gaussian 25 additive noise. Figure 3 The image with subscript B and Figure 3 The image with subscript E in the middle is the denoising result of the test image containing L1 multiplicative noise and the test image containing L4 multiplicative noise, respectively, using the model that first learns L1 multiplicative noise and then learns L4 multiplicative noise. Figure 3 The image with subscript C and Figure 3 The image with subscript F is a denoising result image of a test image containing L4 multiplicative noise and a test image containing Gaussian 50 additive noise, respectively, using a model that first learns L4 multiplicative noise and then learns Gaussian 50 additive noise.

[0154] Table 1 shows the comparison results of average PSNR and SSIM on the test images after continuous learning with additive Gaussian noise of different intensities. Table 2 shows the comparison results of average PSNR and SSIM on the test images after continuous learning with multiplicative gamma noise of different intensities. Table 3 shows the comparison results of average PSNR and SSIM on the test images after continuous learning with different types of noise. Figure 2 , Figure 3 As shown in Tables 1 and 2, the present invention can effectively remove noise of different intensities, including Gaussian additive noise and multiplicative noise of different intensities. Furthermore, the denoising results are significantly better than the fine-tuning results on previously learned tasks. This is because the fine-tuning method can only adapt to newly learned tasks, but it leads to catastrophic forgetting of previously learned tasks. Additionally, the PSNR and SSMI values ​​of the method in this invention are quite close to the results of individual training, but individual training learns a separate model for each noise intensity, which is very computationally and storage-intensive. Figure 2 , Figure 3 As can be seen from Table 3, the method of the present invention can effectively learn continuously from different types of noise. The numerical results on the learned task are significantly better than the fine-tuning results, and the results on both the learned task and the current task are similar to the results of training alone.

[0155] Table 1

[0156]

[0157] Table 2

[0158]

[0159] Table 3

[0160]

[0161] The following describes the image multi-noise interference suppression device for unmanned systems based on self-supervised continuous learning provided by the present invention. The image multi-noise interference suppression device for unmanned systems based on self-supervised continuous learning described below can be referred to in correspondence with the image multi-noise interference suppression method for unmanned systems based on self-supervised continuous learning described above.

[0162] Figure 5 This is a flowchart illustrating the multi-noise interference suppression device for unmanned systems based on self-supervised continuous learning provided by the present invention, as shown below. Figure 5 As shown, the device 500 includes:

[0163] The noise estimation module 510 is used to estimate the noise type and / or noise intensity of the target image of the unmanned system by fitting a density function, so as to obtain the estimated noise type and / or estimated noise intensity of the target image.

[0164] The denoising network branch selection module 520 is used to select a branch of the self-supervised continuous learning neural network that corresponds to the estimated noise type and / or estimated noise intensity from the self-supervised continuous learning neural network that has been trained, and to identify the recognized noise type and / or recognized noise intensity of the target image.

[0165] Denoising module 530 is used to perform denoising processing on the target image based on the self-supervised continuous learning neural network branch;

[0166] It also includes a self-supervised continuous learning neural network training module 540, used for:

[0167] A task is assigned to a noise image of the same noise type and / or noise intensity. The noise images of at least one noise type and / or noise intensity are divided into multiple tasks to obtain a noise image dataset for at least one task.

[0168] The first process is performed on the noisy image dataset for at least one task, and the self-supervised continuous learning neural network updated in the last first process is used as the self-supervised continuous learning neural network that has completed training.

[0169] The first process includes:

[0170] Based on the noisy image dataset corresponding to the task of the current first process and the current self-supervised continuous learning neural network, training is performed, and the first part layer and the second part layer in the trained self-supervised continuous learning neural network are determined. The correlation scores of the combined gating and convolutional kernel elements corresponding to the first part layer are lower than the correlation scores of the combined gating and convolutional kernel elements corresponding to the second part layer. The first part layer does not include the output layer of the self-supervised continuous learning neural network, and the second part layer is the layer in the self-supervised continuous learning neural network other than the first part layer and the output layer.

[0171] Set the convolution kernels of the first layer in the trained self-supervised continuous learning neural network to zero to obtain the pruned self-supervised continuous learning neural network.

[0172] Based on the noisy image dataset corresponding to the task in the current first process, the parameters of the second layer in the pruned self-supervised continuous learning neural network are adjusted to obtain the parameter-adjusted self-supervised continuous learning neural network.

[0173] The parameters of the convolutional kernels and the output layer of the second part of the self-supervised continuous learning neural network are frozen after parameter adjustment. A mask is added to the parameters of the convolutional kernels and the output layer of the second part of the self-supervised continuous learning neural network after parameter adjustment to obtain an updated self-supervised continuous learning neural network. The updated self-supervised continuous learning neural network is used for training on the noisy image dataset of the next task in the next first process.

[0174] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6 As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other through the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute a method for suppressing multiple noise interferences in images of unmanned systems based on self-supervised continuous learning. This method includes:

[0175] By fitting a density function, the noise type and / or noise intensity of the target image of the unmanned system are estimated, and the estimated noise type and / or estimated noise intensity of the target image are obtained.

[0176] From the self-supervised continuous learning neural network that has been trained, select the branch of the self-supervised continuous learning neural network that corresponds to the estimated noise type and / or estimated noise intensity.

[0177] Based on the self-supervised continuous learning neural network branch, the target image is denoised.

[0178] The training process of the self-supervised continuous learning neural network includes:

[0179] A task is assigned to a noise image of the same noise type and / or noise intensity. The noise images of at least one noise type and / or noise intensity are divided into multiple tasks to obtain a noise image dataset for at least one task.

[0180] The first process is performed on the noisy image dataset for at least one task, and the self-supervised continuous learning neural network updated in the last first process is used as the self-supervised continuous learning neural network that has completed training.

[0181] The first process includes:

[0182] Based on the noisy image dataset corresponding to the task of the current first process and the current self-supervised continuous learning neural network, training is performed, and the first part layer and the second part layer in the trained self-supervised continuous learning neural network are determined. The correlation scores of the combined gating and convolutional kernel elements corresponding to the first part layer are lower than the correlation scores of the combined gating and convolutional kernel elements corresponding to the second part layer. The first part layer does not include the output layer of the self-supervised continuous learning neural network, and the second part layer is the layer in the self-supervised continuous learning neural network other than the first part layer and the output layer.

[0183] Set the convolution kernels of the first layer in the trained self-supervised continuous learning neural network to zero to obtain the pruned self-supervised continuous learning neural network.

[0184] Based on the noisy image dataset corresponding to the task in the current first process, the parameters of the second layer in the pruned self-supervised continuous learning neural network are adjusted to obtain the parameter-adjusted self-supervised continuous learning neural network.

[0185] The parameters of the convolutional kernels and the output layer of the second part of the self-supervised continuous learning neural network are frozen after parameter adjustment. A mask is added to the parameters of the convolutional kernels and the output layer of the second part of the self-supervised continuous learning neural network after parameter adjustment to obtain an updated self-supervised continuous learning neural network. The updated self-supervised continuous learning neural network is used for training on the noisy image dataset of the next task in the next first process.

[0186] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0187] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the image multi-noise interference suppression method for unmanned systems based on self-supervised continuous learning provided by the above methods. The method includes:

[0188] By fitting a density function, the noise type and / or noise intensity of the target image of the unmanned system are estimated, and the estimated noise type and / or estimated noise intensity of the target image are obtained.

[0189] From the self-supervised continuous learning neural network that has been trained, select the branch of the self-supervised continuous learning neural network that corresponds to the estimated noise type and / or estimated noise intensity.

[0190] Based on the self-supervised continuous learning neural network branch, the target image is denoised.

[0191] The training process of the self-supervised continuous learning neural network includes:

[0192] A task is assigned to a noise image of the same noise type and / or noise intensity. The noise images of at least one noise type and / or noise intensity are divided into multiple tasks to obtain a noise image dataset for at least one task.

[0193] The first process is performed on the noisy image dataset for at least one task, and the self-supervised continuous learning neural network updated in the last first process is used as the self-supervised continuous learning neural network that has completed training.

[0194] The first process includes:

[0195] Based on the noisy image dataset corresponding to the task of the current first process and the current self-supervised continuous learning neural network, training is performed, and the first part layer and the second part layer in the trained self-supervised continuous learning neural network are determined. The correlation scores of the combined gating and convolutional kernel elements corresponding to the first part layer are lower than the correlation scores of the combined gating and convolutional kernel elements corresponding to the second part layer. The first part layer does not include the output layer of the self-supervised continuous learning neural network, and the second part layer is the layer in the self-supervised continuous learning neural network other than the first part layer and the output layer.

[0196] Set the convolution kernels of the first layer in the trained self-supervised continuous learning neural network to zero to obtain the pruned self-supervised continuous learning neural network.

[0197] Based on the noisy image dataset corresponding to the task in the current first process, the parameters of the second layer in the pruned self-supervised continuous learning neural network are adjusted to obtain the parameter-adjusted self-supervised continuous learning neural network.

[0198] The parameters of the convolutional kernels and the output layer of the second part of the self-supervised continuous learning neural network are frozen after parameter adjustment. A mask is added to the parameters of the convolutional kernels and the output layer of the second part of the self-supervised continuous learning neural network after parameter adjustment to obtain an updated self-supervised continuous learning neural network. The updated self-supervised continuous learning neural network is used for training on the noisy image dataset of the next task in the next first process.

[0199] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the above-described methods for suppressing multiple noise interferences in images of unmanned systems based on self-supervised continuous learning, the method comprising:

[0200] By fitting a density function, the noise type and / or noise intensity of the target image of the unmanned system are estimated, and the estimated noise type and / or estimated noise intensity of the target image are obtained.

[0201] From the self-supervised continuous learning neural network that has been trained, select the branch of the self-supervised continuous learning neural network that corresponds to the estimated noise type and / or estimated noise intensity.

[0202] Based on the self-supervised continuous learning neural network branch, the target image is denoised.

[0203] The training process of the self-supervised continuous learning neural network includes:

[0204] A task is assigned to a noise image of the same noise type and / or noise intensity. The noise images of at least one noise type and / or noise intensity are divided into multiple tasks to obtain a noise image dataset for at least one task.

[0205] The first process is performed on the noisy image dataset for at least one task, and the self-supervised continuous learning neural network updated in the last first process is used as the self-supervised continuous learning neural network that has completed training.

[0206] The first process includes:

[0207] Based on the noisy image dataset corresponding to the task of the current first process and the current self-supervised continuous learning neural network, training is performed, and the first part layer and the second part layer in the trained self-supervised continuous learning neural network are determined. The correlation scores of the combined gating and convolutional kernel elements corresponding to the first part layer are lower than the correlation scores of the combined gating and convolutional kernel elements corresponding to the second part layer. The first part layer does not include the output layer of the self-supervised continuous learning neural network, and the second part layer is the layer in the self-supervised continuous learning neural network other than the first part layer and the output layer.

[0208] Set the convolution kernels of the first layer in the trained self-supervised continuous learning neural network to zero to obtain the pruned self-supervised continuous learning neural network.

[0209] Based on the noisy image dataset corresponding to the task in the current first process, the parameters of the second layer in the pruned self-supervised continuous learning neural network are adjusted to obtain the parameter-adjusted self-supervised continuous learning neural network.

[0210] The parameters of the convolutional kernels and the output layer of the second part of the self-supervised continuous learning neural network are frozen after parameter adjustment. A mask is added to the parameters of the convolutional kernels and the output layer of the second part of the self-supervised continuous learning neural network after parameter adjustment to obtain an updated self-supervised continuous learning neural network. The updated self-supervised continuous learning neural network is used for training on the noisy image dataset of the next task in the next first process.

[0211] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0212] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0213] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for suppressing multiple noise interferences in images of unmanned systems based on self-supervised continuous learning, characterized in that, include: By fitting a density function, the noise type and / or noise intensity of the target image of the unmanned system are estimated, and the estimated noise type and / or estimated noise intensity of the target image are obtained. From the self-supervised continuous learning neural network that has been trained, select the branch of the self-supervised continuous learning neural network that corresponds to the estimated noise type and / or estimated noise intensity. Based on the self-supervised continuous learning neural network branch, the target image is denoised. The training process of the self-supervised continuous learning neural network includes: A task is assigned to a noise image of the same noise type and / or noise intensity. The noise images of at least one noise type and / or noise intensity are divided into multiple tasks to obtain a noise image dataset for at least one task. The first process is performed on the noisy image dataset for at least one task, and the self-supervised continuous learning neural network updated in the last first process is used as the self-supervised continuous learning neural network that has completed training. The first process includes: Based on the noisy image dataset corresponding to the task of the current first process and the current self-supervised continuous learning neural network, training is performed, and the first part layer and the second part layer in the trained self-supervised continuous learning neural network are determined. The correlation scores of the combined gating and convolutional kernel elements corresponding to the first part layer are lower than the correlation scores of the combined gating and convolutional kernel elements corresponding to the second part layer. The first part layer does not include the output layer of the self-supervised continuous learning neural network, and the second part layer is the layer in the self-supervised continuous learning neural network other than the first part layer and the output layer. Set the convolution kernels of the first layer in the trained self-supervised continuous learning neural network to zero to obtain the pruned self-supervised continuous learning neural network. Based on the noisy image dataset corresponding to the task in the current first process, the parameters of the second layer in the pruned self-supervised continuous learning neural network are adjusted to obtain the parameter-adjusted self-supervised continuous learning neural network. The parameters of the convolutional kernels and the output layer of the second part of the self-supervised continuous learning neural network are frozen after parameter adjustment. A mask is added to the parameters of the convolutional kernels and the output layer of the second part of the self-supervised continuous learning neural network after parameter adjustment to obtain an updated self-supervised continuous learning neural network. The updated self-supervised continuous learning neural network is used for training on the noisy image dataset of the next task in the next first process.

2. The method for suppressing multiple noise interferences in images of unmanned systems based on self-supervised continuous learning according to claim 1, characterized in that, Before the first execution of the first process, the method further includes: Adjust the structure of the U-Net network and initialize the adjusted U-Net network to obtain an initialized self-supervised continuous learning neural network; The adjusted U-Net network includes an encoder, a decoder, and skip connections. The encoder consists of convolutional layers, a LeakyReLU activation function, and a max-pooling downsampling module. The decoder consists of bilinear interpolation upsampling, convolution, and activation function modules. The skip connections are used to connect some features of the encoder and the decoder together.

3. The method for suppressing multiple noise interferences in images of unmanned systems based on self-supervised continuous learning according to claim 2, characterized in that, The initialized self-supervised continuous learning neural network includes: 20 convolutional layers, 5 downsampling layers, and 5 upsampling layers, wherein the last 5 convolutional layers of the 20 convolutional layers are the output layers; the structure of the gating module of the self-supervised continuous learning neural network is represented as follows: in This represents the gating module at layer l corresponding to task t in at least one task, ⊙ represents channel multiplication, and h l h represents the input feature map of the l-th layer. l+1 This represents the output feature map of the l-th layer. This represents the sparse feature map output by the l-th layer. This represents the gating value of the i-th channel in the l-th layer. 0 indicates that the channel parameter is masked, and 1 indicates that the channel parameter is selected.

4. The method for suppressing multiple noise interferences in images of unmanned systems based on self-supervised continuous learning according to claim 1, characterized in that, The method further includes: Before executing the first process corresponding to any one of the at least one tasks, the noisy image dataset of the task is preprocessed to obtain multiple sets of similar noisy images after random neighborhood downsampling, which are used as inputs for the training and parameter adjustment in the first process corresponding to the task.

5. The method for suppressing multiple noise interferences in images of unmanned systems based on self-supervised continuous learning according to claim 4, characterized in that, The noisy image dataset for the task is preprocessed to obtain multiple sets of similar noisy images after random neighborhood downsampling, including: The pixel value of the target pixel in the noisy image with multiplicative noise in the noisy image dataset is set to 1, and the noisy image with multiplicative noise is normalized to obtain the processed noisy image dataset, wherein the target pixel is a pixel with a pixel value less than 1. Random neighborhood downsampling is performed on the noisy images in the processed noisy image dataset to obtain multiple sets of similar noisy images after random neighborhood downsampling.

6. The method for suppressing multiple noise interferences in images of unmanned systems based on self-supervised continuous learning according to claim 1, characterized in that, In all first processes except the last one, the training loss function is expressed as: In the last first process, the training loss function is expressed as: in, To reconstruct the loss, For consistency regularization loss, The loss is sparse regularization loss, where γ1 is a hyperparameter controlling the degree of regularization and γ2 is a hyperparameter controlling the degree of sparsity.

7. A device for suppressing multiple noise interferences in images of unmanned systems based on self-supervised continuous learning, characterized in that, include: The noise estimation module is used to estimate the noise type and / or noise intensity of the target image of the unmanned system by fitting a density function, so as to obtain the estimated noise type and / or estimated noise intensity of the target image. The denoising network branch selection module is used to select a branch of the self-supervised continuous learning neural network that corresponds to the estimated noise type and / or estimated noise intensity from the self-supervised continuous learning neural network that has completed training. A denoising module is used to perform denoising processing on the target image based on the self-supervised continuous learning neural network branch. It also includes a self-supervised continuous learning neural network training module, used for: A task is assigned to a noise image of the same noise type and / or noise intensity. The noise images of at least one noise type and / or noise intensity are divided into multiple tasks to obtain a noise image dataset for at least one task. The first process is performed on the noisy image dataset for at least one task, and the self-supervised continuous learning neural network updated in the last first process is used as the self-supervised continuous learning neural network that has completed training. The first process includes: Based on the noisy image dataset corresponding to the task of the current first process and the current self-supervised continuous learning neural network, training is performed, and the first part layer and the second part layer in the trained self-supervised continuous learning neural network are determined. The correlation scores of the combined gating and convolutional kernel elements corresponding to the first part layer are lower than the correlation scores of the combined gating and convolutional kernel elements corresponding to the second part layer. The first part layer does not include the output layer of the self-supervised continuous learning neural network, and the second part layer is the layer in the self-supervised continuous learning neural network other than the first part layer and the output layer. Set the convolution kernels of the first layer in the trained self-supervised continuous learning neural network to zero to obtain the pruned self-supervised continuous learning neural network. Based on the noisy image dataset corresponding to the task in the current first process, the parameters of the second layer in the pruned self-supervised continuous learning neural network are adjusted to obtain the parameter-adjusted self-supervised continuous learning neural network. The parameters of the convolutional kernels and the output layer of the second part of the self-supervised continuous learning neural network are frozen after parameter adjustment. A mask is added to the parameters of the convolutional kernels and the output layer of the second part of the self-supervised continuous learning neural network after parameter adjustment to obtain an updated self-supervised continuous learning neural network. The updated self-supervised continuous learning neural network is used for training on the noisy image dataset of the next task in the next first process.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the image multi-noise interference suppression method for unmanned systems based on self-supervised continuous learning as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the image multi-noise interference suppression method for unmanned systems based on self-supervised continuous learning as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the image multi-noise interference suppression method for unmanned systems based on self-supervised continuous learning as described in any one of claims 1 to 6.

Images