Millimeter wave degraded image denoising and deblurring method, device, equipment and medium

By building a dual-task network model, the local additive residual dense feature fusion and semi-normalized residual feature fusion technology are used to solve the balance problem of denoising and defuzzing of millimeter wave images, and the efficiency of millimeter wave degraded images is achieved. Restore and get a clear image.

CN114936984BActive Publication Date: 2024-11-26CHENGDU QUANYI INTELLECTUAL PROPERTY OPERATION CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210686845.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-06-17
Publication Date
2024-11-26
Estimated Expiration
2042-06-17

AI Technical Summary

Technical Problem

The prior art is difficult to effectively balance the millimeter wave image denoising and defuzzing tasks, resulting in unclear images after denoising and defuzzing.

Method used

Construction of training and test sets of millimeter wave simulation degraded images, using a dual-task network model, including local additive residual dense feature fusion denoising subnetwork, local semi-normalized residual feature intensive fusion from low-order to high-order The denoising and defuzzing subnet and image reconstruction subnet are trained through the Adam algorithm, and the MSE loss and perceived loss are used as the loss function until the loss function reaches the preset convergence conditions.

Benefits of technology

Effective restoration of millimeter wave degraded images with blurred noise is achieved, and clear images are obtained, improving image resolution and quality.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114936984B_ABST
    Figure CN114936984B_ABST
Patent Text Reader

Abstract

The present invention relates to a dual-task restoration method, device, electronic device and computer storage medium for simultaneous denoising and deblurring of millimeter-wave degraded images, the method comprising: constructing a training set and a test set of millimeter-wave simulated degraded images; inputting the training set images into an initial dual-task network model for training until the loss function of the initial dual-task network model reaches a preset convergence condition, wherein the initial dual-task network model structure comprises a local additive residual dense feature fusion denoising subnetwork, a local low-order to high-order semi-normalized residual feature dense fusion denoising and deblurring subnetwork and an image reconstruction subnetwork; inputting the test set images into the trained dual-task network model, and outputting a test restoration image. The present invention can effectively balance the relationship between denoising and deblurring of millimeter-wave degraded images, realize the dual-task problem of denoising and deblurring, and obtain a high-quality clear image.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method, device, electronic equipment and computer storage medium for denoising and deblurring millimeter wave degraded images. Background Art

[0002] Passive millimeter wave (PMMW) imaging relies on receiving millimeter waves radiated by objects in the scene. The millimeter wave imaging system receives millimeter waves radiated by objects through an antenna array. After amplification, detection and signal processing, a two-dimensional image of the detection target can be obtained. In the imaging process, due to the long wavelength of millimeter waves, small antenna aperture, and the introduction of various types of random noise, the spatial resolution of the image after imaging is low. The usual solution is to increase the antenna size or increase the operating frequency to improve the spatial resolution. In recent years, the synthetic aperture technology in radio astronomy has been introduced into millimeter wave imaging detection, which effectively improves the spatial resolution of the system. However, the synthetic aperture technology has the disadvantages of complex structure, high cost and low operating frequency. Therefore, improving the image resolution of the millimeter wave imaging system through image restoration has become an economical, practical and effective choice.

[0003] Most of the existing methods for image restoration require a neural network for restoration. Many scholars have proposed many methods for degraded image restoration, but PMMW images are a dual-task problem with both blur and noise. The processes of image denoising and image deblurring are opposite. Removing high-frequency noise will cause high-frequency edge details to become blurred. The purpose of image deblurring is to restore high-frequency edge detail information, while also enhancing high-frequency noise. Therefore, it is difficult for current denoising and deblurring methods to coordinate the relationship between the two to obtain a good denoising and deblurring result. How to efficiently solve such a dual-task problem remains a challenge that needs to be solved urgently. Summary of the invention

[0004] In view of this, it is necessary to provide a millimeter-wave degraded image denoising and deblurring method, device, electronic device and computer storage medium to solve the problem in the prior art that it is difficult to effectively balance the relationship between the denoising task and the deblurring task, resulting in the image being still unclear after denoising and deblurring.

[0005] In order to solve the above problems, in a first aspect, the present invention provides a millimeter wave degraded image denoising and deblurring method, comprising:

[0006] Construct training and test sets of millimeter-wave simulated degraded images;

[0007] Inputting the training set images into the initial dual-task network model for training until the loss function of the initial dual-task network model reaches a preset convergence condition, wherein the structure of the initial dual-task network model includes a local additive residual dense feature fusion denoising subnetwork, a local low-order to high-order semi-normalized residual feature dense fusion denoising and deblurring subnetwork, and an image reconstruction subnetwork;

[0008] Input the test set image into the trained dual-task network model and output the test restoration image.

[0009] Furthermore, the construction of a training set and a test set of millimeter wave simulated degraded images includes:

[0010] Acquire remote sensing image sets;

[0011] Adding light, medium and heavy Gaussian blur to the remote sensing images in the remote sensing image set, and adding weak, medium and strong Gaussian noise accordingly, so as to form millimeter wave simulated degradation images of three degrees: low, medium and high;

[0012] Splicing the millimeter wave simulated degraded image with a clear image, and cropping and normalizing the spliced ​​image to obtain a training set and a test set of the millimeter wave simulated degraded image;

[0013] The constructing of the training set of millimeter wave simulated degraded images also includes:

[0014] The low, medium and high millimeter-wave simulated degraded images in the training set are augmented to obtain the final training set, and the proportion of low, medium and high millimeter-wave simulated degraded images increases progressively.

[0015] Furthermore, the local additive residual dense feature fusion denoising sub-network includes a convolutional layer and a local additive residual dense feature fusion block;

[0016] The local additive residual dense feature fusion block is composed of a plurality of additive residual block dense feature fusions;

[0017] The first image feature is extracted through the local additive residual dense feature fusion denoising sub-network.

[0018] Furthermore, the denoising and deblurring subnetwork for densely fusing local low-order to high-order semi-normalized residual features includes an encoder and a decoder;

[0019] The encoder includes a plurality of encoding units, and the decoder includes a plurality of decoding units, each encoding unit and each decoding unit includes a convolutional layer and an addition-splicing semi-normalized residual dense block, and the addition-splicing semi-normalized dense residual block is composed of a plurality of semi-normalized residual blocks densely added and spliced;

[0020] The second image feature is extracted through the denoising and deblurring sub-network that densely fuses the local low-order to high-order semi-normalized residual features.

[0021] Furthermore, the multiple semi-normalized dense additions and concatenation connections include:

[0022] Adjacent semi-normalized residual blocks use short skip connections for residual mapping, and non-adjacent semi-normalized residual blocks use long skip connections for residual mapping;

[0023] The semi-normalized residual block includes:

[0024] Half of the channels in the residual block are subjected to instance regularization, while the other half are not subjected to regularization.

[0025] Furthermore, the local denoising and deblurring subnetwork for dense fusion of low-order to high-order semi-normalized residual features further includes:

[0026] Introducing a cross jump connection between the encoder and the decoder to cross-fuse the encoding feature information and the decoding feature information;

[0027] A cross jump connection is introduced between the encoder and the decoder to cross-fuse the encoding feature information and the decoding feature information, specifically including:

[0028] The deep features are upsampled using 3×3 convolution and upsampling calculations and then fused with the shallow features. The fused results are fused with the output features of the decoder decoding block, and the fused features are input into the next decoding block of the decoder.

[0029] Furthermore, the image reconstruction subnetwork includes a local additive residual dense feature fusion block, a convolutional layer and a Tanh activation function;

[0030] The first image feature and the second image feature are fused and reconstructed through the image reconstruction subnetwork.

[0031] Furthermore, the training process of the dual-task network model includes:

[0032] Inputting the training set image into the local additive residual dense feature fusion denoising subnetwork to extract the first training image feature;

[0033] Fusing the first training image feature with the training set image to obtain a fused image feature;

[0034] Inputting the fused image features into the denoising and deblurring subnetwork of dense fusion of local low-order to high-order semi-normalized residual features to extract second training image features;

[0035] The first training image feature and the second training image feature are subjected to feature fusion and then input into the image reconstruction subnetwork, and a restored image in the training process is output;

[0036] The initial dual-task network model is trained using the Adam algorithm, training parameters are set, and MSE loss and perceptual loss are used as loss functions for training;

[0037] The weights of the initial dual-task network model are iteratively updated until the loss function reaches a preset convergence condition.

[0038] In a second aspect, the present invention further provides a millimeter wave degraded image denoising and deblurring device, comprising:

[0039] A data set construction module, used to construct training and test sets of millimeter-wave simulated degraded images;

[0040] A model training module, used for inputting the training set images into the initial dual-task network model for training until the loss function of the initial dual-task network model reaches a preset convergence condition, wherein the structure of the initial dual-task network model includes a local additive residual dense feature fusion denoising subnetwork, a local low-order to high-order semi-normalized residual feature dense fusion denoising and deblurring subnetwork, and an image reconstruction subnetwork;

[0041] The model testing module is used to input the test set images into the trained dual-task network model and output the test restoration images.

[0042] In a third aspect, the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps in the above-mentioned millimeter-wave degraded image denoising and deblurring method when executing the computer program.

[0043] In a fourth aspect, the present invention further provides a computer storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned millimeter-wave degraded image denoising and deblurring method are implemented.

[0044] The beneficial effect of adopting the above embodiment is: the present invention trains the initial dual-task network model based on the training set of the millimeter-wave simulated degraded image, and after the model training is completed, the noise and blur effects in the millimeter-wave degraded image can be removed. Then the test set image is input into the trained dual-task network model, and the image features of the degraded image are extracted by the local additive residual dense feature fusion denoising sub-network, while suppressing certain noise; then the features are input into the local denoising and deblurring sub-network with dense fusion of semi-normalized residual features from low order to high order to quickly extract richer image features; finally, the output feature map is input into the image reconstruction sub-network again to output the test restoration image. The present invention can effectively restore infrared millimeter-wave degraded images with blur noise to obtain clear images. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A schematic diagram of a flow chart of an embodiment of a method for denoising and deblurring a millimeter-wave degraded image provided by the present invention;

[0046] Figure 2 A millimeter wave simulated degraded image provided by an embodiment of the present invention;

[0047] FIG3(a)-(c) are respectively millimeter wave simulated degraded images with degradation levels ranging from light to heavy provided by an embodiment of the present invention, and FIG3(d)-(f) are respectively denoised and deblurred test restored images provided by an embodiment of the present invention;

[0048] Figure 4 A residual block provided by an embodiment of the present invention;

[0049] Figure 5 A semi-normalized residual block provided by an embodiment of the present invention;

[0050] Figure 6 A feature fusion graph between multiple semi-normalized residual blocks provided by an embodiment of the present invention;

[0051] Figure 7 A schematic diagram of a flow chart of a training process of a dual-task network model provided by an embodiment of the present invention;

[0052] Figure 8 A schematic diagram of the structure of a local additive residual dense feature fusion denoising sub-network, a local low-order to high-order semi-normalized residual feature dense fusion denoising and deblurring sub-network and an image reconstruction sub-network provided by an embodiment of the present invention;

[0053] Fig. 9 A schematic structural diagram of an embodiment of a millimeter wave degraded image denoising and deblurring device provided by the present invention;

[0054] Fig.10A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0055] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0056] In the description of the present invention, "multiple" means two or more, unless otherwise clearly and specifically limited. Reference to "embodiment" in this article means that the specific features, structures or characteristics described in conjunction with the embodiment may be included in at least one embodiment of the present invention. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0057] The present invention provides a millimeter wave degraded image denoising and deblurring method, device, equipment and medium, which are described below respectively.

[0058] See also Figure 1 , Figure 1 A flow chart of an embodiment of a method for denoising and deblurring a millimeter-wave degraded image provided by the present invention is provided. A specific embodiment of the present invention discloses an image denoising and deblurring method, comprising:

[0059] Step S101: constructing a training set and a test set of millimeter wave simulated degraded images;

[0060] Among them, during the imaging process of millimeter wave degraded images, due to the long wavelength of millimeter waves, small antenna aperture, and the introduction of various types of random noise, the spatial resolution of the image after imaging is low and the image quality is seriously damaged. Therefore, millimeter wave degraded images have both noise problems and blur problems.

[0061] Therefore, it is necessary to perform image restoration on the millimeter wave degraded image. Before performing image restoration on the millimeter wave degraded image, it is necessary to construct a training set and a test set of millimeter wave simulated degraded images. It can be understood that the training set and the test set of millimeter wave simulated degraded images include image pairs consisting of degraded images and clear images.

[0062] In one embodiment of the present invention, constructing a training set and a test set of millimeter wave simulated degraded images includes:

[0063] Acquire remote sensing image sets;

[0064] Add light, medium and heavy Gaussian blur to the remote sensing images in the remote sensing image set, and add weak, medium and strong Gaussian noise accordingly to form millimeter wave simulated degradation images with low, medium and high degrees;

[0065] Splicing the millimeter wave simulated degraded image with the clear image, and cropping and normalizing the spliced ​​image to obtain a training set and a test set of the millimeter wave simulated degraded image;

[0066] Constructing a training set of millimeter-wave simulated degraded images, including:

[0067] The low, medium and high millimeter-wave simulated degraded images in the training set are augmented to obtain the final training set, and the proportion of low, medium and high millimeter-wave simulated degraded images increases progressively.

[0068] Optionally, the remote sensing images may use a remote sensing image dataset in NWPU VHR-10, which includes 800 remote sensing images. A training set and a test set of millimeter wave simulated degraded images may be produced based on the remote sensing images in the remote sensing image dataset.

[0069] Specifically, all remote sensing images can be subjected to noise blur degradation, and corresponding Gaussian blur and Gaussian noise can be added to obtain millimeter wave simulated degraded images. Specifically, light, medium, and heavy Gaussian blur can be added to the remote sensing images in the remote sensing image set, and weak, medium, and strong Gaussian noise can be added accordingly to form millimeter wave simulated degraded images of low, medium, and high degrees;

[0070] See also Figure 2 , Figure 2 A millimeter wave degraded image is provided by an embodiment of the present invention.

[0071] Among them, the formulas of Gaussian noise N(x) and Gaussian blur B(x) are as follows:

[0072]

[0073]

[0074] γ represents the parameter of Gaussian noise, μ and σ represent the parameters of Gaussian blur, respectively.

[0075] By setting different parameters, images with different degrees of degradation can be obtained. After obtaining the millimeter wave simulated degradation image, the degraded image can be spliced ​​with the clear image to form a degraded-clear image pair, and 600 image pairs are randomly selected as the training set, and the remaining 200 image pairs are used as the test set.

[0076] Furthermore, it is understandable that in order to achieve the purpose of fast training, the image pairs can be cropped into the same size, such as 256×256, and normalized to [-1,1]. In addition, in order to improve the effect of network training, it is necessary to expand the number of training sets and perform data augmentation on the training sets. Specifically, data similar to but different from the original training set can be randomly generated through operations such as horizontal / vertical flipping and rotation. The low, medium, and high millimeter-wave simulated degraded images in the training set can also be augmented to obtain the final training set, and the proportion of the number of low, medium, and high millimeter-wave simulated degraded images increases progressively.

[0077] Step S102: input the training set images into the initial dual-task network model for training until the loss function of the initial dual-task network model reaches a preset convergence condition, wherein the structure of the initial dual-task network model includes a local additive residual dense feature fusion denoising subnetwork, a local low-order to high-order semi-normalized residual feature dense fusion denoising and deblurring subnetwork, and an image reconstruction subnetwork;

[0078] After the training set and test set of millimeter-wave simulated degraded images are constructed, the training set needs to be trained, so an initial dual-task network model can be built. The network structure of this model consists of three parts, namely, a local additive residual dense feature fusion denoising sub-network, a local denoising and deblurring sub-network with dense fusion of low-order to high-order semi-normalized residual features, and an image reconstruction sub-network.

[0079] It should be noted that the local additive residual dense feature fusion denoising subnetwork is composed of the feature addition of dense residual blocks, which can extract image features and suppress certain noise; the local denoising and deblurring subnetwork with dense fusion of low-order to high-order semi-normalized residual features is a network based on an encoder-decoder structure, which can extract richer image features. Specifically, the encoder output image features of the denoising and deblurring subnetwork and the decoder output image features are cross-fused from deep features to shallow features; the image reconstruction subnetwork is to obtain the reconstructed image result after feature fusion of the image features output by the above two subnetworks.

[0080] In the process of training the initial dual-task network model according to the training set, the parameters in the model can be iteratively modified until the loss function of the initial dual-task network model reaches a preset convergence condition until the model training is completed.

[0081] Step S103: Input the test set image into the trained dual-task network model and output the test restored image.

[0082] It can be understood that after the model training is completed, the best fitting solution of a model can be obtained. At this time, the test set image is used as the input of the model. By comparing the output test restoration image with the clear image, it can be seen that the test restoration image can have a high-quality restoration result.

[0083] Please refer to Figure 3, Figure 3 (a)-(c) are millimeter wave simulated degraded images with degradation levels ranging from light to heavy provided by an embodiment of the present invention, and Figure 3 (d)-(f) are corresponding denoised and deblurred test restoration images provided by an embodiment of the present invention.

[0084] In addition, the image quality can be objectively evaluated based on the test restored image and the clear image. The evaluation indicators can be the peak signal-to-noise ratio (PSNR), structural similarity (SSIM) and visual fidelity (VIF) of the image.

[0085] The present invention trains an initial dual-task network model based on a training set of millimeter-wave simulated degraded images. After the model training is completed, the noise and blur effects in the millimeter-wave degraded images can be removed. Then the test set image is input into the trained dual-task network model, and the image features of the degraded image are extracted through the local additive residual dense feature fusion denoising sub-network, while suppressing certain noise; then the features are input into the local denoising and deblurring sub-network with dense fusion of low-order to high-order semi-normalized residual features to quickly extract richer image features; finally, the output feature map is input into the image reconstruction sub-network again to output the test restoration image. The present invention can effectively restore infrared millimeter-wave degraded images with blur noise to obtain clear images.

[0086] In one embodiment of the present invention, the local additive residual dense feature fusion denoising sub-network includes a convolutional layer and a local additive residual dense feature fusion block;

[0087] The local additive residual dense feature fusion block is composed of multiple additive residual block dense feature fusion;

[0088] The first image feature is extracted through the local additive residual dense feature fusion denoising sub-network.

[0089] It can be understood that the local additive residual dense feature fusion denoising sub-network can extract image features from the input degraded image and suppress noise to a certain extent.

[0090] Specifically, the local additive residual dense feature fusion denoising sub-network consists of a convolutional layer and three residual blocks densely connected in a feature addition manner, and the convolution kernel size of the convolution layer is 5×5; the residual block consists of two convolutions with a convolution kernel size of 3×3 and a ReLU activation function, and the number of convolution kernels is 64.

[0091] For example, the millimeter wave degraded image is denoted as X, and the first image feature F is extracted after the local additive residual dense feature fusion denoising sub-network. l , and F l =Net l (X), where Net l represents the local additive residual dense feature fusion denoising sub-network, which fuses the first image feature obtained with the input image and uses it as the input of the next sub-network, that is, X d =X+F l .

[0092] In one embodiment of the present invention, the denoising and deblurring subnetwork for dense fusion of local low-order to high-order semi-normalized residual features includes an encoder and a decoder;

[0093] The encoder includes a plurality of encoding units, and the decoder includes a plurality of decoding units. Each encoding unit and each decoding unit includes a convolutional layer and an addition-splicing semi-normalized residual dense block. The addition-splicing semi-normalized residual dense block is composed of a plurality of semi-normalized residual blocks densely connected, added, and spliced.

[0094] The second image feature is extracted through a denoising and deblurring sub-network that densely fuses local low-order to high-order semi-normalized residual features.

[0095] It can be understood that the denoising and deblurring sub-network that densely fuses local low-order to high-order semi-normalized residual features can learn richer image features, specifically by increasing the depth of the model and the receptive field.

[0096] Specifically, the network structure of the denoising and deblurring subnetwork with dense fusion of local low-order to high-order semi-normalized residual features is based on an encoder-decoder, and image features are learned at multiple scales by performing downsampling and upsampling twice respectively. Optionally, the encoder in the embodiment of the present invention includes three encoding units, each of which is composed of a convolutional layer for processing the input and an addition-splicing semi-normalized dense residual block. The decoder is similar to the encoder and also includes three decoding units, each of which is composed of an addition-splicing semi-normalized dense residual block and a convolutional layer. The convolution kernel size of the convolutional layer of the encoding unit and the decoding unit is 3×3.

[0097] Optionally, the addition-splicing semi-normalized residual dense block in the embodiment of the present invention includes a total of six semi-normalized residual blocks, and the six semi-normalized residual blocks are stacked horizontally. For example, the output result of the overall module is F d =Net d (X d ), where F d Represents the second image feature, Net dRepresents a denoising and deblurring subnetwork with dense fusion of local low-order to high-order semi-normalized residual features.

[0098] Specifically, based on the idea of ​​nested dense skip connections, any two semi-normalized residual blocks are connected, adjacent semi-normalized residual blocks use short skip connections for residual mapping, and non-adjacent semi-normalized residual blocks use long skip connections for residual mapping.

[0099] More specifically, every three semi-normalized residual blocks are divided into one category, and the output features between the semi-normalized residual blocks are fused by feature addition and splicing fusion methods, respectively, to ensure that connections are established between adjacent and non-adjacent semi-normalized residual block layers.

[0100] See also Figure 4 , Figure 4 A residual block provided by an embodiment of the present invention includes two convolutional layers, one activation layer, and a skip connection from input to output.

[0101] See also Figure 5 , Figure 5 A semi-normalized residual block provided in an embodiment of the present invention is composed of a 1×1 convolution, two 3×3 convolutions, an activation function and a skip connection.

[0102] Specifically, first, the feature map input is subjected to a 3×3 convolution, and the output feature map is divided into two parts according to the number of channels. One part of the feature map is normalized by the normalization function, and the other part is not processed by the identity function. This can prevent the network from being affected by the batchsize during training and retain more specific scale information. Secondly, the two parts of the output feature map are channel-joined, and then passed through the activation function and 3×3 convolution, and then the activation function is passed to output the feature map. Finally, the input feature map is added to the output feature map through a jump connection through a 1×1 convolution to obtain the final feature map.

[0103] See also Figure 6 , Figure 6 A feature fusion diagram between multiple semi-normalized residual blocks provided by an embodiment of the present invention. Figure 6 It can be seen that the first three layers and the last three layers are connected from low order to high order, and the input features of the low-order semi-normalized residual block are also used as the input of the high-order semi-normalized residual block. The first three layers are feature added with the input features of the current layer, and the last three layers are feature concatenated with the input features of the current layer. The output of the first three layers is used as the input of the last three layers. The first three layers and the last three layers can be expressed as:

[0104] X n-2 =H(X n-3 )+X n-3 ,

[0105] X n-1 =H(H(X n-3 )+X n-3 )+H(X n-3 )+X n-3 ,

[0106] X n =H(H(H(X n-3 )+X n-3 )+H(X n-3 )+X n-3 )+H(X n-3 +HXn--3+HXn-3+Xn-3,

[0107] T n-2 =C(H(T n-3 ), T n-3 ),

[0108] T n-1 =C(H(C(T n-3 , H(T n-3 ))), H(T n-3 ), T n-3 ),

[0109] T n =

[0110] C(H(C(H(C(T n-3 , H(T n-3 ))), H(T n-3 ), T n-3 )),

[0111] HC (HTn-3, Tn-3), HTn-3, Tn-3.

[0112] The above formula and Figure 6 X in n-1 , X n-2 and X n-3 Represents the input of each semi-normalized residual block of the first three layers, X n and H represent the output results of the first three layers of the module and the semi-normalized residual block function respectively. n-1 、T n-2 and T n-3 They represent the input of each residual block of the last three layers, where the output of the first three layers is the input of the last three layers, i.e., X n =T n-3 , M represents the output of the last three layers T n The result obtained after 1×1 convolution, Y is the final output result obtained by adding and fusion of the input of the previous and next three layers and M, C represents the feature map channel splicing operation, and + represents the feature map addition operation.

[0113] For the first three layers, the feature addition method is used to connect the features between layers, while the feature dimension splicing method is used for the last three layers. In order to ensure that the dimension of the extracted second image feature is consistent with the input feature dimension, 1×1 convolution is used for dimensionality reduction. This can not only extract local features, but also retain the feedforward relationship and make full use of the feature information of each block.

[0114] In one embodiment of the present invention, the denoising and deblurring sub-network for dense fusion of local low-order to high-order semi-normalized residual features further includes:

[0115] Introduce cross-jump connections between the encoder and decoder to cross-fuse the encoding feature information and the decoding feature information;

[0116] A cross-jump connection is introduced between the encoder and decoder to fuse feature information, including:

[0117] The deep features are upsampled using 3×3 convolution and upsampling calculations and then fused with the shallow features. The fused results are fused with the output features of the decoder decoding block, and the fused features are input into the next decoding block of the decoder.

[0118] It can be understood that by introducing a cross-jump connection between the encoder and the decoder to fuse feature information, the encoder can guide the feature reconstruction in the decoder.

[0119] In one embodiment of the present invention, the image reconstruction subnetwork includes a local additive residual dense feature fusion block, a convolutional layer and a Tanh activation function;

[0120] The first image features and the second image features are fused and reconstructed through the image reconstruction subnetwork.

[0121] It can be understood that the image reconstruction subnetwork can reconstruct the acquired feature map to obtain a clear image. The feature map in the embodiment of the present invention is the first image feature and the second image feature. By adding and fusing the first image feature and the second image feature as the input of the image reconstruction subnetwork, the expression ability of the network can be increased.

[0122] Specifically, the image reconstruction subnetwork consists of three residual blocks with dense feature addition, a convolutional layer, and a Tanh activation function, and the convolution kernel size of the convolutional layer is 5×5.

[0123] Optionally, the local additive residual dense feature is fused with the first image feature F output by the denoising subnetwork l The second image feature F is the output of the denoising and deblurring subnetwork densely fused with the local low-order to high-order semi-normalized residual features d Add and fuse them and use them as the input X of the image reconstruction subnetworkr After adding three layers of dense features to the residual block, the convolution layer and the Tanh activation function, the reconstructed image can be obtained. Among them, Net r represents the image reconstruction subnetwork, This is the final output restored image.

[0124] In one embodiment of the present invention, see Figure 7 , Figure 7 A flowchart of a training process of a dual-task network model provided by an embodiment of the present invention includes:

[0125] Step S701: input the training set image into the local additive residual dense feature fusion denoising sub-network to extract the first training image feature;

[0126] Step S702: Fusing the first training image feature with the training set to obtain a fused image feature;

[0127] Step S703: input the fused image features into a denoising and deblurring subnetwork with dense fusion of local low-order to high-order semi-normalized residual features to extract the second training image features;

[0128] Step S704: Fusing the first training image features with the second training image features, and inputting the results into the image reconstruction subnetwork, and outputting the restored image during the training process;

[0129] Step S705: using the Adam algorithm to train the initial dual-task network model, setting training parameters, and using MSE loss and perceptual loss as training loss functions;

[0130] Step S706: Iteratively update the weights of the initial dual-task network model until the loss function reaches a preset convergence condition.

[0131] First of all, it should be noted that the Adam algorithm is used to train the initial dual-task network model in the embodiment of the present invention, and the number of training epochs is 600. The initial learning rate of the network in the embodiment of the present invention is 1e -4 , when the training is halfway through, the learning rate decays linearly until 1e -6 , the batch size is 2.

[0132] For a better understanding of the present invention, please refer to Figure 8 , Figure 8 A network structure diagram of a dual-task network model provided by an embodiment of the present invention.

[0133] The dual-task network model consists of three parts: a local additive residual dense feature fusion denoising sub-network, a local low-order to high-order semi-normalized residual feature dense fusion denoising and deblurring sub-network, and an image reconstruction sub-network. The local additive residual dense feature fusion denoising sub-network consists of a convolutional layer and three residual blocks with dense feature addition, such as Figure 8 As shown in the figure, the convolution kernel size of the convolution layer is 5×5, and the residual block consists of two convolutions with a convolution kernel size of 3×3 and a ReLU activation function, where the number of features of the convolution kernel is 64.

[0134] By inputting the training set image into the local additive residual dense feature fusion denoising sub-network, the first training image feature can be extracted from the degraded image in the training set, and the noise can be suppressed to a certain extent. The output first training image feature is additively fused with the training set as the input of the local denoising and deblurring sub-network with dense fusion of semi-normalized residual features from low order to high order for residual mapping.

[0135] The denoising and deblurring sub-network with dense fusion of local low-order to high-order semi-normalized residual features is based on the encoder-decoder structure, such as Figure 8 As shown, the encoder includes three encoding units, each consisting of a convolutional layer for processing the input and an addition-splicing semi-normalized residual dense block, and the convolution kernel size of the convolutional layer is 3×3. The decoder is similar to the encoder and includes three decoding units, each consisting of an addition-splicing semi-normalized residual dense block and a convolutional layer, and the convolution kernel parameters are consistent. A cross jump connection is introduced between the encoder and the decoder to cross-fuse the encoding feature information and the decoding feature information.

[0136] By inputting the fused image into the denoising and deblurring sub-network that densely fuses the local low-order to high-order semi-normalized residual features, richer image features can be learned, the network depth can be increased, and the receptive field can be enlarged. Specifically, image features can be learned at multiple scales by performing downsampling and upsampling twice respectively.

[0137] It should be noted that, based on the idea of ​​nested dense skip connections, any two semi-normalized residual blocks are connected, adjacent semi-normalized residual blocks use short skip connections for residual mapping, and non-adjacent semi-normalized residual blocks use long skip connections for residual mapping. By dividing every three semi-normalized residual blocks into one category, the output features between the semi-normalized residual blocks are fused by feature addition and splicing, respectively, to ensure that connections are established between adjacent and non-adjacent residual block layers. Cross-skip connections are introduced between the encoder and the decoder to cross-fuse the encoding feature information and the decoding feature information; 3×3 convolution and upsampling calculations are used to upsample the deep features and then fuse them with the shallow features for feature maps, and the fusion results are fused with the decoder decoding block output features, and the fused features are input into the next decoding block of the decoder. Therefore, the second training image output by the denoising and deblurring subnetwork, which has densely fused local semi-normalized residual features from low order to high order, has richer features.

[0138] The image reconstruction subnetwork consists of three residual blocks with dense feature addition, a convolutional layer, and a Tanh activation function, such as Figure 8 As shown, the convolution kernel size of the convolution layer is 3×3.

[0139] The image reconstruction subnetwork increases the network's expressive power by taking the first training image features and the second training image features as input, and can reconstruct a clear image based on the acquired image features, that is, a restored image during the training process.

[0140] It is understandable that, during the training process, the weights of the initial dual-task network model can be iteratively updated based on the comparison between the output restored image and the clear image in the training set. In addition, the embodiment of the present invention uses MSE loss and perceptual loss as the loss function L for training. total :

[0141] L total =λ1L MSE +2L P

[0142] Among them, L MSE The loss represents the MSE loss between the restored image output during training and the real clear image; L P The loss represents the perceptual loss, which can be understood as the similarity between the output image and the real clear image obtained through the high-dimensional feature space of the pre-trained network. And λ1 and λ2 are the weights between the loss functions, which can be set to 1.0 and 0.01 respectively.

[0143] By continuously adjusting the weights of the initial dual-task network until the loss function reaches the preset convergence condition, the training is completed and a dual-task network model is formed.

[0144] In order to better implement the millimeter wave degraded image denoising and deblurring method in the embodiment of the present invention, based on the millimeter wave degraded image denoising and deblurring method, correspondingly, please refer to Fig. 9 , Fig. 9 This is a schematic diagram of the structure of an embodiment of a millimeter wave degraded image denoising and deblurring device provided by the present invention. The embodiment of the present invention provides a millimeter wave degraded image denoising and deblurring device 900, comprising:

[0145] A data set construction module 901 is used to construct a training set and a test set of millimeter wave simulated degraded images;

[0146] The model training module 902 is used to input the training set images into the initial dual-task network model for training until the loss function of the initial dual-task network model reaches a preset convergence condition, wherein the structure of the initial dual-task network model includes a local additive residual dense feature fusion denoising sub-network, a local low-order to high-order semi-normalized residual feature dense fusion denoising and deblurring sub-network, and an image reconstruction sub-network;

[0147] The model testing module 903 is used to input the test set image into the trained dual-task network model and output the test restoration image.

[0148] It should be noted here that: the device 900 provided in the above embodiment can implement the technical solutions described in the above method embodiments. The specific implementation principles of the above modules or units can refer to the corresponding contents in the above method embodiments, which will not be repeated here.

[0149] Based on the above-mentioned millimeter wave degraded image denoising and deblurring method, an embodiment of the present invention also provides an electronic device accordingly, including: a processor and a memory, and a computer program stored in the memory and executable on the processor; when the processor executes the computer program, the steps in the millimeter wave degraded image denoising and deblurring method in the above-mentioned embodiments are implemented.

[0150] Fig.10 1 shows a schematic diagram of the structure of an electronic device 1000 suitable for implementing an embodiment of the present invention. The electronic device in the embodiment of the present invention may include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.10 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0151] The electronic device includes: a memory and a processor, wherein the processor here may be referred to as a processing device 1001 hereinafter, and the memory may include at least one of a read-only memory (ROM) 1002, a random access memory (RAM) 1003, and a storage device 1008 hereinafter, as specifically shown below:

[0152] like Fig.10 As shown, the electronic device 1000 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1008 into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for the operation of the electronic device 1000 are also stored. The processing device 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0153] Typically, the following devices may be connected to the I / O interface 1005: an input device 1006 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 1007 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 1008 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 may allow the electronic device 1000 to communicate with other devices wirelessly or by wire to exchange data. Although Fig.10 The electronic device 1000 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0154] In particular, according to an embodiment of the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 1009, or installed from the storage device 1008, or installed from the ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment of the present invention are executed.

[0155] Based on the above-mentioned millimeter wave degraded image denoising and deblurring method, an embodiment of the present invention also provides a computer-readable storage medium accordingly, which stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the millimeter wave degraded image denoising and deblurring method as in the above-mentioned embodiments.

[0156] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0157] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A millimeter wave degraded image denoising and deblurring method, characterized in that: include: Construct training and test sets of millimeter-wave simulated degraded images; Inputting the training set images into the initial dual-task network model for training until the loss function of the initial dual-task network model reaches a preset convergence condition, wherein the structure of the initial dual-task network model includes a local additive residual dense feature fusion denoising subnetwork, a local low-order to high-order semi-normalized residual feature dense fusion denoising and deblurring subnetwork, and an image reconstruction subnetwork; The denoising and deblurring sub-network for dense fusion of local low-order to high-order semi-normalized residual features includes an encoder and a decoder; The encoder includes a plurality of encoding units, and the decoder includes a plurality of decoding units, each encoding unit and each decoding unit includes a convolutional layer and an addition-splicing semi-normalized residual dense block, and the addition-splicing semi-normalized dense residual block is composed of a plurality of semi-normalized residual blocks densely added and spliced; Extracting the second image feature through the denoising and deblurring sub-network of dense fusion of local low-order to high-order semi-normalized residual features; The local denoising and deblurring sub-network for dense fusion of low-order to high-order semi-normalized residual features also includes: Introducing a cross jump connection between the encoder and the decoder to cross-fuse the encoding feature information and the decoding feature information; A cross jump connection is introduced between the encoder and the decoder to cross-fuse the encoding feature information and the decoding feature information, specifically including: Use 3×3 convolution and upsampling calculation to upsample the deep features and then fuse them with the shallow features. The fused results are fused with the output features of the decoder decoding block, and the fused features are input into the next decoding block of the decoder. Input the test set image into the trained dual-task network model and output the test restoration image.

2. The method for denoising and deblurring millimeter wave degraded images according to claim 1, characterized in that: The step of constructing a training set and a test set of millimeter wave simulated degraded images includes: Acquire remote sensing image sets; Adding light, medium and heavy Gaussian blur to the remote sensing images in the remote sensing image set, and adding weak, medium and strong Gaussian noise accordingly, so as to form millimeter wave simulated degradation images of three degrees: low, medium and high; Splicing the millimeter wave simulated degraded image with a clear image, and cropping and normalizing the spliced ​​image to obtain a training set and a test set of the millimeter wave simulated degraded image; The constructing of the training set of millimeter wave simulated degraded images also includes: The low, medium and high millimeter-wave simulated degraded images in the training set are augmented to obtain the final training set, and the proportion of low, medium and high millimeter-wave simulated degraded images increases progressively.

3. The method for denoising and deblurring millimeter wave degraded images according to claim 1, characterized in that: The local additive residual dense feature fusion denoising sub-network includes a convolutional layer and a local additive residual dense feature fusion block; The local additive residual dense feature fusion block is composed of a plurality of additive residual block dense feature fusions; The first image feature is extracted through the local additive residual dense feature fusion denoising sub-network.

4. The method for denoising and deblurring millimeter wave degraded images according to claim 3, characterized in that: The image reconstruction subnetwork includes a local additive residual dense feature fusion block, a convolutional layer and a Tanh activation function; The first image feature and the second image feature are fused and reconstructed through the image reconstruction subnetwork.

5. The method for denoising and deblurring millimeter wave degraded images according to claim 1, characterized in that: The training process of the dual-task network model includes: Inputting the training set image into the local additive residual dense feature fusion denoising subnetwork to extract the first training image feature; Fusing the first training image feature with the training set image to obtain a fused image feature; Inputting the fused image features into the denoising and deblurring subnetwork of dense fusion of local low-order to high-order semi-normalized residual features to extract second training image features; The first training image features and the second training image features are subjected to feature fusion and then input into the image reconstruction subnetwork, and a restored image in the training process is output; The initial dual-task network model is trained using the Adam algorithm, training parameters are set, and MSE loss and perceptual loss are used as loss functions for training; The weights of the initial dual-task network model are iteratively updated until the loss function reaches a preset convergence condition.

6. A millimeter wave degraded image denoising and deblurring device, characterized in that: include: A data set construction module, used to construct training sets and test sets of millimeter-wave simulated degraded images; A model training module, used for inputting the training set images into the initial dual-task network model for training until the loss function of the initial dual-task network model reaches a preset convergence condition, wherein the structure of the initial dual-task network model includes a local additive residual dense feature fusion denoising subnetwork, a local low-order to high-order semi-normalized residual feature dense fusion denoising and deblurring subnetwork, and an image reconstruction subnetwork; The denoising and deblurring sub-network for dense fusion of local low-order to high-order semi-normalized residual features includes an encoder and a decoder; The encoder includes a plurality of encoding units, and the decoder includes a plurality of decoding units, each encoding unit and each decoding unit includes a convolutional layer and an addition-splicing semi-normalized residual dense block, and the addition-splicing semi-normalized dense residual block is composed of a plurality of semi-normalized residual blocks densely added and spliced; Extracting the second image feature through the denoising and deblurring sub-network of dense fusion of local low-order to high-order semi-normalized residual features; The local denoising and deblurring sub-network for dense fusion of low-order to high-order semi-normalized residual features also includes: Introducing a cross jump connection between the encoder and the decoder to cross-fuse the encoding feature information and the decoding feature information; A cross jump connection is introduced between the encoder and the decoder to cross-fuse the encoding feature information and the decoding feature information, specifically including: Use 3×3 convolution and upsampling calculation to upsample the deep features and then fuse them with the shallow features. The fused results are fused with the output features of the decoder decoding block, and the fused features are input into the next decoding block of the decoder. The model testing module is used to input the test set images into the trained dual-task network model and output the test restoration images.

7. An electronic device, characterized in that: It includes a memory and a processor, wherein the memory is used to store a program; the processor is coupled to the memory and is used to execute the program stored in the memory to implement the steps in the millimeter wave degraded image denoising and deblurring method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: Used to store computer-readable programs or instructions, which, when executed by a processor, can implement the steps of the millimeter wave degraded image denoising and deblurring method described in any one of claims 1 to 5 above.