A Method, System, Storage Medium and Device for Colorizing SAR Images

Abstract

The present invention relates to the field of image processing technology, and discloses a method, system, storage medium and device for colorizing SAR images, including: obtaining a SAR image and extracting its spatial features and frequency features; inputting the spatial features and frequency features into a conditional diffusion model, which includes a spatial-frequency information interaction network, and performing forward noise addition and reverse denoising on the SAR image through the spatial-frequency information interaction network; in the process of forward noise addition, the spatial-frequency information interaction network predicts and outputs the noise to be added for each time series according to the spatial features and frequency features; sequentially adding the noise to be added to the SAR image in the order of the time series to obtain a pure noise image; in the process of reverse denoising, the spatial-frequency information interaction network predicts and outputs the noise to be removed for each time series according to the pure noise image; removing the noise to be removed from the pure noise image in the order of the time series to obtain a color image. The present invention can improve the quality of the colorization result of SAR images.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular, to a method, system, storage medium, and device for colorizing SAR images. Background Art

[0002] Synthetic Aperture Radar (SAR), as an advanced imaging technology, can stably acquire surface images under various complex weather and lighting conditions by virtue of its excellent penetration and all-weather working ability. This unique advantage makes SAR images an indispensable information source in the field of earth observation such as remote sensing detection, geological exploration, environmental monitoring, disaster assessment, and resource exploration. SAR images not only provide detailed spatial distribution information but also contain rich radiation data, which are of extremely important value for accurately identifying surface features and understanding surface change processes.

[0003] Although SAR images play a crucial role in the field of earth observation, their inherent speckle noise and gray-scale characteristics have become key factors restricting their application effects. The existence of speckle noise often masks important visual information in the image, reduces the clarity and interpretability of the image, and makes it extremely difficult to identify and analyze surface features. At the same time, the gray-scale characteristics of SAR images also limit the information depth and intuitiveness of the images, which is not conducive to professional analysts quickly and accurately obtaining the required information.

[0004] In view of the importance of SAR images in the field of earth observation and the challenges they face, it is particularly important to develop effective SAR image colorization techniques, which can more accurately analyze image information and thus play a greater role in practical applications. Summary of the Invention

[0005] Based on this, it is necessary to propose a method for colorizing SAR images in view of the above problems.

[0006] A method for colorizing SAR images, the method comprising the following steps:

[0007] Obtain a SAR image, and extract the spatial features and frequency features of the SAR image;

[0008] Input the spatial features and frequency features into a conditional diffusion model, the conditional diffusion model includes a spatial-frequency information interaction network, and perform forward denoising and backward denoising on the SAR image through the spatial-frequency information interaction network;

[0009] During the forward denoising process, the spatial-frequency information interaction network predicts and outputs the noise to be added for each time series according to the spatial features and frequency features;

[0010] Add the noise to be added to the SAR image in sequence according to the order of the time series to obtain a pure noise image;

[0011] In the reverse denoising process, the spatial frequency information interaction network predicts and outputs the noise to be removed for each time series according to the pure noise image;

[0012] Remove the noise to be removed from the pure noise image in sequence according to the order of the time series to obtain a color image.

[0013] In the above solution, the extraction of the spatial features and frequency features of the SAR image specifically includes:

[0014] Perform a learnable wavelet transform on the SAR image to obtain frequency features;

[0015] Perform downsampling on the SAR image to obtain spatial features.

[0016] In the above solution, the spatial frequency information interaction network includes a convolutional layer, two downsampling processing modules, a multi-scale spatial frequency interaction module MSFI, two upsampling processing modules, and a convolutional layer connected in sequence; the upsampling processing module consists of a multi-scale spatial frequency interaction module MSFI and upsampling, and the downsampling processing module consists of a multi-scale spatial frequency interaction module MSFI and downsampling.

[0017] In the above solution, the spatial frequency information interaction network predicts and outputs the noise to be added for each time series according to the spatial features and frequency features, specifically including:

[0018] Fuse the spatial features and frequency features and input them into a convolutional layer to obtain a first output feature;

[0019] Input the first output feature into the downsampling processing module of the first layer to obtain a second output feature;

[0020] Input the second output feature into the downsampling processing module of the second layer to obtain a third output feature;

[0021] Input the third output feature into the multi-scale spatial frequency interaction module MSFI to obtain a fourth output feature;

[0022] Input the fourth output feature into the upsampling processing module of the first layer to obtain a fifth output feature;

[0023] Input the fifth output feature into the upsampling processing module of the second layer to obtain a sixth output feature;

[0024] Input the sixth output feature into a convolutional layer to obtain the noise to be added.

[0025] In the above solution, the multi-scale spatial frequency interaction module MSFI includes two learnable wavelet transform layers, two learnable inverse wavelet transform layers, three convolutional layers, and two spatial frequency feature interaction layers SFIB.

[0026] In the above solution, inputting the third output feature into the multi-scale spatial frequency interaction module MSFI to obtain a fourth output feature specifically includes:

[0027] Inputting the third output feature into the learnable wavelet transform layer of the first layer and the convolutional layer of the first layer respectively to obtain corresponding first and second processing results, where the first processing result includes high-frequency components and low-frequency components;

[0028] Inputting the first processing result into the convolutional layer of the second layer to obtain a first feature to be fused;

[0029] Inputting the low-frequency component into the learnable wavelet transform layer of the second layer to obtain a secondary low-frequency component and inputting it into the convolutional layer of the third layer to obtain a convolutional result, and inputting the convolutional result into the learnable wavelet transform layer of the first layer to obtain a second feature to be fused;

[0030] Inputting the first feature to be fused and the second feature to be fused into the first spatial frequency feature interaction layer SFIB to obtain an interaction feature and then inputting it into the learnable wavelet transform layer of the second layer to obtain a third feature to be fused;

[0031] Inputting the third feature to be fused and the second processing result into the second spatial frequency feature interaction layer SFIB to obtain a fourth output feature.

[0032] In the above solution, inputting the third feature to be fused and the second processing result into the second spatial frequency feature interaction layer SFIB to obtain a fourth output feature specifically includes:

[0033] Inputting the third feature to be fused into a convolutional layer to extract its corresponding spatial feature;

[0034] Inputting the second processing result into a learnable inverse wavelet transform layer to obtain its corresponding frequency feature;

[0035] Performing feature expansion on the spatial feature and the frequency feature to obtain corresponding first and second sequence features;

[0036] Based on a multi-layer perceptron MPL, performing feature mapping and position encoding on the first and second sequence features to obtain corresponding first and second encoded features;

[0037] Map the first coding feature and the second coding feature into a plurality of feature vectors respectively, and obtain the interaction calculation vector features of the first coding feature and the second coding feature with the frequency sequence feature;

[0038] Map the vector feature into a feature map;

[0039] Determine the fourth output feature of the interaction calculation according to the feature map.

[0040] The present application also proposes a colorization method system for SAR images, and the system includes: an image acquisition unit, a feature extraction unit, and an image processing unit;

[0041] The image acquisition unit is used to acquire SAR images;

[0042] The feature extraction unit is used to extract the spatial feature and the frequency feature of the input SAR image;

[0043] The image processing unit includes a spatial-frequency information interaction network, a noise addition module, and a noise removal module, and is used to perform forward noise addition and reverse noise removal on the SAR image, wherein:

[0044] Spatial-frequency information interaction network: Predict and output the noise to be added and the noise to be removed for each time series according to the spatial feature and the frequency feature;

[0045] Noise addition module: Add the noise to be added output by the spatial-frequency information interaction network to the SAR image in sequence according to the time series to obtain a pure noise image;

[0046] Noise removal module: Remove the noise from the pure noise image in sequence according to the time series of the noise to be removed output by the spatial-frequency information interaction network to obtain a color image.

[0047] The present application also proposes a readable storage medium, storing a computer program, and when the computer program is executed by a processor, the processor is caused to execute the following steps:

[0048] Acquire a SAR image, and extract the spatial feature and the frequency feature of the SAR image;

[0049] Input the spatial feature and the frequency feature into a conditional diffusion model, the conditional diffusion model includes a spatial-frequency information interaction network, and perform forward noise addition and reverse noise removal on the SAR image through the spatial-frequency information interaction network;

[0050] In the forward noise addition process, the spatial-frequency information interaction network predicts and outputs the noise to be added for each time series according to the spatial feature and the frequency feature;

[0051] Add the noise to be added to the SAR image in sequence according to the order of the time series to obtain a pure noise image;

[0052] In the reverse denoising process, the spatial frequency information interaction network predicts and outputs the noise to be removed for each time series according to the pure noise image;

[0053] Remove the noise to be removed from the pure noise image in sequence according to the order of the time series to obtain a color image.

[0054] This application also proposes a computer device, including a memory and a processor. The memory stores a computer program, and the computer program is executed by the processor to perform the following steps:

[0055] Obtain a SAR image, and extract the spatial features and frequency features of the SAR image;

[0056] Input the spatial features and frequency features into a conditional diffusion model. The conditional diffusion model includes a spatial frequency information interaction network, and perform forward noise addition and reverse denoising on the SAR image through the spatial frequency information interaction network;

[0057] In the forward noise addition process, the spatial frequency information interaction network predicts and outputs the noise to be added for each time series according to the spatial features and frequency features;

[0058] Add the noise to be added to the SAR image in sequence according to the order of the time series to obtain a pure noise image;

[0059] In the reverse denoising process, the spatial frequency information interaction network predicts and outputs the noise to be removed for each time series according to the pure noise image;

[0060] Remove the noise to be removed from the pure noise image in sequence according to the order of the time series to obtain a color image.

[0061] Adopting the embodiments of the present invention has the following beneficial effects: First, obtain the SAR image and extract the spatial features and frequency features of the SAR image; input the spatial features and frequency features into the conditional diffusion model, where the conditional diffusion model includes a spatial-frequency information interaction network, and perform forward noise addition and reverse denoising on the SAR image through the spatial-frequency information interaction network; during the forward noise addition process, the spatial-frequency information interaction network predicts and outputs the noise to be added for each time series according to the spatial features and frequency features; add the noise to be added to the SAR image in sequence according to the order of the time series to obtain a pure noise image; during the reverse denoising process, the spatial-frequency information interaction network predicts and outputs the noise to be removed for each time series according to the pure noise image; remove the noise to be removed from the pure noise image in sequence according to the order of the time series to obtain a color image. The present invention accurately extracts the spatial and frequency features of the SAR image and uses the spatial-frequency information interaction network in the conditional diffusion model to perform forward noise addition and reverse denoising processing, successfully realizing the colorization of the SAR image and improving the visual effect and practicality of the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention, and those of ordinary skill in the art can obtain other drawings without creative efforts based on these drawings.

[0063] Among them:

[0064] Figure 1 It is a schematic flowchart of a method for colorizing a SAR image in an embodiment;

[0065] Figure 2 It is a schematic structural diagram of a spatial-frequency information interaction network in an embodiment;

[0066] Figure 3 It is a schematic structural diagram of a multi-scale spatial-frequency interaction module MSFI in an embodiment;

[0067] Figure 4 It is a schematic diagram of the interactive calculation of spatial domain features and frequency domain features in an embodiment;

[0068] Figure 5 It is a schematic flowchart of the colorization of a SAR image based on the interactive diffusion of spatial domain features and frequency domain features in an embodiment. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0069] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without making creative efforts belong to the scope of protection of the present invention.

[0070] In the following description, numerous specific details are given to provide a more thorough understanding of the present invention. However, it is obvious to those skilled in the art that the present invention can be implemented without one or more of these details. In other examples, in order to avoid confusion with the present invention, some well-known technical features in the art are not described. It should be understood that the present invention can be implemented in different forms and should not be construed as limited to the embodiments presented here. On the contrary, providing these embodiments will make the disclosure thorough and complete and will fully convey the scope of the present invention to those skilled in the art.

[0071] The purpose of the terms used herein is only to describe specific embodiments and is not a limitation of the present invention. When used herein, the singular forms "a", "an" and "the" are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the terms "comprising" and / or "including", when used in this specification, determine the presence of the described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups. When used herein, the term "and / or" includes any and all combinations of the related listed items.

[0072] The colorization technology can not only significantly increase the visual information content of SAR images, enhance the detailed features of the images, and improve their intuitiveness, but also further improve the recognition accuracy of surface features on the basis of retaining the core spatial and radiometric information of the images. This application proposes a colorization method for SAR images. Through colorization processing, the key information in SAR images can be presented more intuitively, thus helping professional analysts to more accurately analyze surface information.

[0073] To thoroughly understand the present invention, detailed structures will be presented in the following description to explain the technical solutions proposed by the present invention. The optional embodiments of the present invention are described in detail as follows. However, in addition to these detailed descriptions, the present invention can also have other implementation manners.

[0074] As Figure 1 shown, in one embodiment, a colorization method for SAR images is provided. The colorization method for SAR images includes steps S101 to S106, which are described in detail as follows:

[0075] S101. Obtain the SAR image and extract the spatial features and frequency features of the SAR image;

[0076] Extracting the key spatial and frequency features from the SAR image provides the necessary information basis for subsequent processing.

[0077] In some embodiments, extracting the spatial features and frequency features of the SAR image specifically includes:

[0078] Perform a learnable wavelet transform on the SAR image to obtain the frequency features;

[0079] Perform downsampling on the SAR image to obtain the spatial features.

[0080] In some embodiments, from the SAR image Extract the frequency features , which can be used as a conditional constraint to assist in coloring during the image processing process. Among them, the global frequency information extraction process can be represented by the following formula:

[0081]

[0082] In the formula, is the learnable wavelet transform layer, is the SAR image, is the frequency feature;

[0083] Perform downsampling on the SAR image according to the following formula to obtain the spatial features :

[0084]

[0085] In the formula, is the downsampling process, is the spatial feature, is the SAR image.

[0086] Preferably, the above-mentioned spatial features , frequency features can be feature concatenated to obtain the fused feature for subsequent processing:

[0087] In the formula, is the feature fusion process, is the spatial feature, is the frequency feature, is the fused feature.

[0088] S102. Input the spatial features and frequency features into the conditional diffusion model. The conditional diffusion model includes a spatial-frequency information interaction network. Forward add noise and backward denoise the SAR image through the spatial-frequency information interaction network.

[0089] Through the spatial-frequency information interaction network, the effective utilization of SAR image features is realized, and the forward add noise and backward denoise processes of the conditional diffusion model are initiated, providing an effective technical means for subsequent colorization processing.

[0090] Such as Figure 2 shown, in some embodiments, the spatial-frequency information interaction network includes a convolutional layer, two downsampling processing modules, a multi-scale spatial-frequency interaction module MSFI, two upsampling processing modules, and a convolutional layer connected in sequence; the upsampling processing module consists of a multi-scale spatial-frequency interaction module MSFI and upsampling, and the downsampling processing module consists of a multi-scale spatial-frequency interaction module MSFI and downsampling.

[0091] S103. During the forward add noise process, the spatial-frequency information interaction network predicts and outputs the noise to be added for each time series according to the spatial features and frequency features.

[0092] The spatial-frequency information interaction network can accurately predict the noise to be added for each time series according to the input spatial and frequency features, providing precise guidance for generating a pure noise image.

[0093] In some embodiments, the spatial-frequency information interaction network predicts and outputs the noise to be added for each time series according to the spatial features and frequency features, specifically including:

[0094] Fuse the spatial features and frequency features and input them into the convolutional layer to obtain the first output feature.

[0095] Input the first output feature into the downsampling processing module of the first layer to obtain the second output feature.

[0096] Input the second output feature into the downsampling processing module of the second layer to obtain the third output feature.

[0097] Input the third output feature into the multi-scale spatial-frequency interaction module MSFI to obtain the fourth output feature.

[0098] Input the fourth output feature into the upsampling processing module of the first layer to obtain the fifth output feature.

[0099] Input the fifth output feature into the upsampling processing module of the second layer to obtain the sixth output feature.

[0100] Input the sixth output feature into the convolutional layer to obtain the noise to be added.

[0101] Such asFigure 3 As shown, in some embodiments, the multi-scale spatial frequency interaction module MSFI includes two layers of learnable wavelet transform layers, two layers of learnable inverse wavelet transform layers, three convolutional layers, and two spatial frequency feature interaction layers SFIB.

[0102] In some embodiments, inputting the third output feature into the multi-scale spatial frequency interaction module MSFI to obtain the fourth output feature specifically includes:

[0103] S301: Input the third output feature into the learnable wavelet transform layer of the first layer and the convolutional layer of the first layer respectively to obtain the corresponding first processing result and second processing result, where the first processing result contains high-frequency components and low-frequency components;

[0104] S302: Input the first processing result into the convolutional layer of the second layer to obtain the first feature to be fused;

[0105] S303: Input the low-frequency component into the learnable wavelet transform layer of the second layer to obtain the secondary low-frequency component, and input the secondary low-frequency component into the convolutional layer of the third layer to obtain the convolutional result. Then input the convolutional result into the learnable wavelet transform layer of the first layer to obtain the second feature to be fused;

[0106] S304: Input the first feature to be fused and the second feature to be fused into the first spatial frequency feature interaction layer SFIB to obtain the interaction feature, and then input the interaction feature into the learnable wavelet transform layer of the second layer to obtain the third feature to be fused;

[0107] S305: Input the third feature to be fused and the second processing result into the second spatial frequency feature interaction layer SFIB to obtain the fourth output feature.

[0108] Specifically, Figure 2 it includes five multi-scale spatial frequency interaction modules MSFI, and the processing flow of each multi-scale spatial frequency interaction module MSFI for the input feature is the same as that of steps S301 - S305.

[0109] Such as Figure 3 As shown, in some embodiments, inputting the given feature into the multi-scale spatial frequency interaction module MSFI to obtain the corresponding output feature specifically includes:

[0110] Given input feature , the first-level feature extraction result can be expressed as:

[0111]

[0112]

[0113] In the formula, is the convolution process, It is a learnable wavelet transform layer, which is the feature result after the first-level feature extraction, For the variable after decomposition;

[0114] Perform second-level feature extraction on the low-frequency component , and the second-level feature extraction result can be expressed as:

[0115]

[0116] wherein, For the variable after decomposition.

[0117] The output result of feature interaction can be expressed as:

[0118]

[0119] wherein, is the final output result, is the convolution operation, is a learnable wavelet transform layer, is a learnable inverse wavelet transform layer, is the spatial frequency feature interaction layer.

[0120] In some embodiments, inputting the third feature to be fused and the second processing result into the second spatial frequency feature interaction layer SFIB to obtain the fourth output feature specifically includes:

[0121] S301. Input the third feature to be fused into the convolutional layer to extract its corresponding spatial feature;

[0122] S302. Input the second processing result into the learnable inverse wavelet transform layer to obtain its corresponding frequency feature;

[0123] S303. Perform feature expansion on the spatial feature and the frequency feature to obtain the corresponding first sequence feature and second sequence feature;

[0124] S304. Based on the multi-layer perceptron MPL, perform feature mapping and position encoding on the first sequence feature and the second sequence feature to obtain the corresponding first encoded feature and second encoded feature;

[0125] S305. Map the first encoded feature and the second encoded feature into a plurality of feature vectors respectively and obtain the vector feature of the interactive calculation of the first encoded feature and the second encoded feature and the frequency sequence feature;

[0126] S306. Map the vector feature into a feature map;

[0127] S307. Determine the fourth output feature of the interaction calculation based on the feature map.

[0128] Specifically, Figure 3 It includes two spatial frequency feature interaction layers SFIB, and the processing flow of each spatial frequency feature interaction layer SFIB for the input feature is the same as that of steps S301 - S307.

[0129] As Figure 4 shown, in some embodiments, the spatial domain feature and the frequency feature are input into the spatial frequency feature interaction layer SFIB to obtain the corresponding output features, specifically including:

[0130] The spatial feature extracted by the convolutional layer and the frequency feature extraction result obtained by the learnable inverse wavelet transform layer are used as the input features of the spatial frequency feature interaction layer SFIB, and the above features are feature-expanded (Img2Seq) to obtain sequence features for subsequent interaction calculation;

[0131] Based on the multi-layer perceptron MPL, the sequence features and are feature-mapped, and position encoding (Position embedding PE) is performed on the sequence feature information to obtain the encoded features and ;

[0132] Taking the sequence feature after position encoding as an example, it is mapped into three feature vectors . Then the interaction calculation process with the frequency sequence feature can be expressed as:

[0133]

[0134] The vector feature after interaction calculation can be obtained by learning the weight calculation;

[0135] After calculating the interaction result of the frequency sequence feature , perform a sequence-to-feature map mapping (Seq2Img) on the sequence feature after spatial-frequency domain interaction to obtain the interaction feature map, and perform element-wise addition to obtain the final output feature of the interaction calculation.

[0136] S104. Add the noise to be added to the SAR image in sequence according to the time series to obtain a pure noise image;

[0137] By gradually adding the predicted noise, a pure noise image corresponding to the SAR image was successfully generated, providing a basis for subsequent denoising processing.

[0138] Specifically, the original data was transformed into pure noise through a series of steps. In each step, a small amount of controllable Gaussian noise was added to the image, making it gradually distorted and approaching random noise. Among them, the forward noise addition diffusion process can be expressed by the following formula:

[0139]

[0140] In the formula, is the image state at the current time step ; is the image state at the previous time step , which provides a starting point for adding noise at the step; is a variable controlling the noise variance; is the identity matrix. When it is multiplied by , it can determine the variance of the noise evenly added to each element.

[0141] S105. In the reverse denoising process, the spatial frequency information interaction network predicts and outputs the noise to be removed for each time series based on the pure noise image;

[0142] The spatial frequency information interaction network also performs excellently in the reverse denoising stage, being able to accurately predict the noise to be removed for each time series, providing strong support for restoring the color image.

[0143] Specifically, by iteratively reducing the noise level, the original data is reconstructed from the noise, and the SAR image is colored. The reverse denoising process can be expressed by the following formula:

[0144]

[0145] In the formula, is a function used to predict the distribution mean; is a function used to determine the distribution variance.

[0146] S106. Remove the noise to be removed from the pure noise image in the order of the time series to obtain the color image.

[0147] By gradually removing the predicted noise, the color image can be successfully restored. Compared with the original SAR image in terms of visual effect, the restored color image has richer color information and higher clarity.

[0148] In summary, asFigure 5 As shown, the dashed box represents the conditional diffusion model, which includes the forward noise-adding diffusion process and the reverse denoising process. The solid box represents the spatial-frequency information interaction network, which is closely connected to the conditional diffusion model. This figure describes the entire processing flow from the input SAR image to the output of the final colorized SAR image:

[0149] (1) Prior condition acquisition:

[0150] From the SAR image Frequency features are extracted through learnable wavelet transform, and the original input SAR image is downsampled to obtain spatial features ;

[0151] (2) Conditional diffusion model:

[0152] For the above spatial features and frequency features Feature splicing is performed to obtain fused features .

[0153] The fused features are input into the conditional diffusion model for the forward noise-adding diffusion process and the reverse denoising process:

[0154] Forward noise-adding diffusion process: By gradually adding controllable Gaussian noise, the original SAR image is converted into a pure noise image.

[0155] Reverse denoising process: Through an iterative optimization algorithm, the noise is gradually removed, and the original data is reconstructed from the pure noise image, and finally a colorized SAR image is obtained.

[0156] (3) Spatial-frequency information interaction network:

[0157] The spatial-frequency information interaction network runs through the entire conditional diffusion model, realizing the interaction between spatial and frequency information, and making full use of the characteristics of local spatial information and global frequency information, thus significantly improving the coloring performance of SAR images.

[0158] Finally, through the processing of the conditional diffusion model, the original SAR image is successfully converted into a colorized image. These colorized images have higher visual clarity and information content, which helps professional analysts to more accurately analyze surface information. On the Figure 1 right side, the output of the colorized SAR image is shown.

[0159] This application also proposes a colorization method system for SAR images. The system includes: an image acquisition unit, a feature extraction unit, and an image processing unit;

[0160] An image acquisition unit for acquiring SAR images;

[0161] A feature extraction unit for extracting the spatial features and frequency features of the input SAR image;

[0162] An image processing unit, including a spatial-frequency information interaction network, a noise addition module, and a noise removal module, for forward noise addition and reverse noise removal of the SAR image, where:

[0163] The spatial-frequency information interaction network: predicts and outputs the noise to be added and the noise to be removed for each time series according to the spatial features and frequency features;

[0164] The noise addition module: sequentially adds the noise to be added output by the spatial-frequency information interaction network to the SAR image in the order of time series to obtain a pure noise image;

[0165] The noise removal module: removes the noise from the pure noise image in the order of time series according to the noise to be removed output by the spatial-frequency information interaction network to obtain a color image.

[0166] This application also proposes a readable storage medium storing a computer program, which when executed by a processor causes the processor to perform the following steps:

[0167] Acquire a SAR image and extract the spatial features and frequency features of the SAR image;

[0168] Input the spatial features and frequency features into a conditional diffusion model, where the conditional diffusion model includes a spatial-frequency information interaction network, and perform forward noise addition and reverse noise removal on the SAR image through the spatial-frequency information interaction network;

[0169] During the forward noise addition process, the spatial-frequency information interaction network predicts and outputs the noise to be added for each time series according to the spatial features and frequency features;

[0170] Sequentially add the noise to be added to the SAR image in the order of time series to obtain a pure noise image;

[0171] During the reverse noise removal process, the spatial-frequency information interaction network predicts and outputs the noise to be removed for each time series according to the pure noise image;

[0172] Remove the noise to be removed from the pure noise image in the order of time series to obtain a color image.

[0173] This application also proposes a computer device, including a memory and a processor, where the memory stores a computer program, and the computer program is executed by the processor as follows:

[0174] Acquire a SAR image and extract the spatial features and frequency features of the SAR image;

[0175] Input the spatial features and frequency features into the conditional diffusion model. The conditional diffusion model includes a spatial-frequency information interaction network, and perform forward noise addition and reverse denoising on the SAR image through the spatial-frequency information interaction network;

[0176] During the forward noise addition process, the spatial-frequency information interaction network predicts and outputs the noise to be added for each time series according to the spatial features and frequency features;

[0177] Add the noise to be added to the SAR image in sequence according to the time series to obtain a pure noise image;

[0178] During the reverse denoising process, the spatial-frequency information interaction network predicts and outputs the noise to be removed for each time series according to the pure noise image;

[0179] Remove the noise to be removed from the pure noise image in sequence according to the time series to obtain a color image.

[0180] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope recorded in this specification.

[0181] The above-described embodiments only represent several implementation manners of the present application. The description is relatively specific and detailed, but it should not be construed as a limitation on the patent scope of the present application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several deformations and improvements can still be made, and these all belong to the protection scope of the present application. The above-disclosed is only the preferred embodiment of the present invention, and of course, it cannot be used to limit the scope of rights of the present invention. Therefore, the equivalent changes made according to the claims of the present invention still fall within the scope covered by the present invention.

Claims

1. A method for colorizing a SAR image, characterized in that: The method comprises: Acquire a SAR image, and extract spatial features and frequency features of the SAR image; Inputting the spatial features and frequency features into a conditional diffusion model, wherein the conditional diffusion model includes a spatial frequency information interaction network, and performing forward denoising and reverse denoising on the SAR image through the spatial frequency information interaction network; In the forward noise addition process, the spatial-frequency information interaction network predicts and outputs the noise to be added of each time series according to the spatial features and the frequency features, specifically including: fusing the spatial features and the frequency features, and inputting them into the convolution layer to obtain the first output feature; Inputting the first output feature into a downsampling processing module of the first layer to obtain a second output feature; Inputting the second output feature into the downsampling processing module of the second layer to obtain a third output feature; Inputting the third output feature into a multi-scale spatial frequency interaction module MSFI to obtain a fourth output feature specifically includes: Inputting the third output feature into the learnable wavelet transform layer of the first layer and the convolution layer of the first layer respectively, obtaining corresponding first processing results and second processing results, wherein the first processing result includes a high-frequency component and a low-frequency component; Inputting the first processing result into the convolution layer of the second layer to obtain the first feature to be fused; Input the low-frequency component into the learnable wavelet transform layer of the second layer to obtain a secondary low-frequency component, input the secondary low-frequency component into the convolution layer of the third layer to obtain a convolution result, and input the convolution result into the learnable inverse wavelet transform layer of the first layer to obtain a second feature to be fused; Input the first feature to be fused and the second feature to be fused into the first spatial frequency feature interaction layer SFIB to obtain the interaction feature, and then input the interaction feature into the learnable inverse wavelet transform layer of the second layer to obtain the third feature to be fused; Input the third feature to be fused and the second processing result into the second spatial frequency feature interaction layer SFIB to obtain a fourth output feature; Inputting the fourth output feature into the upsampling processing module of the first layer to obtain a fifth output feature; Inputting the fifth output feature into the upsampling processing module of the second layer to obtain a sixth output feature; Inputting the sixth output feature into the convolution layer to obtain noise to be added; Adding the noise to be added to the SAR image in sequence according to the order of the time series to obtain a pure noise image; In the reverse denoising process, the spatial frequency information interaction network predicts and outputs the noise to be removed of each of the time series according to the pure noise image; The noise to be removed is removed from the pure noise image in the order of the time series to obtain a color image.

2. The SAR image colorization method according to claim 1, characterized in that: The extracting of the spatial features and frequency features of the SAR image specifically includes: Performing a learnable wavelet transform on the SAR image to obtain frequency features; The SAR image is downsampled to obtain spatial features.

3. The SAR image colorization method according to claim 1, characterized in that: The spatial frequency information interaction network includes a convolution layer, two downsampling processing modules, a multi-scale spatial frequency interaction module MSFI, two upsampling processing modules and a convolution layer connected in sequence; the upsampling processing module is composed of a multi-scale spatial frequency interaction module MSFI and upsampling, and the downsampling processing module is composed of a multi-scale spatial frequency interaction module MSFI and downsampling.

4. The method for colorizing a SAR image according to claim 1, characterized in that: The multi-scale spatial frequency interaction module MSFI includes two learnable wavelet transform layers, two learnable inverse wavelet transform layers, three convolutional layers and two spatial frequency feature interaction layers SFIB.

5. The method for colorizing a SAR image according to claim 1, characterized in that: Inputting the third feature to be fused and the second processing result into the second spatial frequency feature interaction layer SFIB to obtain the fourth output feature specifically includes: Input the third feature to be fused into the convolution layer to extract the corresponding spatial feature; Inputting the second processing result into a learnable inverse wavelet transform layer to obtain a frequency feature corresponding to the second processing result; Perform feature expansion on the spatial features and the frequency features to obtain corresponding first sequence features and second sequence features; Perform feature mapping and position encoding on the first sequence features and the second sequence features based on a multi-layer perceptron MPL to obtain corresponding first encoding features and second encoding features; Mapping the first coding feature and the second coding feature into a plurality of feature vectors respectively and obtaining vector features calculated by interaction of the first coding feature and the second coding feature with the frequency sequence feature; Mapping the vector features into a feature map; A fourth output feature of the interactive calculation is determined according to the feature graph.

6. A SAR image colorization method system, characterized in that: The system comprises: an image acquisition unit, a feature extraction unit and an image processing unit; The image acquisition unit is used to acquire the SAR image; The feature extraction unit is used to extract the spatial features and frequency features of the input SAR image; The image processing unit includes a space-frequency information interaction network, a denoising module and a denoising module, which are used to perform forward denoising and reverse denoising on the SAR image, wherein: Space-frequency information interaction network: predicting and outputting noise to be added and noise to be removed of each time series according to the space feature and the frequency feature, wherein the outputting noise to be added of each time series comprises: fusing the space feature and the frequency feature, and inputting the result into a convolution layer to obtain a first output feature; Inputting the first output feature into a downsampling processing module of the first layer to obtain a second output feature; Inputting the second output feature into the downsampling processing module of the second layer to obtain a third output feature; Inputting the third output feature into a multi-scale spatial frequency interaction module MSFI to obtain a fourth output feature specifically includes: Inputting the third output feature into the learnable wavelet transform layer of the first layer and the convolution layer of the first layer respectively, obtaining corresponding first processing results and second processing results, wherein the first processing result includes a high-frequency component and a low-frequency component; Inputting the first processing result into the convolution layer of the second layer to obtain the first feature to be fused; Input the low-frequency component into the learnable wavelet transform layer of the second layer to obtain a secondary low-frequency component, input the secondary low-frequency component into the convolution layer of the third layer to obtain a convolution result, and input the convolution result into the learnable inverse wavelet transform layer of the first layer to obtain a second feature to be fused; Input the first feature to be fused and the second feature to be fused into the first spatial frequency feature interaction layer SFIB to obtain the interaction feature, and then input the interaction feature into the learnable inverse wavelet transform layer of the second layer to obtain the third feature to be fused; Input the third feature to be fused and the second processing result into the second spatial frequency feature interaction layer SFIB to obtain a fourth output feature; Inputting the fourth output feature into the upsampling processing module of the first layer to obtain a fifth output feature; Inputting the fifth output feature into the upsampling processing module of the second layer to obtain a sixth output feature; Inputting the sixth output feature into the convolution layer to obtain noise to be added; Noise adding module: according to the noise to be added output by the space-frequency information interaction network, it is added to the SAR image in the order of time series to obtain a pure noise image; Denoising module: According to the noise to be removed output by the space-frequency information interaction network, the noise is removed from the pure noise image in the order of the time series to obtain a color image.

7. A readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor is caused to execute the steps of the method according to any one of claims 1 to 5.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 5.

Images