Synthetic aperture radar image to rgb conversion algorithm based on generative adversarial network

Abstract

The application discloses an algorithm for converting synthetic aperture radar images into RGB based on an adversarial generative network, acquires a SAR image and a visible light image dataset; the visible light image is screened; a cloud detection model is trained using the visible light image; it is judged whether the visible light image contains clouds; a dataset is made according to the screened SAR and visible light images; the new data model is trained; it is judged whether the model converges, the application can learn the features of the corresponding visible light image of the SAR image based on the adversarial generative network GAN, can generate an image extremely similar to the normal visible light image on the basis of retaining most of the features of the SAR image, realizes the conversion of the SAR image into the visible light image, can effectively reconstruct the information of the SAR image, expands the application field of the SAR image, makes the SAR image act on more fields, and also makes up for the pain point of the lack of remote sensing datasets in the remote sensing field, and improves the use degree of the SAR image.

Description

Technical Field

[0001] This invention relates to the field of remote sensing imaging technology, specifically to an algorithm for converting synthetic aperture radar images to RGB based on generative adversarial networks. Background Technology

[0002] Synthetic Aperture Radar (SAR) is a type of microwave remote sensing imaging radar. SAR images are formed by superimposing coherent signals from the echoes scattered by ground objects. Compared with optical and infrared radars, its imaging is less restricted by weather conditions and has the characteristics of all-weather, all-climate, and multi-angle operation. Therefore, it is widely used in military reconnaissance, surface mapping, marine resource exploration, and environmental monitoring. Because SAR images and visible light images use different sensors, they also have significant differences. Visible light images usually contain grayscale information from multiple bands to facilitate target identification and classification, while SAR images only record the echo information of one band, recorded in binary complex form. Based on the complex data of each pixel, the corresponding amplitude and phase information can be extracted. The amplitude information usually corresponds to the backscattering intensity of radar waves by ground targets and is closely related to the target medium, water content, and roughness. However, since SAR images are created by transmitting signals to objects via antennas and then receiving those signals, and this transmission and reception process is divided into horizontal transmission and reception and vertical transmission and reception, SAR images will contain a large amount of noise information after imaging. This results in relatively low resolution and low signal-to-noise ratio of SAR images, which greatly limits the application areas of the images. Summary of the Invention

[0003] This invention provides an algorithm for converting synthetic aperture radar (SAR) images to RGB based on generative adversarial networks. This algorithm can effectively solve the problem mentioned in the background art that SAR images are generated by transmitting signals to objects through an antenna and then receiving signals. This signal transmission and reception process is divided into horizontal transmission and reception and vertical transmission and reception. As a result, SAR images will contain a large amount of noise information after imaging, leading to relatively low resolution and low signal-to-noise ratio of SAR images, which greatly limits the application of the images.

[0004] To achieve the above objectives, the present invention provides the following technical solution: an algorithm for converting synthetic aperture radar images to RGB based on generative adversarial networks, the algorithm specifically including the following steps: S1. Obtain SAR and visible light image datasets; S2. Filter the visible light images; S3. Train a cloud detection model using visible light images; S4. Determine whether the visible light image does not contain clouds; S5. Create a dataset based on the filtered SAR and visible light images; S6. Train using the new data model; S7. Determine whether the model has converged; S8. Use the model to perform visible light image conversion on the input SAR image.

[0005] According to the above technical solution, in S1, acquiring SAR image and visible light image datasets mainly refers to acquiring SAR images of Sentinel-1 and visible light images of Sentinel-2 through the Sentinel dataset. In the specific acquisition process, the dataset is downloaded by connecting to the open-source dataset of the MultiEarth 2022 competition, and the Sentinel-2 dataset is fused into RGB visible light images through channel fusion.

[0006] According to the above technical solution, in S2, the screening of visible light images mainly refers to dividing a small dataset, calculating the overall error value of the image using the mean square error formula, and initially dividing the image into cloud-covered images and cloudless images, and then making the final screening and division by manual observation.

[0007] According to the above technical solution, in S3, training the cloud detection model using visible light images specifically refers to using the VGG11 model in torch to train the classification model on the segmented data to obtain the final classification model.

[0008] According to the above technical solution, in step S4, determining whether a visible light image does not contain clouds is mainly used to determine whether a visible light image contains clouds. When the visible light does not contain clouds, the trained model is used to classify the image, exclude images in the visible light dataset that contain clouds, and delete the corresponding SAR images. When the visible light does contain clouds, the process proceeds to the next step.

[0009] According to the above technical solution, in step S5, when the visible light image contains clouds, the dataset is mainly created based on the filtered SAR and visible light images.

[0010] According to the above technical solution, in S5, during the specific process of creating the dataset, the 16-bit depth Sentinel-1 SAR image is stretched into an 8-bit depth visible light image using the Linear 2% stretching algorithm, and then matched with the corresponding Sentinel-2 visible light dataset to create a new dataset.

[0011] According to the above technical solution, in S6, training with a new data model mainly refers to iteratively training the prepared dataset into a Pix2Pix model with a GAN backbone network until the model converges.

[0012] According to the above technical solution, in step S7, determining whether the model has converged mainly involves determining whether the model in step S6 has converged. If the model has not converged, the process returns to step S6 for repeated training until the model converges.

[0013] According to the above technical solution, in S8, the visible light image conversion of the input SAR image using the model mainly refers to inputting the 8-bit depth Sentinel-1 SAR image into the network model after the model converges, and then outputting the RGB visible light image.

[0014] Compared with the prior art, the beneficial effects of the present invention are: the present invention has a scientific and reasonable structure and is safe and convenient to use. This invention utilizes Generative Adversarial Networks (GANs) to learn the features of visible light images corresponding to SAR images. While retaining most of the features of SAR images, it can generate an image that is extremely similar to a normal visible light image, thus realizing the conversion of SAR images into visible light images. At the same time, it can effectively reconstruct information from SAR images, expanding the application fields of SAR images to more areas. It also makes up for the lack of remote sensing datasets in the field of remote sensing and improves the usability of SAR images. Attached Figure Description

[0015] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.

[0016] In the attached diagram: Figure 1 This is a flowchart of the algorithm of this invention; Figure 2 This is a flowchart of the algorithm steps of this invention; Figure 3 This is a diagram illustrating the output results of the present invention. Detailed Implementation

[0017] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0018] Example 1: As Figure 1-3 As shown, this invention provides a technical solution for converting synthetic aperture radar images to RGB based on generative adversarial networks. The algorithm specifically includes the following steps: S1. Obtain SAR and visible light image datasets; S2. Filter the visible light images; S3. Train a cloud detection model using visible light images; S4. Determine whether the visible light image does not contain clouds; S5. Create a dataset based on the filtered SAR and visible light images; S6. Train using the new data model; S7. Determine whether the model has converged; S8. Use the model to perform visible light image conversion on the input SAR image.

[0019] Based on the above technical solution, in S1, acquiring SAR image and visible light image datasets mainly refers to acquiring SAR images of Sentinel-1 and visible light images of Sentinel-2 through the Sentinel dataset. In the specific acquisition process, the dataset is downloaded by connecting to the open-source dataset of the MultiEarth 2022 competition, and the Sentinel-2 dataset is fused into RGB visible light images.

[0020] Based on the above technical solution, in S2, the screening of visible light images mainly refers to dividing a small dataset, calculating the overall error value of the image using the mean square error formula, and initially dividing the image into cloud-covered images and cloudless images, and then making the final screening and division through manual observation.

[0021] Based on the above technical solution, in S3, training the cloud detection model using visible light images specifically refers to using the VGG11 model in torch to train the classification model on the segmented data, and obtain the final classification model.

[0022] Based on the above technical solution, in S4, determining whether a visible light image does not contain clouds is mainly used to determine whether a visible light image does not contain clouds. When the visible light does not contain clouds, the pre-trained model is used to classify the image, exclude images containing clouds in the visible light dataset, and delete the corresponding SAR images.

[0023] Example 2: Figure 1-3 As shown, this invention provides a technical solution for converting synthetic aperture radar images to RGB based on generative adversarial networks. The algorithm specifically includes the following steps: S1. Obtain SAR and visible light image datasets; S2. Filter the visible light images; S3. Train a cloud detection model using visible light images; S4. Determine whether the visible light image does not contain clouds; S5. Create a dataset based on the filtered SAR and visible light images; S6. Train using the new data model; S7. Determine whether the model has converged; S8. Use the model to perform visible light image conversion on the input SAR image.

[0024] Based on the above technical solution, in S1, acquiring SAR image and visible light image datasets mainly refers to acquiring SAR images of Sentinel-1 and visible light images of Sentinel-2 through the Sentinel dataset. In the specific acquisition process, the dataset is downloaded by connecting to the open-source dataset of the MultiEarth 2022 competition, and the Sentinel-2 dataset is fused into RGB visible light images.

[0025] Based on the above technical solution, in S2, the screening of visible light images mainly refers to dividing a small dataset, calculating the overall error value of the image using the mean square error formula, and initially dividing the image into cloud-covered images and cloudless images, and then making the final screening and division through manual observation.

[0026] Based on the above technical solution, in S3, training the cloud detection model using visible light images specifically refers to using the VGG11 model in torch to train the classification model on the segmented data, and obtain the final classification model.

[0027] Based on the above technical solution, in S4, determining whether the visible light image does not contain clouds is mainly used to determine the visible light image. If the visible light image contains clouds, then proceed to the next step.

[0028] Based on the above technical solution, in S5, when the visible light image contains clouds, the dataset is mainly created based on the filtered SAR and visible light images.

[0029] Based on the above technical solution, in S5, during the specific process of creating the dataset, the 16-bit depth Sentinel-1 SAR image is stretched into an 8-bit depth visible light image using the Linear 2% stretching algorithm, and then matched with the corresponding Sentinel-2 visible light dataset to create a new dataset.

[0030] Based on the above technical solution, in S6, training with the new data model mainly refers to iteratively training the prepared dataset into the Pix2Pix model with a GAN backbone network until the model converges.

[0031] Based on the above technical solution, in S7, determining whether the model has converged mainly involves judging the model in step S6. If the model has not converged, return to step S6 for repeated training until the model converges.

[0032] Example 3: As Figure 1-3As shown, this invention provides a technical solution for converting synthetic aperture radar images to RGB based on generative adversarial networks. The algorithm specifically includes the following steps: S1. Obtain SAR and visible light image datasets; S2. Filter the visible light images; S3. Train a cloud detection model using visible light images; S4. Determine whether the visible light image does not contain clouds; S5. Create a dataset based on the filtered SAR and visible light images; S6. Train using the new data model; S7. Determine whether the model has converged; S8. Use the model to perform visible light image conversion on the input SAR image.

[0033] Based on the above technical solution, in S1, acquiring SAR image and visible light image datasets mainly refers to acquiring SAR images of Sentinel-1 and visible light images of Sentinel-2 through the Sentinel dataset. In the specific acquisition process, the dataset is downloaded by connecting to the open-source dataset of the MultiEarth 2022 competition, and the Sentinel-2 dataset is fused into RGB visible light images.

[0034] Based on the above technical solution, in S2, the screening of visible light images mainly refers to dividing a small dataset, calculating the overall error value of the image using the mean square error formula, and initially dividing the image into cloud-covered images and cloudless images, and then making the final screening and division through manual observation.

[0035] Based on the above technical solution, in S3, training the cloud detection model using visible light images specifically refers to using the VGG11 model in torch to train the classification model on the segmented data, and obtain the final classification model.

[0036] Based on the above technical solution, in S4, determining whether the visible light image does not contain clouds is mainly used to determine the visible light image. If the visible light image contains clouds, then proceed to the next step.

[0037] Based on the above technical solution, in S5, when the visible light image contains clouds, the dataset is mainly created based on the filtered SAR and visible light images.

[0038] Based on the above technical solution, in S5, during the specific process of creating the dataset, the 16-bit depth Sentinel-1 SAR image is stretched into an 8-bit depth visible light image using the Linear 2% stretching algorithm, and then matched with the corresponding Sentinel-2 visible light dataset to create a new dataset.

[0039] Based on the above technical solution, in S6, training with the new data model mainly refers to iteratively training the prepared dataset into the Pix2Pix model with a GAN backbone network until the model converges.

[0040] Based on the above technical solution, in S7, determining whether the model has converged mainly involves judging the model in step S6.

[0041] Based on the above technical solution, in S8, the visible light image conversion of the input SAR image is performed using the model. This mainly refers to inputting the 8-bit depth Sentinel-1 SAR image into the network model after the model converges, and then outputting the RGB visible light image.

[0042] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An algorithm for converting synthetic aperture radar images to RGB based on generative adversarial networks, characterized by: The algorithm specifically includes the following steps: S1. Obtain SAR and visible light image datasets; S2. Filter the visible light images; S3. Train a cloud detection model using visible light images; S4. Determine whether the visible light image does not contain clouds; S5. Create a dataset based on the filtered SAR and visible light images; S6. Train using the new data model; S7. Determine whether the model has converged; S8. Use the model to perform visible light image conversion on the input SAR image; In S1, acquiring SAR image and visible light image datasets refers to acquiring SAR images of Sentinel-1 and visible light images of Sentinel-2 through the Sentinel dataset. In the specific acquisition process, the dataset is downloaded through the open-source dataset of the MultiEarth 2022 competition, and the Sentinel-2 dataset is fused into RGB visible light images. In S2, the visible light image is screened by dividing a small dataset, calculating the overall error value of the image using the mean square error formula, and initially dividing the image into cloud-covered and cloudless images, and then making the final screening and division by manual observation. In S3, training the cloud detection model using visible light images specifically refers to using the VGG11 model in torch to train a classification model on the segmented data to obtain the final classification model. In step S4, it is determined whether the visible light image contains clouds. If the visible light image does not contain clouds, the pre-trained model is used to classify the image, excluding images containing clouds in the visible light dataset and deleting the corresponding SAR images. If the visible light image contains clouds, the process proceeds to the next step.

2. The algorithm for converting synthetic aperture radar images to RGB based on generative adversarial networks according to claim 1, characterized in that: In step S5, when the visible light image contains clouds, a dataset is created based on the filtered SAR and visible light images.

3. The algorithm for converting synthetic aperture radar images to RGB based on generative adversarial networks according to claim 2, characterized in that: In S5, during the specific process of creating the dataset, the 16-bit depth Sentinel-1 SAR image is stretched into an 8-bit depth visible light image using the Linear2% stretching algorithm. Then, it is matched with the corresponding Sentinel-2 visible light dataset to create a new dataset.

4. The algorithm for converting synthetic aperture radar images to RGB based on generative adversarial networks according to claim 1, characterized in that: In step S6, training with the new data model means feeding the prepared dataset into the Pix2Pix model with a GAN backbone for iterative training until the model converges.

5. The algorithm for converting synthetic aperture radar images to RGB based on generative adversarial networks according to claim 1, characterized in that: In step S7, the model is judged to determine whether it has converged. If the model has not converged, the process returns to step S6 to repeat the training until the model converges.

6. The algorithm for converting synthetic aperture radar images to RGB based on generative adversarial networks according to claim 1, characterized in that: In S8, the model is used to perform visible light image conversion on the input SAR image. This means that after the model converges, an 8-bit Sentinel-1 SAR image is input into the network model, and then an RGB visible light image is output.

Images