Generative confrontation-based unsupervised domain adaptive remote sensing target recognition method and device

Abstract

The invention discloses an unsupervised domain adaptive remote sensing target recognition method and device based on generative adversarial, and the method comprises the steps: building a target detection data set, and taking the target detection data set as the image data of a target domain; acquiring a remote sensing image data set as image data of a source domain; zooming and / or cutting the source domain image and / or the target domain image, and aligning the feature space of the source domain image and the feature space of the target domain image; migrating the source domain image to a target domain by adopting an unsupervised module to obtain a first migrated image; converting the first migration image from an original color space to a YCbCr color space, and converting the corresponding source domain image from the original color space to the YCbCr color space; and performing color space channel fusion on the first migration image and the corresponding source domain image to obtain a second migration image, and taking the second migration image as a domain adaptive image. By adopting the technical scheme, the detail and texture information of the source domain image can be fully reserved during domain migration, so that the training performance of the model is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing target detection, and in particular to a method and device for remote sensing target recognition based on generative adversarial unsupervised domain adaptation. Background Art

[0002] With the rapid development of deep learning technology, it has demonstrated excellent performance in the field of remote sensing target detection. However, the domain shift problem causes the detection model that performs well on the training set to perform poorly in actual applications. The domain shift problem refers to the distribution difference between the training dataset and the test dataset (or data in actual applications). This difference may cause the model to perform well during training, but the performance may drop significantly in the new environment.

[0003] In order to overcome the domain shift problem, the existing technology provides a domain migration technology solution for migrating the trained model from the source domain to the target domain, so that the model can be used for image recognition in the target domain. However, in actual application, problems with domain migration were found. On the one hand, domain migration is directly aimed at model application. Migrating the model from the source domain to the target domain is very time-consuming and computationally expensive. On the other hand, the recognition effect of the model after domain migration is not ideal. Summary of the invention

[0004] Purpose of the invention: The present invention provides a method and device for unsupervised domain adaptation remote sensing target recognition based on generative adversarial methods, aiming to solve the technical problems in the prior art that domain migration for model application consumes time and computing power, and the recognition effect of the model after domain migration is not ideal.

[0005] Technical solution: The present invention provides an unsupervised domain adaptation remote sensing target recognition method based on generative adversarial network, comprising: establishing a target detection data set as image data of the target domain; obtaining a remote sensing image data set as image data of the source domain; the images of the target detection data set include the original recognition targets of the remote sensing target recognition model; the images of the remote sensing image data set include the newly recognized targets; performing scaling and / or cropping on the source domain image and / or the target domain image to align the feature spaces of the source domain image and the target domain image; using an unsupervised module to migrate the source domain image to the target domain to obtain a first migration image; the unsupervised module is constructed based on a generative adversarial network; converting the first migration image from the original color space to the YCbCr color space, converting the corresponding source domain image from the original color space to the YCbCr color space, performing color space channel fusion on the first migration image and the corresponding source domain image to obtain a second migration image, and using the second migration image as a domain adaptation image; using the remote sensing target recognition model to perform target recognition on the domain adaptation image to obtain a newly recognized target.

[0006] Specifically, target images with the same size are obtained based on the same data source, and the target images are divided according to the size of the original recognized targets. Target images with the same or similar sizes of the original recognized targets are divided into the same set as the corresponding target detection data set.

[0007] Specifically, according to the size of the original identified target, the original identified target is divided into small targets, medium targets and large targets.

[0008] Specifically, RGB channel data of images in the remote sensing image dataset are generated, and the images in the remote sensing image dataset are cropped to the same size as the images in the target detection dataset.

[0009] Specifically, the resolutions of the source domain image and the target domain image are compared; if the resolution of the source domain image is higher than that of the target domain image, the target domain image is enlarged and cropped, and the resolution of the cropped target domain image is the same as that of the source domain image; if the resolution of the source domain image is lower than that of the target domain image, the target domain image is reduced and the source domain image is cropped, and the resolution of the cropped target domain image is the same as that of the source domain image.

[0010] Specifically, the unsupervised module is constructed based on a cyclic generative adversarial network.

[0011] Specifically, the Cb channel and the Cr channel of the first migration image are fused with the Y channel of the source domain image to obtain the second migration image.

[0012] Specifically, the remote sensing target recognition model is any of the following models: a single-stage detection model, a recognition model based on deformable convolution and Transformer, a two-stage recognition model based on region proposal network, a fully convolutional one-stage recognition model, and a recognition model based on generative adversarial network and color feature recognition.

[0013] Specifically, peak signal-to-noise ratio and structural similarity index are used to evaluate the image quality of domain adapted images.

[0014] The present invention also provides an unsupervised domain adaptation remote sensing target recognition device based on generative adversarial, comprising: a data set establishment unit, a feature space alignment unit, a migration unit and a recognition unit, wherein: the data set establishment unit is used to establish a target detection data set as image data of the target domain; obtain a remote sensing image data set as image data of the source domain; the image of the target detection data set includes the original recognition target of the remote sensing target recognition model; the image of the remote sensing image data set includes the new recognition target; the feature space alignment unit is used to perform scaling and / or cropping on the source domain image and / or the target domain image, and convert the source domain image and the target domain image into a new image. The feature space of the image is aligned; the migration unit is used to migrate the source domain image to the target domain using an unsupervised module to obtain a first migration image; the unsupervised module is constructed based on a generative adversarial network; the first migration image is converted from the original color space to the YCbCr color space, the corresponding source domain image is converted from the original color space to the YCbCr color space, the first migration image and the corresponding source domain image are fused in color space channels to obtain a second migration image, and the second migration image is used as a domain adaptation image; the recognition unit is used to use a remote sensing target recognition model to perform target recognition on the domain adaptation image to obtain a new recognized target.

[0015] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages: domain adaptation of remote sensing images significantly reduces the time and computing power cost required for migration, and at the same time can fully retain the details and texture information of the source domain image during domain migration, thereby improving the training performance of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A schematic diagram of the steps of the unsupervised domain adaptation remote sensing target recognition method based on generative adversarial technology provided by the present invention; Figure 2 A schematic diagram of the identification target provided by the present invention; Figure 3 A schematic diagram showing the comparison of images before and after style migration; Figure 4 Schematic diagram of recognition results before and after style transfer; Figure 5 A diagram quantifying the recognition effect of the domain-adapted remote sensing data provided by the present invention. DETAILED DESCRIPTION

[0017] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0018] See also Figure 1 , which is a schematic diagram of the steps of the unsupervised domain adaptation remote sensing target recognition method based on generative adversarial technology provided by the present invention.

[0019] In an embodiment of the present invention, a target detection data set is established as image data of a target domain; a remote sensing image data set is acquired as image data of a source domain; the images of the target detection data set include original recognized targets of a remote sensing target recognition model; and the images of the remote sensing image data set include newly recognized targets.

[0020] In a specific implementation, the remote sensing target recognition model is a trained model, and the original recognition target originally used by the remote sensing target recognition model to recognize exists in the image of the target detection data set. The images in the remote sensing image data set include new recognition targets, and the new recognition targets are targets that are expected to be recognized. The present invention migrates remote sensing image data from the source domain to the target domain, and the remote sensing data adapts to the image data distribution of the target domain. The remote sensing target recognition model is used to recognize the remote sensing image after domain adaptation, and the new recognition targets in the remote sensing image are recognized.

[0021] In a specific implementation, domain migration is performed on image data so that the migrated image data can adapt to the remote sensing image data distribution of the target domain, and the remote sensing target recognition model can effectively identify new targets in the domain-adapted image. Compared with the domain migration solution implemented for the model in the prior art, the domain adaptation solution for image data provided by the present invention can save more time and computing power, and it has been verified that the recognition effect is more ideal.

[0022] See also Figure 2 , which is a schematic diagram of the recognition target provided by the present invention. Figure 2 (a) Figure 2 (b) Figure 2 (c) and Figure 2 (d) Images showing the targets being identified as storage tanks, airplanes, wind turbines, and runways, respectively.

[0023] In an embodiment of the present invention, target images with the same size are obtained based on the same data source, and the target images are divided according to the size of the original recognized target. The target images with the same and similar sizes of the original recognized target are divided into the same set as the corresponding target detection data set.

[0024] In the embodiment of the present invention, the original identified targets are divided into small targets, medium targets and large targets according to their sizes.

[0025] In the specific implementation, the object detection dataset consists of 14640 Google tiles of size 1024×1024. The image size of the dataset is consistent but the resolution can be different. The dataset can be divided into training set, test set and validation set in a ratio of 6:2:2 to ensure that each dataset is representative and balanced in terms of target type and distribution.

[0026] In a specific implementation, the target domain image will include the original recognition target, and the target domain image can be divided and grouped according to the size of the original recognition target. The source domain image can be migrated to target domain image groups of different sizes, thereby improving the similarity between the source domain image and the target domain image in distribution, details and texture after migration.

[0027] In a specific implementation, examples are given of the original identification target types including: storage tanks, airplanes, wind turbines, and runways. Among them, airplanes and storage tanks are small targets, wind turbines are medium targets, and their sizes are between small and large targets, while runways are typical large targets. The size ranges of small targets, medium targets, and large targets can be determined according to actual application scenarios. The method provided by the present invention adapts source domain image data to target domain image data, and realizes the remote sensing target recognition model to recognize small targets, medium targets, and large targets (three sizes of new recognition targets).

[0028] In specific implementation, GF-2 remote sensing image data can be selected as the image data of the source domain, and radiation calibration, geometric correction and atmospheric correction can be performed on the GF-2 remote sensing image to improve the image quality of the remote sensing image through preprocessing.

[0029] In an embodiment of the present invention, RGB channel data (R channel is a red channel, G channel is a green channel, and B channel is a blue channel) of an image in a remote sensing image dataset is generated, and the image in the remote sensing image dataset is cropped to the same size as the image in the target detection dataset.

[0030] In the specific implementation, the images in the remote sensing image dataset are cropped to the same size as the images in the target detection dataset to ensure the similarity of the layout.

[0031] In the embodiment of the present invention, scaling and / or cropping is performed on the source domain image and / or the target domain image to align the feature spaces of the source domain image and the target domain image.

[0032] In the specific implementation, the resolution of the target domain image is made consistent with that of the source domain image by scaling, which further ensures the similarity in distribution between the target domain image and the source domain image and improves the effect of domain migration.

[0033] In an embodiment of the present invention, the resolutions of a source domain image and a target domain image are compared; if the resolution of the source domain image is higher than that of the target domain image, the target domain image is enlarged and cropped, and the resolution of the cropped target domain image is the same as that of the source domain image; if the resolution of the source domain image is lower than that of the target domain image, the target domain image is reduced and the source domain image is cropped, and the resolution of the cropped target domain image is the same as that of the source domain image.

[0034] In a specific implementation, enlarging an image can improve the resolution, and in order to ensure the clarity of the image, the newly added pixels can be interpolated; reducing an image can reduce the resolution, and similarly, in order to ensure the clarity of the image, interpolation or resampling can be used.

[0035] In a specific implementation, the sizes of the cropping windows used for the target domain image and the source domain image are consistent, so that the sizes of all the target domain images and the source domain images after cropping can be consistent.

[0036] In a specific implementation, the specific scaling and cropping of the target domain image and the source domain image can be performed using the following formula: R T <R S When 𝜑(𝛿(xT,r),S), R T >R S When 𝛿(xT,r), 𝜑(xS,S xT,r ), Among them, 𝛿 and 𝜑 represent uniform scaling and fixed window cropping, respectively, xT and xS represent the target domain image and the source domain image, respectively, and R T and R S They represent the target domain image resolution and the source domain image resolution respectively, S represents the input size, and r represents the size adjustment factor, which is R T and R S The ratio of S xT,r Represents the size of the adjusted target domain image.

[0037] In the embodiment of the present invention, an unsupervised module is used to migrate a source domain image to a target domain to obtain a first migrated image.

[0038] In an embodiment of the present invention, the unsupervised module is constructed based on a cyclic generative adversarial network.

[0039] In specific implementations, Generative Adversarial Networks (GAN) can be used to generate data similar to the style of the target domain, thereby helping the model to better adapt to the target domain. Furthermore, Cycle-Consistent Generative Adversarial Networks (CycleGAN) can achieve image style transfer without using paired data, helping to reduce the visual domain gap.

[0040] See also Figure 3, which is a schematic diagram of the comparison of images before and after style transfer, wherein (a) represents the source domain image, (b) represents the image after transfer using CycleGAN, and (c) represents the image after transfer using the JCST_GAN module provided by the present invention.

[0041] In the embodiment of the present invention, the first migration image is converted from the original color space to the YCbCr color space, and the corresponding source domain image is converted from the original color space to the YCbCr color space.

[0042] In the embodiment of the present invention, the first migration image and the corresponding source domain image are subjected to channel fusion in color space to obtain a second migration image, and the second migration image is used as the domain adaptation image.

[0043] In the embodiment of the present invention, the Cb channel (blue concentration offset channel) and the Cr channel (red concentration offset channel) of the first migration image are fused with the Y channel (brightness channel) of the source domain image to obtain the second migration image.

[0044] In the specific implementation, in order to further improve the domain migration effect of the generative adversarial network, a joint color space transform (JCST) (used to perform the color space channel fusion provided by the present invention) module, namely the JCST_GAN module, is added on the basis of the original GAN ​​module to convert and separate the color space of the images before and after the migration, so as to avoid the loss of some details and texture information of the migrated images.

[0045] In the specific implementation, due to the diversity of remote sensing image acquisition conditions, remote sensing images present many different features in terms of imaging mode, object scale, color saturation and shooting angle. In practical applications, these problems are intertwined, resulting in significant style differences between remote sensing images in different fields, which brings about a reduction in the accuracy of model recognition, so the problems existing in remote sensing data recognition are solved by channel fusion. In addition, remote sensing target detection is usually carried out using high-resolution images, which usually contain rich detail information, but CycleGAN is mainly designed for style transfer of medium and low resolution images. When it comes to high-resolution images, the existing network depth and complexity are not enough to capture all the detail information. And the generator cannot effectively capture and reproduce subtle texture and edge information when generating high-resolution images, resulting in the loss of a large amount of detail and texture information in the migrated image. Therefore, considering the above factors, the present invention provides a style transfer based on the JCST module, i.e., the JCST_GAN module. First, the source domain image is style-transferred using the CycleGAN network and converted into a target domain image (the first transfer image). After obtaining the transferred target domain image, the color space is converted from the RGB color space to the YCbCr color space. The YCbCr color space is a representation method that separates brightness information (Y channel) from chrominance information (Cb and Cr channels). Next, the Y, Cb, and Cr channels of the transferred target domain image are separated. The chrominance information (Cb and Cr channels) is extracted, and the Y channel of the original source domain image is combined with the Cb and Cr channels of the transferred target domain image. This combination method utilizes the brightness information of the source domain image, maintains the structure and details of the original image, and at the same time gives it the color style of the target domain through the chrominance information of the target domain image. Finally, the combined image not only maintains the details and structure of the source domain image, but also integrates the style features of the target domain, thereby generating a target domain image with visual consistency and high quality.

[0046] In the embodiment of the present invention, a remote sensing target recognition model is used to perform target recognition on a domain adaptation image to obtain a new recognized target.

[0047] In a specific implementation, the remote sensing target recognition model can effectively identify new targets in domain-adapted images.

[0048] See also Figure 4 , which is a schematic diagram of the recognition results before and after style transfer.

[0049] In the embodiments of the present invention, the following methods can be used to identify the position and category of the target in the domain adaptation image: (1) a single-stage detection model (YOLO VS), which directly predicts the position and category of all targets in the image through a single forward propagation; (2) a model based on deformable convolution and Transformer architecture (RE_DETR), which combines the deformable convolution network and the Transformer mechanism, can effectively capture the shape changes and contextual information of the target, and is suitable for target detection and pose estimation in complex scenes; (3) a two-stage model (Faster R-CNN) based on the region proposal network (RPN), which first generates candidate target regions and then classifies and regresses these regions into bounding boxes; (4) a fully convolutional one-stage model, which directly predicts the position and category of the target on feature maps of different scales through a fully convolutional network; (5) target detection based on the improved JCST_GAN model (a recognition model based on generative adversarial networks and color feature recognition) of the present invention.

[0050] In the embodiment of the present invention, the peak signal-to-noise ratio (PSNR) and the structural similarity index (SSIM) are used to evaluate the image quality of the domain adapted image.

[0051] In the specific implementation, the image quality after domain migration using the traditional CycleGAN is: PSNR: 23.99; SSIM: 0.85; the image quality after domain migration using the JCST_GAN provided by the present invention is: PSNR: 30.82; SSIM: 0.99. It can be seen that the image quality generated by the JCST_GAN model is significantly better than other mainstream models, further verifying its advantages in style migration tasks.

[0052] In the specific implementation, in order to evaluate the performance of the detection model in the experiment, five classic evaluation indicators derived from the confusion matrix were used: precision P, recall R, F1 score and mAP value. Precision measures the ability of the model to predict positive samples, recall measures the ability of the model to correctly identify positive samples, F1 score is the harmonic mean of precision and recall, providing a balanced measure of the performance of the model, and mAP reflects the overall performance of the model on all categories.

[0053] See also Figure 5 , which is a quantitative diagram of the recognition effect of the domain-adaptive remote sensing data provided by the present invention, showing the detection results of targets such as wind turbines, storage tanks, airplanes and airport runways.

[0054] By comparison, we can find that the domain-adapted remote sensing images after style transfer perform better than the original images in the target detection task, and the mAP (mean average precision) score is improved by 2% to 4%. This result shows that the CycleGAN improvement method provided by the present invention not only successfully completes the style transfer of the image, but also effectively improves the performance of the target detection model, especially showing significant advantages in multi-target detection scenarios.

[0055] In terms of specific model performance, YOLOv8 and FCOS showed comprehensive improvements in all target detection tasks, with F1 scores increasing by 2% to 6%. This shows that style transfer enhances the detection capabilities of these models, especially in complex multi-target scenes. The RE-DETR model also showed an improvement in F1 scores in the detection of wind turbines, aircraft, and airport runways, but the F1 score did not change in the detection of tank targets because the improvement in accuracy offset this advantage. In contrast, Faster R-CNN's F1 score decreased in the detection of wind turbines, mainly due to the decrease in recall after style transfer, and the model failed to identify some wind turbine targets, but its accuracy increased, indicating that the images after style transfer are closer to the data distribution of the model training set.

[0056] In specific implementation, compared with the prior art that directly performs domain migration on the trained model, the solution provided by the present invention directly performs domain migration on the remote sensing image data to obtain remote sensing image data adapted to the target domain, which significantly reduces the time and computing power costs, and also significantly improves the target recognition effect through the style migration solution of color space fusion.

[0057] The present invention also provides an unsupervised domain adaptation remote sensing target recognition device based on generative adversarial, comprising: a data set establishment unit, a feature space alignment unit, a migration unit and a recognition unit, wherein: the data set establishment unit is used to establish a target detection data set as image data of the target domain; obtain a remote sensing image data set as image data of the source domain; the image of the target detection data set includes the original recognition target of the remote sensing target recognition model; the image of the remote sensing image data set includes the new recognition target; the feature space alignment unit is used to perform scaling and / or cropping on the source domain image and / or the target domain image, and convert the source domain image and the target domain image into a new image. The feature space of the image is aligned; the migration unit is used to migrate the source domain image to the target domain using an unsupervised module to obtain a first migration image; the unsupervised module is constructed based on a generative adversarial network; the first migration image is converted from the original color space to the YCbCr color space, and the corresponding source domain image is converted from the original color space to the YCbCr color space; the first migration image and the corresponding source domain image are channel-fused in color space to obtain a second migration image, and the second migration image is used as a domain adaptation image; the recognition unit is used to use a remote sensing target recognition model to perform target recognition on the domain adaptation image to obtain a new recognized target.

[0058] In a specific implementation, the present invention provides an unsupervised domain adaptation remote sensing target recognition device based on generative adversarial means, in which an execution unit for executing methods, functions and steps, the methods, functions and steps executed by the execution unit can refer to the unsupervised domain adaptation remote sensing target recognition method based on generative adversarial means provided by the present invention.

Claims

1. An unsupervised domain adaptation remote sensing target recognition method based on generative adversarial, characterized in that: include: Establish a target detection dataset as the image data of the target domain; Obtain remote sensing image datasets as image data in the source domain; The images in the target detection data set include the original identified targets of the remote sensing target recognition model; the images in the remote sensing image data set include the newly identified targets; Performing scaling and / or cropping on the source domain image and / or the target domain image to align feature spaces of the source domain image and the target domain image; An unsupervised module is used to migrate the source domain image to the target domain to obtain a first migrated image; The unsupervised module is constructed based on a generative adversarial network; Convert the first migration image from the original color space to the YCbCr color space, convert the corresponding source domain image from the original color space to the YCbCr color space, perform channel fusion of the color space on the first migration image and the corresponding source domain image to obtain a second migration image, and use the second migration image as a domain adaptation image; The remote sensing target recognition model is used to perform target recognition on the domain adaptation image and obtain new recognized targets.

2. The unsupervised domain adaptation remote sensing target recognition method based on generative adversarial according to claim 1 is characterized in that: The establishing of the target detection data set comprises: Based on the same data source, target images with the same size are obtained, and the target images are divided according to the size of the original recognized target. Target images with the same or similar original recognized target sizes are divided into the same set as the corresponding target detection data set.

3. The unsupervised domain adaptation remote sensing target recognition method based on generative adversarial according to claim 2 is characterized in that: According to the size of the original identified target, the original identified target is divided into small targets, medium targets and large targets.

4. The unsupervised domain adaptation remote sensing target recognition method based on generative adversarial according to claim 2 is characterized in that: The step of acquiring the remote sensing image data set further includes: Generate RGB channel data of images in the remote sensing image dataset and crop the images in the remote sensing image dataset to the same size as the images in the target detection dataset.

5. The unsupervised domain adaptation remote sensing target recognition method based on generative adversarial according to claim 4 is characterized in that: The performing scaling and / or cropping on the source domain image and / or the target domain image comprises: Compare the resolution between source domain images and target domain images; If the resolution of the source domain image is higher than that of the target domain image, the target domain image is enlarged and cropped, and the resolution of the cropped target domain image is the same as that of the source domain image; If the resolution of the source domain image is lower than that of the target domain image, the target domain image is downsized and the source domain image is cropped, and the resolution of the cropped target domain image is the same as that of the source domain image.

6. The unsupervised domain adaptation remote sensing target recognition method based on generative adversarial according to claim 1 is characterized in that: The unsupervised module is built based on a recurrent generative adversarial network.

7. The unsupervised domain adaptation remote sensing target recognition method based on generative adversarial according to claim 1 is characterized in that: The step of performing channel fusion in a color space on the first migration image and the corresponding source domain image includes: The Cb channel and the Cr channel of the first migration image are fused with the Y channel of the source domain image to obtain a second migration image.

8. The unsupervised domain adaptation remote sensing target recognition method based on generative adversarial according to claim 7 is characterized in that: The remote sensing target recognition model is any of the following models: A single-stage detection model, a recognition model based on deformable convolution and Transformer, a two-stage recognition model based on region proposal network, a fully convolutional one-stage recognition model, and a recognition model based on generative adversarial network and color feature recognition.

9. The unsupervised domain adaptation remote sensing target recognition method based on generative adversarial according to claim 1 is characterized in that: Peak signal-to-noise ratio and structural similarity index were used to evaluate the image quality of domain adapted images.

10. An unsupervised domain adaptation remote sensing target recognition device based on generative adversarial, characterized in that: include: Dataset building unit, feature space alignment unit, migration unit and recognition unit, where: The data set establishment unit is used to establish a target detection data set as image data of the target domain; obtain a remote sensing image data set as image data of the source domain; the image of the target detection data set includes the original recognized target of the remote sensing target recognition model; the image of the remote sensing image data set includes the newly recognized target; The feature space alignment unit is used to perform scaling and / or cropping on the source domain image and / or the target domain image to align the feature spaces of the source domain image and the target domain image; The migration unit is used to migrate the source domain image to the target domain using an unsupervised module to obtain a first migration image; the unsupervised module is constructed based on a generative adversarial network; the first migration image is converted from an original color space to a YCbCr color space, the corresponding source domain image is converted from an original color space to a YCbCr color space, the first migration image and the corresponding source domain image are fused in color space channels to obtain a second migration image, and the second migration image is used as a domain adaptation image; The recognition unit is used to perform target recognition on the domain adaptation image using the remote sensing target recognition model to obtain a new recognized target.

Images