A domain adaptation based unsupervised SAR image ship target detection method

Abstract

The application provides a kind of unsupervised SAR image ship target detection method based on domain adaptation, gradually transfer knowledge in optical image domain to SAR image domain from pixel level and feature level.In pixel level, introduce diffusion model, convert ship target slice in optical domain into pseudo-SAR domain slice, then combine with pure sea background to generate rich high-quality transition domain image, effectively reduce the semantic gap between optical image domain and SAR image domain.In feature level, by constructing a transformer-based detector, feature alignment operation is performed before result prediction after feature extraction, and SAR domain detector can learn invariant features from transition domain detector to improve target detection capability.Finally, in SAR image domain, the learned detector is used to detect ship targets in SAR images.This method can alleviate the huge labor and time consumption caused by labeling SAR images to a certain extent.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of synthetic aperture radar image detection, and in particular to an unsupervised SAR image ship target detection method based on domain adaptation. BACKGROUND

[0002] Synthetic aperture radar (SAR) is a kind of active microwave imaging sensor, which can observe the earth all day and all weather, and is widely used in military and civilian fields. Ship target detection, as an important part of SAR image interpretation, has attracted more and more attention due to its important role in marine detection and intelligence analysis.

[0003] With the continuous application of deep learning, the effect of SAR image interpretation is continuously improved. However, the SAR ship target detection method based on CNN and other neural networks still needs a large number of labeled quantities to train the detector. Due to the special imaging mechanism of SAR image, artificial labeling needs specially trained interpretation experts, which is expensive and time-consuming. In comparison, optical images have higher readability and more abundant data resources, and unsupervised domain adaptation methods can be used to transfer knowledge from labeled optical image data to unlabeled SAR image data, solving the problem of lack of labeled data for SAR images. However, SAR images and optical images are affected by different imaging mechanisms, and there is a large semantic gap. The current method of using optical images for domain adaptation for SAR ship detection has the problem of insufficient precision.

[0004] In summary, with the increasing amount of SAR image acquisition and the increasing demand for SAR ship target detection under the condition of difficult labeling, the demand for high-precision unsupervised SAR ship target detection algorithm is becoming more and more urgent. SUMMARY

[0005] In view of the deficiencies in the prior art, the present application provides an unsupervised SAR image ship target detection method based on domain adaptation, to solve the technical problem that the demand for high-precision unsupervised SAR ship target detection algorithm is becoming more and more urgent with the increasing amount of SAR image acquisition and the increasing demand for SAR ship target detection under the condition of difficult labeling in the prior art.

[0006] The present application provides an unsupervised SAR image ship target detection method based on domain adaptation, comprising:

[0007] S1, generating a pseudo-SAR image from an optical image in a source domain, and constructing a transition domain;

[0008] S2, construct a transformer-based detector, design a transition domain pseudo-SAR image and target domain SAR image feature alignment strategy, and perform knowledge transfer between the transition domain and the target domain;

[0009] S3, using the obtained detector to detect SAR image ship targets.

[0010] Optionally, the pseudo-SAR image is generated from the optical image in the source domain, and the transition domain is constructed, comprising:

[0011] S101, generating Gaussian noise on the SAR image ship target slice in the source domain to obtain a noisy SAR image ship target slice;

[0012] S102, training a noise estimation model using a UNet model;

[0013] S103, combining the noisy SAR image ship target slice and the optical image ship target slice, and denoising according to the noise estimation model to obtain a pseudo-SAR image.

[0014] Optionally, the Gaussian noise is randomly generated on the SAR image ship target slice in the source domain to obtain a noisy SAR image ship target slice, comprising:

[0015] The Gaussian noise is randomly generated on the SAR image ship target slice in the source domain to destroy the original slice, which can be represented by the following formula:

[0016]

[0017] wherein x0 represents the input SAR image ship target slice, x t represents the SAR image ship target slice after adding Gaussian noise, α t is a pre-set hyperparameter, and ε~N(0, 1) is Gaussian noise.

[0018] Optionally, the noisy SAR image ship target slice and the optical image ship target slice are combined, and denoising is performed according to the noise estimation model to obtain a pseudo-SAR image, comprising:

[0019] The noisy SAR image ship target slice and the optical image ship target slice are combined, and the noise is estimated through the noise estimation model through a reverse process, and the damaged noise SAR image ship slice is gradually restored to the SAR image ship target slice x′0 in the slice combination through multiple iterations, at this time the optical image ship target slice x′0 in the slice combination in the slice combination is the pseudo-SAR image, wherein the iteration process can be represented as:

[0020]

[0021] wherein x′ t represents the combination of the noisy SAR image ship target slice and the optical image ship target slice, σ t ~ N(0, 1) is a Gaussian noise, the noise model ε θ (x′ t , t) is trained by x′ t and t, and θ is the training parameter of the model.

[0022] Optionally, the constructing the transformer-based detector comprises:

[0023] S201, features in the pseudo-SAR image and the SAR image are extracted by using a CNN backbone network, then tokens containing semantic information are generated under the action of position coding, subsequently, the tokens are encoded by using a Transformer Encoder and decoded by using a Transformer Decoder, and model optimization is performed under the constraint of a loss function, thereby forming a transformer detector composed of multiple self-attention layers, Batch Normalization layers and feedforward layers.

[0024] Optionally, the designing a feature alignment strategy for the transition domain pseudo-SAR image and the target domain SAR image comprises:

[0025] S202, the backbone network uses a generator and a discriminator to perform feature alignment, the generator maps pseudo-SAR image domain data containing labels in the transition domain and SAR image domain data not containing labels in the target domain to a feature space, so that the discriminator cannot distinguish whether the data comes from the transition domain or the target domain, and the constraint rule used is as follows:

[0026]

[0027] wherein G represents a feature generator of the backbone network, used to confuse the domain classifier, D is a domain classifier, used to judge whether the feature comes from the target domain or the transition domain, X S represents data in the transition domain, and X T represents data in the target domain.

[0028] Optionally, the performing knowledge transfer between the transition domain and the target domain comprises:

[0029] S203, a GRL layer is used to realize the transfer of deep feature knowledge from the pseudo-SAR domain to the SAR domain, and meanwhile, a loss function is used to constrain the detector in the transition domain, so as to obtain a backbone network capable of correctly detecting ship targets in images.

[0030] The loss function can be expressed as follows:

[0031] L = L adv1 + L adv2 + L det

[0032] Wherein, L adv1 and L adv2 respectively represent the loss function in the two feature alignment, L det represent the loss function constraint applied to the transition area detector, L det can be expressed as follows:

[0033]

[0034]

[0035]

[0036] is the Hungarian loss function, b i is a vector, which defines the center coordinates of the real frame and its height and width relative to the image size, c i is the target class label, L iou is the IoU loss function, λ iou is an adjustment coefficient;

[0037] Thus, the detector for SAR image ship target detection under the unlabeled training condition can be obtained.

[0038] Optionally, the obtained detector is used to detect the SAR image ship target, comprising:

[0039] Using an unsupervised detection method, the detector is used to detect the SAR image ship target, and the SAR image ship target detection result is obtained.

[0040] Compared with the prior art, the present application has the following beneficial effects:

[0041] Knowledge in optical image domain is gradually transferred to SAR image domain at pixel level and feature level. At pixel level, a diffusion model is introduced to convert ship target slices in optical domain into pseudo-SAR domain, and then combined with pure sea background to generate rich and high-quality transition domain images, effectively reducing the semantic gap between optical image domain and SAR image domain. At feature level, by constructing a transformer-based detector, feature alignment operation is performed before feature extraction and result prediction respectively, and SAR domain detector can learn invariant features from transition domain detector to improve target detection capability. Finally, the learned detector is used in SAR image domain to detect ship targets in SAR images. The application can obtain good ship detection performance without SAR image labeling, to a certain extent, it can alleviate the huge labor and time consumption caused by labeling SAR images, and has strong application value. BRIEF DESCRIPTION OF DRAWINGS

[0042] The accompanying drawings, which are incorporated herein and constitute part of the specification, illustrate embodiments consistent with the application and, together with the description, serve to explain the principles of the application.

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, for those of ordinary skill in the art, other drawings can also be obtained without creative labor based on these drawings.

[0044] Figure 1 The method flowchart of the present application is shown in the figure.

[0045] Figure 2 The structure flowchart of the present application is shown in the figure.

[0046] Figure 3 The diffusion model flowchart in the present application is shown in the figure.

[0047] Figure 4 The knowledge transfer flowchart in the present application is shown in the figure. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical scheme and advantages of the embodiments of the present application more clear, the technical scheme of the embodiments of the present application will be described clearly and completely in the following with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor belong to the scope of protection of the present application. The function units with the same and similar structures and functions in the present application have the same and similar structures and functions.

[0049] Referring to Figure 1 The application provides a domain adaptation-based unsupervised SAR image ship target detection method, comprising the following steps:

[0050] S1. generating pseudo SAR images from optical images in a source domain to construct a transition domain;

[0051] S2. constructing a transformer-based detector and designing a feature alignment strategy for the pseudo SAR images in the transition domain and the target domain SAR images to perform knowledge transfer between the transition domain and the target domain;

[0052] S3. using the obtained detector to detect SAR image ship targets.

[0053] Referring to Figure 2 In the embodiment, S1. generating pseudo SAR images from optical images in a source domain to construct a transition domain.

[0054] The optical image domain and the SAR image domain are different source image domains, and there is a semantic gap between them, so direct data transfer cannot achieve good results. Therefore, one of the design focuses of the application is to use a diffusion model to construct a pseudo SAR image transition domain from labeled optical images. The steps include the following:

[0055] a1. using a diffusion model to convert optical image slices into pseudo SAR image slices;

[0056] Referring to Figure 3 The application uses a diffusion model to realize the conversion from optical image ship target slices to pseudo SAR image ship target slices, and the main idea is to combine optical slices with SAR slices with added noise, and after denoising, the SAR image slices can be obtained again, that is, the optical slices are converted into pseudo SAR image slices.

[0057] First, the SAR image ship target slices in the source domain are selected, and Gaussian noise is randomly generated to destroy the original slices, which can be represented by the following formula:

[0058]

[0059] where x0 represents the input SAR image ship target slice, x t represents the SAR image ship target slice after adding Gaussian noise. alpha t is a pre-set hyperparameter, called Noise schedule, which is a small value. epsilon ~ N(0, 1) is Gaussian noise. The above formula represents the forward process of the diffusion model.

[0060] The noisy SAR image ship target slice is combined with the optical image ship target slice. Through a reverse process, the noise is estimated using the predicted noise model. After multiple iterations, the noisy SAR image ship target slice is gradually restored to the SAR image ship target slice x′0 in the slice combination. At this point, the optical image ship target slice x′0 in the slice combination is the pseudo-SAR image. One iteration can be represented as:

[0061]

[0062] Here, x′ t σ represents the combination of a noisy SAR image slice and an optical image slice of a ship target. t ~N(0,1) is a Gaussian noise used to represent the difference between the estimate and the actual value. The key to training is to train a signal generated by x′. t And the model for estimating noise ε θ (x′ t ,t), where θ are the training parameters of the model. A noisy model ε is trained using a UNet model. θ (x′ t Under the influence of the loss function, the predicted value (t) is improved. It is similar to the ε that is actually used for destruction.

[0063] Next, the obtained noise model ε can be used. θ (x′ t The optical image ship target slice is transformed using the method t). A noisy SAR image ship slice x is randomly generated from N(0,1) and used as the noisy SAR image ship slice. T Then, it is combined with the optical ship slice and gradually restored to x0 using a denoising formula. At this point, the optical image ship target slice is converted into a pseudo SAR image.

[0064] a2. Synthesizing pseudoSAR images using labeled optical images

[0065] Considering that ship targets in SAR images do not have wakes, this invention removes the wakes of ship targets in optical images, resulting in higher conversion quality. Then, based on the original optical image labels, the converted SAR-like image slices of ship targets are combined with a pure ocean background image, thus obtaining a labeled pseudo-SAR image. This pseudo-SAR image, situated between the optical and SAR images, serves as a bridge for knowledge transfer, effectively reducing the semantic gap between the original optical and SAR images.

[0066] S2, construct a transformer-based detector, design a transition domain pseudo-SAR image and target domain SAR image feature alignment strategy, and perform knowledge transfer between the transition domain and the target domain.

[0067] According to the pseudo-SAR image of the transition domain and the SAR image in the source domain (target domain) as a data set, a detector based on the transformer structure is constructed. Then, alignment operations are performed after feature extraction and before output prediction, respectively, to perform knowledge transfer at the feature level and the prediction level. The specific steps are as follows:

[0068] a1. Construct a transformer-based detector

[0069] At present, the transformer detector represented by DETR has good effects in SAR image target detection, therefore, the detector used in the present application is also based on the transformer structure.

[0070] Firstly, a backbone CNN network such as ResNet is used to extract features in the image, then under the action of position coding, tokens containing semantic information are generated. Then, the tokens are encoded by using the Transformer Encoder, and decoded by using the Transformer Decoder, and the model is optimized under the constraint of the loss function. The basic module of the transformer is composed of multiple layers of self-attention layers, Batch Normalization layers and feedforward layers. In the present application, 3 basic modules are cascaded in the encoder part, and the decoder is composed of one basic module, and the rest is of the same type as the DETR model, and will not be repeated here.

[0071] a2. Perform feature alignment to transfer knowledge from the pseudo-SAR image domain to the SAR image domain

[0072] The present application performs feature alignment after the backbone network of the detector extracts features and before the prediction result.

[0073] Referring to Figure 4 The backbone network used in the present application is a ResNet-50 network, under the action of 4 layers of convolution modules, deep semantic information containing image target structure and other features can be obtained, feature alignment at this point can help the detector to transfer knowledge from the beginning of training, and reduce the difficulty of knowledge transfer. Before the prediction result, position decoding is needed, then feature alignment is performed to help the detector to calibrate before prediction and improve the unsupervised detection accuracy.

[0074] Take the feature alignment operation after the backbone network extracts the feature as an example to describe the feature alignment strategy used in the application. In the application, the pseudo SAR image domain data containing labels is the transition domain data, and the SAR image domain data without labels is the target domain data. The idea of adversarial learning is used to perform feature alignment by using the functions of the generator and the discriminator. The generator is used to map the data of the transition domain and the target domain to the feature space, and the goal is to make the discriminator unable to distinguish whether the data is from the transition domain or the target domain, and the constraint rule used is as follows.

[0075]

[0076] Wherein, G represents the feature generator of the backbone network, D is a domain classifier used to judge whether the feature is from the target domain or the transition domain, and the role of G is to confuse the domain classifier. S X T represents the data in the transition domain, and X adv1 represents the data in the target domain. Under the interaction of the GRL layer, the feature generator G can better learn the feature representation from the source domain, and realize knowledge transfer.

[0077] Similarly, before the prediction result, the GRL layer is used to realize the transfer of deep feature knowledge from the pseudo SAR domain to the SAR domain. At the same time, the loss function is used to constrain the detector of the source domain to obtain a network capable of correctly detecting the ship target in the image.

[0078] The loss function of this stage can be expressed as follows:

[0079] L = L adv1 + L adv2 + L det

[0080] Wherein, L adv1 and L adv2 represent the loss functions in the two feature alignment operations respectively, L det represents the loss function constraint applied to the detector of the source domain, and L det can be expressed as follows:

[0081]

[0082]

[0083]

[0084] is the Hungarian loss function, b i is a vector defining the center coordinates of the real box and its height and width relative to the image size, c i is the target class label, L iou is the IoU loss function, and λ iouis an adjustment coefficient.

[0085] Thus, the detector for SAR image ship target detection under the label-free training condition can be obtained.

[0086] S3, using the obtained detector to detect the SAR image ship target.

[0087] Using the unsupervised detection method, the detector is used for SAR image detection, and better SAR image ship target detection results are obtained.

[0088] It is worth noting that in the test, the optical DOTA data set is used as the source domain, the AIR-SARship-1.0 data set is used as the target domain for verification, and the training set and the test set are divided in the ratio of 8:2. The method has an advantage of about 0.3 in the F1 comprehensive index evaluation compared with the traditional unsupervised SAR image target detection method CFAR.

[0089] The application provides an unsupervised SAR image ship target detection method based on domain adaptation. Knowledge in the optical image domain is gradually transferred to the SAR image domain at the pixel level and the feature level. At the pixel level, a diffusion model is introduced to convert the ship target slices in the optical domain into pseudo-SAR domain slices, which are then combined with pure ocean background to generate rich high-quality transition domain images, effectively reducing the semantic gap between the optical image domain and the SAR image domain. At the feature level, a transformer-based detector is constructed, and feature alignment operations are performed using an adversarial learning strategy before feature extraction and result prediction. The SAR domain detector can learn invariant features from the transition domain detector, improving the target detection capability. Finally, the learned detector is used in the SAR image domain to detect ship targets in the SAR image to be processed. The method proposed in the application can obtain good ship detection performance without SAR image labeling, to some extent, relieving the huge labor and time consumption caused by labeling SAR images, and has strong application value.

[0090] It has to be noted that, in the present document, relational terms are intended only to convey a possible relationship between elements or

[0091] The foregoing is considered as illustrative only of the principles of the application. Numerous modifications and changes will readily occur to those skilled in the art, which modifications and changes are to be understood as intended to be encompassed by the general scope of the application. Accordingly, the application is not to be limited to the above described or illustrated embodiments that are merely given by way of example. It is also be understood that various combinations of the above described embodiments and variations thereof are encompassed by the application.

Claims

1. A method for ship target detection based on domain adaptation in unsupervised SAR images, characterized in that, include: S1. Generate a pseudo-SAR image from the optical image in the source domain and construct the transition domain; S2. Features are extracted from pseudo-SAR and SAR images using a CNN backbone network. The backbone network uses a generator and a discriminator for feature alignment. The generator maps labeled pseudo-SAR image domain data in the transition domain and unlabeled SAR image domain data in the target domain to the feature space, making it impossible for the discriminator to distinguish whether the data comes from the transition domain or the target domain. The constraint rules used are as follows: Where G represents the feature generator of the backbone network, used to confuse the domain classifier; D is the domain classifier, used to determine whether the feature comes from the target domain or the transition domain; X... S X represents data in the transition domain. T It represents the data in the target domain and performs knowledge transfer between the transition domain and the target domain. S3. Use the obtained detector to detect ship targets in SAR images.

2. The unsupervised SAR image ship target detection method based on domain adaptation as described in claim 1, characterized in that, The process of generating a pseudo-SAR image from an optical image in the source domain and constructing a transition domain includes: S101. Randomly generate Gaussian noise for the SAR image ship target slice in the source domain to obtain a noisy SAR image ship target slice. S102. Train a noise prediction model using the UNet model; S103. Combine the noisy SAR image ship target slice and the optical image ship target slice, and denoise according to the estimated noise model to obtain a pseudo SAR image.

3. The unsupervised SAR image ship target detection method based on domain adaptation as described in claim 2, characterized in that, The step of randomly generating Gaussian noise for the SAR image ship target slice in the source domain to obtain a noisy SAR image ship target slice includes: Select a SAR image slice of a ship target from the source domain, randomly generate Gaussian noise to destroy the original slice, which can be expressed by the following formula: Where x0 represents the input SAR image ship target slice, x t This represents a slice of ship targets in a SAR image after Gaussian noise has been added, α t These are pre-defined hyperparameters, and εN(0,1) is Gaussian noise.

4. The unsupervised SAR image ship target detection method based on domain adaptation as described in claim 3, characterized in that, The process of combining noisy SAR image ship target slices and optical image ship target slices, and denoising them according to the estimated noise model to obtain a pseudo SAR image includes: The noisy SAR image ship target slice is combined with the optical image ship target slice. Through a reverse process, the noise is estimated using the predicted noise model. After multiple iterations, the noisy SAR image ship target slice is gradually restored to the SAR image ship target slice x0′ in the slice combination. At this point, the optical image ship target slice x0′ in the slice combination is the pseudo-SAR image. One iteration can be represented as: Where, x t ′ represents the combination of a noisy SAR image slice and an optical image slice of a ship target, σ t N(0,1) is Gaussian noise, and the noise model ε θ (x′ t ,t) by x t ' and t are derived from training, and θ is the training parameter of the model.

5. The unsupervised SAR image ship target detection method based on domain adaptation as described in claim 4, characterized in that, The construction of the transformer-based detector includes: S201. Features in pseudo-SAR and SAR images are extracted using a CNN backbone network. Then, under the action of position encoding, tokens containing semantic information are generated. Subsequently, the tokens are encoded using a Transformer Encoder and decoded using a Transformer Decoder. The model is optimized under the constraint of the loss function to form a transformer detector consisting of multiple layers of self-attention layers, Batch Normalization layers, and feedforward layers.

6. The unsupervised SAR image ship target detection method based on domain adaptation as described in claim 5, characterized in that, The knowledge transfer between the transition domain and the target domain includes: S202. Using the GRL layer, the deep feature knowledge is transferred from the pseudo-SAR domain to the SAR domain. At the same time, the loss function is used to constrain the detector in the transition domain to obtain a backbone network that can correctly detect ship targets in the image. The loss function described above can be expressed as follows: L=L adv1 +L adv2 +L det Among them, L adv1 and L adv2 L represents the loss function in each of the two feature alignments. det L represents the loss function constraint imposed on the detector in the transition domain. det It can be expressed as the following formula: It is the Hungarian loss function, b i It is a vector that defines the center coordinates of the ground truth bounding box and its height and width relative to the image size, c i It is a target category tag, L iou It is the IoU loss function, λ iou It is an adjustment factor; Thus, a detector for ship target detection in SAR images under label-free training conditions can be obtained.

7. The unsupervised SAR image ship target detection method based on domain adaptation as described in claim 6, characterized in that, The detection of ship targets in SAR images using the obtained detector includes: Using an unsupervised detection method, the detector is used to detect ship targets in SAR images, and the SAR image ship target detection results are obtained.

Images