Step-by-step optical-SAR remote sensing image cross-modal target detection method and device

Abstract

The invention relates to a step-by-step optical-SAR remote sensing image cross-modal target detection method and device, and the method comprises the steps: carrying out the preprocessing of an optical remote sensing image, obtaining a similar SAR sample image represented by a similar SAR image, carrying out the training of a preliminary domain alignment network through employing the similar SAR sample image with a label and an SAR remote sensing sample image without a label, the trained preliminary domain alignment network has the capability of generating a pseudo label according to an SAR remote sensing image, in the network, a domain discrimination unit is utilized to ensure that a feature extraction unit generates domain invariant features to realize cross-modal feature alignment, and the preliminary domain alignment network is utilized to generate the pseudo label of the SAR remote sensing sample image. And according to the obtained pseudo tag and the SAR remote sensing image data set, self-training of the SAR remote sensing image is carried out on a teacher-student network, and a student network with the ability to carry out target identification on the SAR remote sensing image is obtained. By adopting the method, the problem of insufficient SAR label training data can be effectively solved.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a method and apparatus for cross-modal target detection in split-mode optical-SAR remote sensing images. Background Technology

[0002] Target detection in remote sensing images is a crucial topic in the field of refined remote sensing interpretation. Automated analysis of the number, type, and location of targets in remote sensing images can provide valuable remote sensing information for experts and researchers. Currently, target detection in optical remote sensing images has reached a relatively mature stage, with a large number of publicly available labeled datasets and various high-performance detection algorithms. However, in practice, target detection in remote sensing images often encounters adverse conditions, such as cloud cover or insufficient lighting, where optical remote sensing images cannot be acquired. In such cases, it is often necessary to utilize other modalities of remote sensing images for Earth observation. SAR images, which actively image through the transmission and reception of electromagnetic waves, can still acquire effective images under these adverse conditions, enabling all-weather, 24 / 7 Earth observation. However, due to its different imaging mechanism, SAR images differ significantly from intuitive visual representation, requiring expert knowledge for annotation, which is extremely costly. Furthermore, factors such as speckle noise further complicate SAR image annotation.

[0003] Cross-modal learning offers a novel approach to target detection in SAR images, leveraging optical images with a relatively robust research foundation to guide target detection and alleviate the need for target annotation in SAR images. However, the significant modal differences between optical remote sensing images and SAR images due to their different imaging mechanisms pose a core challenge to optical-SAR cross-modal target detection tasks. These substantial modal differences render current domain adaptation methods ineffective. On one hand, the optimization direction of modal alignment conflicts with the optimization direction of target detection under large modal differences; on the other hand, training with two highly different images can confuse the network's discriminative capabilities. Therefore, a suitable method for cross-modal target detection in optical-SAR remote sensing images is needed. Summary of the Invention

[0004] Therefore, it is necessary to provide a step-by-step optical-SAR remote sensing image cross-modal target detection method and apparatus that can solve the problem of cross-modal target detection between optical-SAR images with extremely large modal differences, in order to address the above-mentioned technical problems.

[0005] A step-by-step optical-SAR remote sensing image cross-modal target detection method, the method comprising: Acquire an optical remote sensing image dataset and a SAR remote sensing image dataset, wherein the optical remote sensing image dataset includes multiple labeled optical remote sensing sample images and the SAR remote sensing image dataset includes multiple unlabeled SAR remote sensing sample images. Preprocess each sample image in the optical remote sensing image dataset to obtain SAR-like sample images that resemble SAR image representations. A preliminary domain alignment network is trained using labeled SAR-like sample images and unlabeled SAR remote sensing sample images. The trained preliminary domain alignment network has the ability to generate pseudo-labels based on SAR remote sensing images. The preliminary domain alignment network includes a feature extraction unit and parallel detection head and domain discrimination unit. The domain discrimination unit is used to ensure that the feature extraction unit generates domain-invariant features, thereby achieving cross-modal feature alignment. The trained preliminary domain alignment network is used to generate pseudo-labels for SAR remote sensing sample images. Based on the obtained pseudo-labels and the SAR remote sensing image dataset, the SAR remote sensing images are self-trained on a teacher-student network to obtain a student network capable of target recognition of SAR remote sensing images. Acquire the SAR remote sensing image to be detected, input the SAR remote sensing image to be detected into the trained student network, and obtain the target detection result.

[0006] In one embodiment, the preprocessing of each sample image in the optical remote sensing image dataset includes sequentially performing grayscale transformation processing, frequency domain transformation processing, edge enhancement processing, and multiplicative noise addition processing on the optical remote sensing sample images.

[0007] In one embodiment, when performing frequency domain transformation on the sample image after grayscale transformation: Randomly select a SAR image, and perform fast Fourier transform on the SAR image and the sample image respectively to obtain the amplitude spectrum and phase spectrum of the corresponding two images; Using a mask, a new amplitude spectrum is generated based on the amplitude spectra in the sample image and the SAR image; An inverse Fourier transform is performed on the new amplitude spectrum and the phase spectrum of the sample image to generate a preliminary pseudo-SAR image.

[0008] In one embodiment, the Canny operator is used when performing edge enhancement processing on the preliminary pseudo-SAR image; When performing multiplicative noise addition processing on the preliminary pseudoSAR image after edge enhancement, Rayleigh distributed multiplicative noise is added.

[0009] In one embodiment, the domain discriminant unit includes a gradient inversion layer and a domain discriminator constructed from a multilayer perceptron.

[0010] In one embodiment, during the training of the initial domain alignment network: The feature extraction unit extracts SAR-like image features and SAR image features from the SAR-like sample image and the SAR remote sensing sample image, respectively. The detection head is used to perform target detection based on the SAR-like image features and SAR image features respectively to obtain a first prediction result and a second prediction result. A first detection loss is calculated based on the prediction result and the corresponding label. A second prediction result that is greater than the threshold is also used as a label to calculate a second detection loss. The domain discrimination unit is used to perform domain classification based on the SAR image features and SAR image features, thereby calculating the domain discrimination loss; The preliminary domain alignment network is trained based on the first detection loss, the second detection loss, and the domain discrimination loss.

[0011] In one embodiment, during the self-training of SAR remote sensing images: Weak augmentation and strong augmentation processing are performed on each SAR remote sensing sample image in the SAR remote sensing image dataset to obtain weakly augmented SAR sample images and strongly augmented SAR sample images. The first pseudo-label is generated based on the weak augmented SAR sample image using the pre-trained preliminary domain alignment network, and the second pseudo-label is generated based on the weak augmented SAR sample image using the teacher branch network. The first pseudo-label and the second pseudo-label are then combined to generate a fused pseudo-label. The student branch network is trained using the fused pseudo-labels and strongly augmented SAR sample images, wherein the parameters of the teacher branch network are updated using an exponential moving average through the parameters of the student branch network.

[0012] In one embodiment, a non-maximum suppression strategy is used to fuse the first pseudo-label and the second pseudo-label.

[0013] This application also provides a multimodal target detection device for split-mode optical-SAR remote sensing images, the device comprising: The dataset acquisition module acquires an optical remote sensing image dataset and a SAR remote sensing image dataset. The optical remote sensing image dataset includes multiple labeled optical remote sensing sample images, and the SAR remote sensing image dataset includes multiple unlabeled SAR remote sensing sample images. The preprocessing module is used to preprocess each sample image in the optical remote sensing image dataset to obtain SAR-like sample images that are similar to SAR image representations. The preliminary domain alignment network training module is used to train the preliminary domain alignment network using labeled SAR-like sample images and unlabeled SAR remote sensing sample images. The trained preliminary domain alignment network has the ability to generate pseudo-labels based on SAR remote sensing images. The preliminary domain alignment network includes a feature extraction unit and parallel detection head and domain discrimination unit. The domain discrimination unit is used to ensure that the feature extraction unit generates domain-invariant features, thereby achieving cross-modal feature alignment. The student network training module is used to generate pseudo-labels for SAR remote sensing sample images using the trained preliminary domain alignment network. Based on the obtained pseudo-labels and the SAR remote sensing image dataset, the module performs self-training of SAR remote sensing images on the teacher-student network to obtain a student network capable of target recognition of SAR remote sensing images. The target detection module is used to acquire the SAR remote sensing image to be detected, input the SAR remote sensing image to be detected into the trained student network, and obtain the target detection result.

[0014] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps in the above-described stepwise optical-SAR remote sensing image cross-modal target detection method.

[0015] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the above-described stepwise optical-SAR remote sensing image cross-modal target detection method.

[0016] The aforementioned stepwise optical-SAR remote sensing image cross-modal target detection method and apparatus preprocesses optical remote sensing images to obtain SAR-like sample images with similar SAR image representations. A preliminary domain alignment network is trained using labeled SAR-like sample images and unlabeled SAR remote sensing sample images. The trained preliminary domain alignment network has the ability to generate pseudo-labels based on SAR remote sensing images. This network utilizes a domain discriminant unit to ensure that the feature extraction unit generates domain-invariant features, achieving cross-modal feature alignment. Pseudo-labels for SAR remote sensing sample images are generated using the preliminary domain alignment network. Based on the obtained pseudo-labels and the SAR remote sensing image dataset, self-training of SAR remote sensing images is performed on a teacher-student network, resulting in a student network capable of target recognition from SAR remote sensing images. This method effectively solves the problem of insufficient labeled training data caused by the high cost of labeling SAR images when using neural networks for target detection in SAR images. By leveraging the research foundation and richer publicly available data of optical remote sensing images for auxiliary guidance, target detection can be achieved even in the absence of labeled SAR images. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating a step-by-step optical-SAR remote sensing image cross-modal target detection method in one embodiment; Figure 2 This is a schematic diagram of the optical image preprocessing stage in one embodiment; Figure 3 This is a schematic diagram of the network architecture for the initial domain adaptation phase in one embodiment; Figure 4 This is a schematic diagram of the network architecture for the self-training stage of SAR images in one embodiment; Figure 5 This is a structural block diagram of a multimodal target detection device for distributed optical-SAR remote sensing images in one embodiment; Figure 6 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0019] To address the problem of cross-modal target detection between optical-SAR images with significant modal differences, this application, such as Figure 1 As shown, a step-by-step method for cross-modal target detection in optical-SAR remote sensing images is provided, which specifically includes the following steps: Step S100: Obtain an optical remote sensing image dataset and a SAR remote sensing image dataset. The optical remote sensing image dataset includes multiple labeled optical remote sensing sample images, and the SAR remote sensing image dataset includes multiple unlabeled SAR remote sensing sample images.

[0020] Step S110: Preprocess each sample image in the optical remote sensing image dataset to obtain SAR-like sample images that resemble SAR image representations.

[0021] Step S120: The preliminary domain alignment network is trained using labeled SAR-like sample images and unlabeled SAR remote sensing sample images. The trained preliminary domain alignment network has the ability to generate pseudo-labels based on SAR remote sensing images. The preliminary domain alignment network includes a feature extraction unit and parallel detection heads and domain discrimination units. The domain discrimination unit is used to ensure that the feature extraction unit generates domain-invariant features, thereby achieving cross-modal feature alignment.

[0022] Step S130: Use the trained preliminary domain alignment network to generate pseudo-labels for SAR remote sensing sample images. Based on the obtained pseudo-labels and the SAR remote sensing image dataset, perform self-training of SAR remote sensing images on the teacher-student network to obtain a student network capable of target recognition of SAR remote sensing images.

[0023] Step S140: Obtain the SAR remote sensing image to be detected, input the SAR remote sensing image to be detected into the trained student network, and obtain the target detection result.

[0024] In this method, the labeled optical images are first preprocessed to generate SAR-like images that are closer to the representation of SAR images. Then, adversarial learning is used to train the domain alignment learning network and detection network of the SAR-like images and real SAR images to achieve preliminary domain alignment. After that, the preliminary domain alignment network is used to infer the unlabeled SAR images, and the high-quality results in the obtained inference results are used as pseudo-labels to start the self-training learning of the unlabeled SAR images.

[0025] In step S100, if the target category in the optical remote sensing image dataset is equal to or greater than the target category in the SAR remote sensing image dataset, then during subsequent training, there is no strong correspondence between the optical remote sensing sample images and the SAR remote sensing sample images.

[0026] In step S110, the preprocessing of each sample image in the optical remote sensing image dataset includes sequentially performing grayscale transformation processing, frequency domain transformation processing, edge enhancement processing, and multiplicative noise addition processing on the optical remote sensing sample images, such as... Figure 2 As shown, the optical image is the optical remote sensing sample image, and the preprocessed optical image is the SAR-like sample image.

[0027] In this embodiment, the optical remote sensing sample images are first processed. Perform grayscale transformation to convert the RGB three-channel optical remote sensing image into a single-channel grayscale image. .

[0028] In this embodiment, when performing frequency domain conversion processing, fast Fourier transform is first performed on optical remote sensing sample images and random SAR remote sensing sample images respectively to separate their respective amplitude and phase components. The low-frequency component in the amplitude of the SAR image is used to replace the low-frequency part in the amplitude of the optical image. The updated amplitude map and phase map are used to perform inverse Fourier transform to generate a preliminary pseudo SAR image corresponding to each optical image.

[0029] Specifically, randomly select a SAR image SAR images respectively and sample images Performing a Fast Fourier Transform yields the amplitude and phase spectra of the corresponding two images, represented as follows: ; In the above formula, This indicates the Fast Fourier Transform operation. For amplitude spectrum, The phase spectrum is used. Low-frequency amplitudes in SAR images characterize background radiation properties, while phase information determines semantic structure. This is then analyzed using a mask. A new amplitude spectrum is generated based on the amplitude spectra in the sample image and the SAR image. This involves replacing the low-frequency amplitude at the center of the optical image with the low-frequency components of the SAR image, as shown below: ; In the above formula, the mask Defined as: ; In the above formula, For the height and width of the image, Control the range of low-frequency components to be replaced.

[0030] Furthermore, regarding the new amplitude spectrum and the phase spectrum of the sample image Perform inverse Fourier transform to generate a preliminary pseudo-SAR image, which is represented as follows: ; The frequency domain transformation process described above can weaken irrelevant spectral features in optical images and retain more effective target semantic information.

[0031] In this embodiment, edge enhancement is performed on the edge features in the preliminary pseudo-SAR image to simulate significant edge features in the SAR image. The Canny operator is used to extract the pseudo-SAR image. Edge binary map And the strength is enhanced, the process of which is represented as follows: ; In the above formula, This represents the parameter used to control the amplification magnitude, preferably... . This indicates that the Canny operator is used to extract the edge map of the image. This operation can enhance the continuity of the edges and closely approximate the geometric distortion characteristics of SAR imaging.

[0032] In this embodiment, when performing multiplicative noise addition processing on the preliminary pseudoSAR image after edge enhancement, Rayleigh distributed multiplicative noise is added, which is expressed as: ; In the above formula, Its probability density function is: ; The noise model can be used to match the statistical characteristics of SAR images, and can further fit the noise characteristics in SAR images.

[0033] In step S110, after the above-mentioned multiple preprocessing steps, a preprocessed optical image that is closer to the SAR image representation can be obtained, which is helpful for the subsequent training of the cross-modal target detection network.

[0034] Next, after completing the optical image preprocessing, the process proceeds to the preliminary domain alignment stage in step S120. The domain alignment network is trained using data from both modalities, as illustrated in the network diagram below. Figure 3 As shown.

[0035] In this embodiment, the initial domain alignment network is based on an arbitrary detector network, which can be divided into "feature extraction units" (i.e.,... Figure 3 The feature extraction network in the middle) and the "detection head" (i.e. Figure 3 In the detection head network, an additional "domain discrimination unit" (i.e., a domain discrimination unit) is added after the "feature extraction unit". Figure 3 The domain discrimination network in the middle is used as a branch network, running in parallel with the "detection head".

[0036] In this embodiment, the domain discriminant unit includes a gradient inversion layer and a domain discriminator constructed from a multilayer perceptron.

[0037] In this embodiment, when training the initial domain alignment network, a mixture of SAR-like images and SAR remote sensing sample images is used for training. The detection loss and domain discrimination loss of the SAR-like images and SAR remote sensing sample images are combined to perform backpropagation updates on the network.

[0038] Specifically, during the training of the preliminary domain alignment network, the feature extraction unit extracts SAR-like image features and SAR image features from SAR-like sample images and SAR remote sensing sample images, respectively. The detection head performs target detection based on the SAR-like image features and SAR image features, respectively, to obtain a first prediction result and a second prediction result. The first detection loss is calculated based on the prediction results and the corresponding labels. The second prediction result that is greater than a threshold is also used as a label to calculate the second detection loss. The domain discrimination unit performs domain classification based on the SAR-like image features and SAR image features, thereby calculating the domain discrimination loss. The preliminary domain alignment network is trained based on the first detection loss, the second detection loss, and the domain discrimination loss.

[0039] Furthermore, for SAR-like images It has a truth label GT, and the detection loss is first calculated by the feature extraction unit and the detection head. The detection loss is the loss function corresponding to the detector network used. In one embodiment, Faster R-CNN can be used as the detector, and the loss function of Faster R-CNN is used as the detection loss.

[0040] Furthermore, for SAR remote sensing sample images The prediction result can be obtained first through feature extraction unit and forward propagation of detection head. ,in For target-level prediction results, The bounding box of the prediction result, For the category of the predicted result, The confidence score of the prediction result. This represents the total number of predictions for all images. After obtaining the predictions, a threshold is applied. Filter out prediction results with confidence scores greater than a threshold. ) as SAR remote sensing sample images The pseudo-labels are then analyzed, and their detection loss is calculated similarly. .

[0041] In one embodiment, the threshold The value is set to 0.8.

[0042] Furthermore, for SAR-like images input to the preliminary domain alignment network... and SAR remote sensing sample images After feature extraction, the images are sent to the domain discrimination unit for domain classification. The image obtains a feature map after passing through the feature extraction unit. After passing through the gradient inversion layer, the input is fed into the domain discriminator. : ; In the above formula, For the Sigmoid function, Discriminator parameters. Domain discrimination loss. Defined as: ; In the above formula, The domain label indicates that the input image is a SAR remote sensing sample image. This indicates that the domain label of the input image is a SAR-like image. This represents the number of images in the same training batch. After the gradient inversion layer and domain discriminator described above, the feature extraction network is forced to generate domain-invariant features using adversarial training, achieving cross-modal feature alignment.

[0043] After the initial domain alignment network training in step S120, the network proceeds to self-training learning of SAR images. The network framework is as follows: Figure 4 As shown.

[0044] In step S130, the SAR image self-training process includes: performing weak augmentation and strong augmentation processing on each SAR remote sensing sample image in the SAR remote sensing image dataset to obtain a weakly augmented SAR sample image (i.e. Figure 4 Weakly augmented SAR images and strongly augmented SAR sample images (i.e., Figure 4 The training network generates a first pseudo-label from a weak augmented SAR sample image (using a strong augmented SAR image), and a second pseudo-label from a teacher branch network. The first and second pseudo-labels are then combined to form a fused pseudo-label. The fused pseudo-label and the strong augmented SAR sample image are used to train the student branch network. The parameters of the teacher branch network are updated using an exponential moving average through the parameters of the student branch network.

[0045] In this embodiment, unlabeled SAR remote sensing sample images Two views are generated by applying different weak and strong augmentation methods. and Weak augmentation includes random flipping and random translation, while strong augmentation includes common data augmentation methods such as random scaling, random rotation, random flipping, random noise addition, and random cropping.

[0046] In this embodiment, the preliminary domain alignment network trained in step S120 is used to process weakly augmented SAR sample images. Perform inference and prediction to obtain the inference result. ,in For target-level prediction results, The bounding box of the prediction result, For the category of the predicted result, The confidence score of the prediction result. This represents the total number of all predicted results in the image. This is determined based on the confidence score threshold. The prediction results obtained from the inference are filtered, and the prediction results are scored. Above the threshold The prediction result is used as the pseudo-label, i.e., the first pseudo-label. Among them, the preferred ones are... The value is set to 0.75.

[0047] In this embodiment, a symmetric teacher-student network is constructed based on any detector. The teacher branch network is responsible for processing weakly augmented SAR sample images. Perform forward propagation and filter to obtain pseudo-labels It does not participate in backpropagation. The student branch network merges two sets of pseudo-labels. and The obtained pseudo-labels are used to supervise the training of strongly augmented SAR sample images. For the target detection task, backpropagation is used to update the network parameters, and the parameters of the teacher network are updated using exponential moving average (EMA).

[0048] Specifically, the teacher branch network for weakly augmented SAR sample images Perform forward propagation to obtain the prediction results. ,in For target-level prediction results, The bounding box of the prediction result, For the category of the predicted result, The confidence score of the prediction result. This represents the total number of all predicted results in the image. This is determined based on the confidence score threshold. The prediction results obtained from the inference are filtered, and the prediction results are scored. Above the threshold The prediction results are used as pseudo-labels Preferred, The value is set to 0.75.

[0049] In this embodiment, the first pseudo-label Second pseudo-label Non-Maximum Suppression (NMS) was used for fusion. Specifically, an NMS threshold of 0.75 was set to filter out overlapping labels from the two sets of pseudo-labels, resulting in new fused pseudo-labels. .

[0050] Furthermore, the parameters of the teacher branch network are updated using an exponential moving average (EMA) through the parameters of the student branch network. Let the parameters of the teacher branch network be... The parameters of the student branch network are Given the attenuation rate , The update formula is: .

[0051] In step S140, the student branch network trained in step S130, with the assistance of unlabeled SAR training data and labeled optical images, achieves accurate target detection on the measured SAR images.

[0052] In this paper, to verify the effectiveness of the above-mentioned method in cross-modal target detection of unlabeled SAR images using labeled optical images, experiments were conducted on the labeled optical remote sensing image target detection dataset DIOR and the unlabeled SAR image target detection dataset MSAR. All experimental results were fairly validated on the same MSAR dataset validation set, using common target detection metrics. The results of supervised training using only optical images were compared with those of the existing state-of-the-art cross-modal object detection method ALDI. The experimental results are shown in Table 1.

[0053] Table 1

[0054] In summary, this paper proposes a step-by-step cross-modal target detection method for optical-SAR remote sensing images. By implementing image representation transformation, modal domain alignment, and self-training in stages, it achieves robust cross-modal target detection even when optical-SAR images exhibit significant modal differences. Through this step-by-step, multi-stage processing, the proposed method overcomes problems such as inconsistent network optimization directions and interference between different modal data during training caused by large modal differences. Final experimental verification demonstrates that the proposed method surpasses the performance of networks trained solely on optical images and advanced cross-modal target detection methods. The proposed method can be effectively applied to cross-modal target detection tasks in optical-SAR images, enabling target detection in SAR images without the need for annotation.

[0055] In the aforementioned stepwise optical-SAR remote sensing image cross-modal target detection method, the labeled optical image is first preprocessed to generate a SAR-like image that is relatively closer to the SAR image representation. Then, using adversarial learning, a domain alignment learning network and a detection network for the SAR-like image and the real SAR image are trained to achieve preliminary domain alignment. Afterwards, the preliminary domain alignment network is used to infer on the unlabeled SAR image, and the high-quality results from the inference are used as pseudo-labels to initiate self-training learning for the unlabeled SAR image. This method effectively solves the problem of insufficient labeled training data caused by the high cost of labeling SAR images when using neural networks for target detection in SAR images. By leveraging the research foundation and the richer publicly available data of optical remote sensing images for auxiliary guidance, target detection can be achieved even in the case of unlabeled SAR images.

[0056] It should be understood that, although Figure 1The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0057] In one embodiment, such as Figure 5 As shown, a multimodal target detection device for optical-SAR remote sensing images is provided, comprising: a dataset acquisition module 200, a preprocessing module 210, a preliminary domain alignment network training module 220, a student network training module 230, and a target detection module 240, wherein: The dataset acquisition module 200 acquires an optical remote sensing image dataset and a SAR remote sensing image dataset. The optical remote sensing image dataset includes multiple labeled optical remote sensing sample images, and the SAR remote sensing image dataset includes multiple unlabeled SAR remote sensing sample images. Preprocessing module 210 is used to preprocess each sample image in the optical remote sensing image dataset to obtain SAR-like sample images that are similar to SAR image representations. The preliminary domain alignment network training module 220 is used to train the preliminary domain alignment network using labeled SAR-like sample images and unlabeled SAR remote sensing sample images. The trained preliminary domain alignment network has the ability to generate pseudo-labels based on SAR remote sensing images. The preliminary domain alignment network includes a feature extraction unit and parallel detection heads and domain discrimination units. The domain discrimination unit is used to ensure that the feature extraction unit generates domain-invariant features, thereby achieving cross-modal feature alignment. The student network training module 230 is used to generate pseudo-labels for SAR remote sensing sample images using the trained preliminary domain alignment network. Based on the obtained pseudo-labels and the SAR remote sensing image dataset, the SAR remote sensing images are self-trained on the teacher-student network to obtain a student network with the ability to identify targets in SAR remote sensing images. The target detection module 240 is used to acquire the SAR remote sensing image to be detected, input the SAR remote sensing image to be detected into the trained student network, and obtain the target detection result.

[0058] Specific limitations regarding the multimodal target detection device for distributed optical-SAR remote sensing images can be found in the limitations of the multimodal target detection method for distributed optical-SAR remote sensing images described above, and will not be repeated here. Each module in the aforementioned multimodal target detection device for distributed optical-SAR remote sensing images can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0059] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 6 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a step-by-step optical-SAR remote sensing image cross-modal target detection method. The display screen can be a liquid crystal display (LCD) or an e-ink display. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0060] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0061] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps: Acquire an optical remote sensing image dataset and a SAR remote sensing image dataset, wherein the optical remote sensing image dataset includes multiple labeled optical remote sensing sample images and the SAR remote sensing image dataset includes multiple unlabeled SAR remote sensing sample images. Preprocess each sample image in the optical remote sensing image dataset to obtain SAR-like sample images that resemble SAR image representations. A preliminary domain alignment network is trained using labeled SAR-like sample images and unlabeled SAR remote sensing sample images. The trained preliminary domain alignment network has the ability to generate pseudo-labels based on SAR remote sensing images. The preliminary domain alignment network includes a feature extraction unit and parallel detection head and domain discrimination unit. The domain discrimination unit is used to ensure that the feature extraction unit generates domain-invariant features, thereby achieving cross-modal feature alignment. The trained preliminary domain alignment network is used to generate pseudo-labels for SAR remote sensing sample images. Based on the obtained pseudo-labels and the SAR remote sensing image dataset, the SAR remote sensing images are self-trained on a teacher-student network to obtain a student network capable of target recognition of SAR remote sensing images. Acquire the SAR remote sensing image to be detected, input the SAR remote sensing image to be detected into the trained student network, and obtain the target detection result.

[0062] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor: Obtain an optical remote sensing image dataset and a SAR remote sensing image dataset. The optical remote sensing image dataset includes multiple labeled optical remote sensing sample images, and the SAR remote sensing image dataset includes multiple unlabeled SAR remote sensing sample images. Preprocessing is performed on each sample image in the optical remote sensing image dataset to obtain SAR-like sample images that resemble SAR image representations; A preliminary domain alignment network is trained using labeled SAR-like sample images and unlabeled SAR remote sensing sample images. The trained preliminary domain alignment network has the ability to generate pseudo-labels based on SAR remote sensing images. The preliminary domain alignment network includes a feature extraction unit and parallel detection head and domain discrimination unit. The domain discrimination unit is used to ensure that the feature extraction unit generates domain-invariant features, thereby achieving cross-modal feature alignment. The trained preliminary domain alignment network is used to generate pseudo-labels for SAR remote sensing sample images. Based on the obtained pseudo-labels and the SAR remote sensing image dataset, the SAR remote sensing images are self-trained on a teacher-student network to obtain a student network capable of target recognition of SAR remote sensing images. Acquire the SAR remote sensing image to be detected, input the SAR remote sensing image to be detected into the trained student network, and obtain the target detection result.

[0063] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0064] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0065] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for cross-modal target detection in split-step optical-SAR remote sensing images, characterized in that, The method includes: Obtain an optical remote sensing image dataset and a SAR remote sensing image dataset. The optical remote sensing image dataset includes multiple labeled optical remote sensing sample images, and the SAR remote sensing image dataset includes multiple unlabeled SAR remote sensing sample images. Preprocessing is performed on each sample image in the optical remote sensing image dataset to obtain SAR-like sample images that resemble SAR image representations; A preliminary domain alignment network is trained using labeled SAR-like sample images and unlabeled SAR remote sensing sample images. The trained preliminary domain alignment network has the ability to generate pseudo-labels based on SAR remote sensing images. The preliminary domain alignment network includes a feature extraction unit and parallel detection head and domain discrimination unit. The domain discrimination unit is used to ensure that the feature extraction unit generates domain-invariant features, thereby achieving cross-modal feature alignment. The trained preliminary domain alignment network is used to generate pseudo-labels for SAR remote sensing sample images. Based on the obtained pseudo-labels and the SAR remote sensing image dataset, the SAR remote sensing images are self-trained on a teacher-student network to obtain a student network capable of target recognition of SAR remote sensing images. Acquire the SAR remote sensing image to be detected, input the SAR remote sensing image to be detected into the trained student network, and obtain the target detection result.

2. The method for cross-modal target detection in split-mode optical-SAR remote sensing images according to claim 1, characterized in that, The preprocessing of each sample image in the optical remote sensing image dataset includes sequentially performing grayscale transformation processing, frequency domain transformation processing, edge enhancement processing, and multiplicative noise addition processing on the optical remote sensing sample images.

3. The method for cross-modal target detection in split-mode optical-SAR remote sensing images according to claim 2, characterized in that, When performing frequency domain transformation on sample images after grayscale transformation: Randomly select a SAR image, and perform fast Fourier transform on the SAR image and the sample image respectively to obtain the amplitude spectrum and phase spectrum of the corresponding two images; Using a mask, a new amplitude spectrum is generated based on the amplitude spectra in the sample image and the SAR image; An inverse Fourier transform is performed on the new amplitude spectrum and the phase spectrum of the sample image to generate a preliminary pseudo-SAR image.

4. The method for cross-modal target detection in split-mode optical-SAR remote sensing images according to claim 3, characterized in that, The Canny operator is used when performing edge enhancement processing on the preliminary pseudo-SAR image; When performing multiplicative noise addition processing on the preliminary pseudoSAR image after edge enhancement, Rayleigh distributed multiplicative noise is added.

5. The method for cross-modal target detection in split-mode optical-SAR remote sensing images according to any one of claims 1-4, characterized in that, The domain discriminant unit includes a gradient inversion layer and a domain discriminator constructed from a multilayer perceptron.

6. The method for cross-modal target detection in split-mode optical-SAR remote sensing images according to claim 5, characterized in that, When training the initial domain alignment network: The feature extraction unit extracts SAR-like image features and SAR image features from the SAR-like sample image and the SAR remote sensing sample image, respectively. The detection head is used to perform target detection based on the SAR-like image features and SAR image features respectively to obtain a first prediction result and a second prediction result. A first detection loss is calculated based on the prediction result and the corresponding label. A second prediction result that is greater than the threshold is also used as a label to calculate a second detection loss. The domain discrimination unit is used to perform domain classification based on the SAR image features and SAR image features, thereby calculating the domain discrimination loss; The preliminary domain alignment network is trained based on the first detection loss, the second detection loss, and the domain discrimination loss.

7. The method for cross-modal target detection in split-mode optical-SAR remote sensing images according to claim 6, characterized in that, During the self-training of SAR remote sensing images: Weak augmentation and strong augmentation processing are performed on each SAR remote sensing sample image in the SAR remote sensing image dataset to obtain weakly augmented SAR sample images and strongly augmented SAR sample images. The first pseudo-label is generated based on the weak augmented SAR sample image using the pre-trained preliminary domain alignment network, and the second pseudo-label is generated based on the weak augmented SAR sample image using the teacher branch network. The first pseudo-label and the second pseudo-label are then combined to generate a fused pseudo-label. The student branch network is trained using the fused pseudo-labels and strongly augmented SAR sample images, wherein the parameters of the teacher branch network are updated using an exponential moving average through the parameters of the student branch network.

8. The method for cross-modal target detection in split-mode optical-SAR remote sensing images according to claim 7, characterized in that, A non-maximum suppression strategy is used to fuse the first pseudo-label and the second pseudo-label.

9. A multimodal target detection device for distributed optical-SAR remote sensing images, characterized in that, The device includes: The dataset acquisition module acquires an optical remote sensing image dataset and a SAR remote sensing image dataset. The optical remote sensing image dataset includes multiple labeled optical remote sensing sample images, and the SAR remote sensing image dataset includes multiple unlabeled SAR remote sensing sample images. The preprocessing module is used to preprocess each sample image in the optical remote sensing image dataset to obtain SAR-like sample images that are similar to SAR image representations. The preliminary domain alignment network training module is used to train the preliminary domain alignment network using labeled SAR-like sample images and unlabeled SAR remote sensing sample images. The trained preliminary domain alignment network has the ability to generate pseudo-labels based on SAR remote sensing images. The preliminary domain alignment network includes a feature extraction unit and parallel detection head and domain discrimination unit. The domain discrimination unit is used to ensure that the feature extraction unit generates domain-invariant features, thereby achieving cross-modal feature alignment. The student network training module is used to generate pseudo-labels for SAR remote sensing sample images using the trained preliminary domain alignment network. Based on the obtained pseudo-labels and the SAR remote sensing image dataset, the module performs self-training of SAR remote sensing images on the teacher-student network to obtain a student network capable of target recognition of SAR remote sensing images. The target detection module is used to acquire the SAR remote sensing image to be detected, input the SAR remote sensing image to be detected into the trained student network, and obtain the target detection result.

Images