SAR Image Target Detection Method and Apparatus Based on Heterogeneous Image Source
By constructing a dual-stream branch feature alignment extraction network based on heterogeneous images, selecting training data with similar feature distributions and designing loss branches, the problem of limited measurement accuracy and applicability in target detection of small-sample SAR images is solved, and efficient target detection is achieved.
Patent Information
- Application Number
- CN202410185610.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-02-19
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2044-02-19
AI Technical Summary
In SAR image target detection under small sample conditions, the measurement accuracy and applicable conditions are limited, which cannot meet the detection needs of radar and optical uncertainties under complex electromagnetic environments or changing weather conditions.
By acquiring source and target domain datasets of heterogeneous images, a dual-stream branch feature alignment extraction network is trained. Training data with similar feature distributions are selected, and a weight-shared feature extraction network is constructed. A loss branch inspired by the distribution distance loss function and the difference in domain knowledge distribution is designed to achieve feature alignment and knowledge sharing.
It improves the accuracy and efficiency of target detection in SAR images, overcomes the difficulty of detection under small sample conditions, and enhances the target recognition capability in complex environments.
Smart Images

Figure CN118015259B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of SAR image detection technology, and in particular to a method and apparatus for SAR image target detection based on heterogeneous images. Background Technology
[0002] Synthetic Aperture Radar (SAR) is an indispensable monitoring tool in the field of remote sensing. It is an active microwave remote sensing imaging system with high spatial resolution imaging capabilities. It can monitor land and ocean around the clock and in all weather conditions, and is not limited by sunlight or weather conditions, thus having significant application value in the field of remote sensing. Unlike common visible light images, SAR images can reveal the microwave characteristics of ground objects and targets. Its imaging quality is not only limited by various factors such as length, incident angle, and polarization, but also closely related to the internal structure, arrangement, and material characteristics of the target.
[0003] However, in SAR image target detection scenarios with small sample conditions, the measurement accuracy and applicable conditions are limited, which cannot meet the needs of radar and optical uncertainty detection under complex electromagnetic environments or changing weather conditions. Summary of the Invention
[0004] Therefore, it is necessary to provide a method, apparatus, device, and medium for SAR image target detection based on heterogeneous images that can efficiently and accurately perform target detection on high-resolution remote sensing images, addressing the aforementioned technical problems.
[0005] A SAR image target detection method based on heterogeneous images, the method comprising:
[0006] Obtain source domain datasets and target domain datasets for the same type of target, wherein both the source domain datasets and target domain datasets include multiple sample images;
[0007] The source domain dataset and the target domain dataset are divided into multiple training datasets. Each training dataset includes a portion of the source domain sample images from the source domain dataset and all the target domain sample images from the target domain dataset.
[0008] Calculate the feature distribution of sample images in each of the training datasets, and filter the source domain sample images in the training datasets according to the distance between the feature distribution of the target domain sample images and the feature distribution of the source domain sample images to obtain the filtered training datasets;
[0009] The dual-stream branch feature alignment extraction network is trained using multiple selected training datasets and a target domain dataset. In the dual-stream branch feature alignment extraction network, a weight-shared hybrid domain feature extraction network and a target domain feature extraction network are used to extract features from sample images in the selected training dataset and target domain dataset, respectively, to obtain corresponding hybrid image features and target domain image features. The feature alignment network is used to align the hybrid image features and target domain image features to obtain aligned features. The detection head then detects targets in the target domain sample images based on the aligned features.
[0010] The adjustable parameters in the dual-stream branch feature alignment extraction network are optimized according to the total loss function of object detection until convergence, resulting in a trained dual-stream branch feature alignment extraction network. The total loss function of object detection includes a distribution distance loss function, which is calculated based on the distance between the feature distribution of the target domain sample image and the feature distribution of the source domain sample image in the training dataset.
[0011] The target domain feature extraction network, feature alignment network, and detection head are extracted from the trained dual-stream branch feature alignment extraction network to construct a SAR image target detection network. The SAR image to be detected is input into the SAR image target detection network to detect the target in the SAR image.
[0012] In one embodiment, the sample images in the source domain dataset are optical images, and the sample images in the target domain dataset are SAR images.
[0013] In one embodiment, calculating the feature distribution of sample images in each of the training datasets and filtering the source domain sample images in the training datasets based on the distance between the feature distribution of the target domain sample images and the feature distribution of the source domain sample images includes:
[0014] A feature extraction network is used to extract features from the sample images in the training dataset, and the feature distribution of each sample image is calculated.
[0015] Calculate the target domain average feature distribution of multiple target domain sample images in the training dataset, and calculate the distribution distance between the feature distribution of each source domain sample image in the training dataset and the target domain average feature distribution;
[0016] Based on the calculated distribution distance and the filtering strategy, source domain sample images in the training dataset whose feature distributions are far from the average feature distribution of the target domain are filtered out.
[0017] In one embodiment, the distribution distance metric function used to calculate the distance between the feature distribution of each source domain sample image in the training dataset and the average feature distribution of the target domain is expressed as:
[0018]
[0019] In the above formula, x and x' represent the feature distribution of the source domain sample image and the average feature distribution of the target domain, respectively, and n t This indicates the number of SAR image samples in the target domain.
[0020] In one embodiment, the filtering strategy is expressed as:
[0021]
[0022] In the above formula, This represents the filtered training dataset. These represent the sample images and their corresponding ground truth labels in the filtered training dataset. This indicates that the sample images are sorted in ascending order based on their distribution distance. This indicates that the perception enhancement strategy generates new data, φ opt Let n represent the new optimization function. a n b And k represents the number of samples before screening, the number of samples after screening, and the shrinkage rate, respectively.
[0023] In one embodiment, the total target detection loss function includes a distribution distance loss function, a target confidence loss function for the target domain, and a bounding box loss function, as well as a target confidence loss function for the source domain and a bounding box loss function.
[0024] Furthermore, weights are set for the target confidence loss function and the bounding box loss function, respectively.
[0025] In one embodiment, the SAR image detection method is applied to a SAR image target detection task where the target is a ship.
[0026] The application also provides a SAR image target detection device based on heterogeneous images, the device comprising:
[0027] The dataset acquisition module is used to acquire source domain datasets and target domain datasets for the same type of target, wherein both the source domain datasets and the target domain datasets include multiple sample images;
[0028] The training dataset construction module is used to divide the source domain dataset and the target domain dataset into multiple training datasets. Each training dataset includes a portion of the source domain sample images in the source domain dataset and all the target domain sample images in the target domain dataset.
[0029] The training dataset optimization module is used to calculate the feature distribution of sample images in each training dataset, and to filter the source domain sample images in the training dataset according to the distance between the feature distribution of the target domain sample images and the feature distribution of the source domain sample images, so as to obtain the filtered training dataset.
[0030] The neural network training module is used to train the dual-stream branch feature alignment extraction network using multiple selected training datasets and a target domain dataset. In the dual-stream branch feature alignment extraction network, a weight-shared hybrid domain feature extraction network and a target domain feature extraction network are used to extract features from the sample images in the selected training dataset and the target domain dataset, respectively, to obtain corresponding hybrid image features and target domain image features. The feature alignment network is used to align the hybrid image features and the target domain image features to obtain aligned features. The detection head then detects the target in the target domain sample image based on the aligned features.
[0031] The trained neural network module is used to optimize the adjustable parameters in the dual-stream branch feature alignment extraction network according to the total loss function of object detection until convergence, thereby obtaining the trained dual-stream branch feature alignment extraction network. The total loss function of object detection includes a distribution distance loss function, which is calculated based on the distance between the feature distribution of the target domain sample image and the feature distribution of the source domain sample image in the training dataset.
[0032] The SAR image target detection module is used to extract the target domain feature extraction network and the detection head from the trained dual-stream branch feature alignment extraction network to construct a SAR image target detection network. The SAR image to be detected is input into the SAR image target detection network to detect the target in the SAR image.
[0033] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program performing the following steps:
[0034] Obtain source domain datasets and target domain datasets for the same type of target, wherein both the source domain datasets and target domain datasets include multiple sample images;
[0035] The source domain dataset and the target domain dataset are divided into multiple training datasets. Each training dataset includes a portion of the source domain sample images from the source domain dataset and all the target domain sample images from the target domain dataset.
[0036] Calculate the feature distribution of sample images in each of the training datasets, and filter the source domain sample images in the training datasets according to the distance between the feature distribution of the target domain sample images and the feature distribution of the source domain sample images to obtain the filtered training datasets;
[0037] The dual-stream branch feature alignment extraction network is trained using multiple selected training datasets and a target domain dataset. In the dual-stream branch feature alignment extraction network, a weight-shared hybrid domain feature extraction network and a target domain feature extraction network are used to extract features from sample images in the selected training dataset and target domain dataset, respectively, to obtain corresponding hybrid image features and target domain image features. The feature alignment network is used to align the hybrid image features and target domain image features to obtain aligned features. The detection head then detects targets in the target domain sample images based on the aligned features.
[0038] The adjustable parameters in the dual-stream branch feature alignment extraction network are optimized according to the total loss function of object detection until convergence, resulting in a trained dual-stream branch feature alignment extraction network. The total loss function of object detection includes a distribution distance loss function, which is calculated based on the distance between the feature distribution of the target domain sample image and the feature distribution of the source domain sample image in the training dataset.
[0039] The target domain feature extraction network and detection head are extracted from the trained dual-stream branch feature alignment extraction network to construct a SAR image target detection network. The SAR image to be detected is input into the SAR image target detection network to detect the target in the SAR image.
[0040] A computer-readable storage medium having a computer program stored thereon, the computer program performing the following steps when executed by a processor:
[0041] Obtain source domain datasets and target domain datasets for the same type of target, wherein both the source domain datasets and target domain datasets include multiple sample images;
[0042] The source domain dataset and the target domain dataset are divided into multiple training datasets. Each training dataset includes a portion of the source domain sample images from the source domain dataset and all the target domain sample images from the target domain dataset.
[0043] Calculate the feature distribution of sample images in each of the training datasets, and filter the source domain sample images in the training datasets according to the distance between the feature distribution of the target domain sample images and the feature distribution of the source domain sample images to obtain the filtered training datasets;
[0044] The dual-stream branch feature alignment extraction network is trained using multiple selected training datasets and a target domain dataset. In the dual-stream branch feature alignment extraction network, a weight-shared hybrid domain feature extraction network and a target domain feature extraction network are used to extract features from sample images in the selected training dataset and target domain dataset, respectively, to obtain corresponding hybrid image features and target domain image features. The feature alignment network is used to align the hybrid image features and target domain image features to obtain aligned features. The detection head then detects targets in the target domain sample images based on the aligned features.
[0045] The adjustable parameters in the dual-stream branch feature alignment extraction network are optimized according to the total loss function of object detection until convergence, resulting in a trained dual-stream branch feature alignment extraction network. The total loss function of object detection includes a distribution distance loss function, which is calculated based on the distance between the feature distribution of the target domain sample image and the feature distribution of the source domain sample image in the training dataset.
[0046] The target domain feature extraction network and detection head are extracted from the trained dual-stream branch feature alignment extraction network to construct a SAR image target detection network. The SAR image to be detected is input into the SAR image target detection network to detect the target in the SAR image.
[0047] The aforementioned SAR image target detection method and apparatus based on heterogeneous images divides the source domain dataset and target domain dataset into multiple training datasets, calculates the feature distribution of each sample image in the training dataset, and filters out source domain sample images whose feature distributions are far from those of the target domain sample images, thereby optimizing the training dataset. Then, the optimized training dataset is used to train a dual-stream branch alignment extraction network with weight sharing, and the target domain feature extraction network and detection head are extracted from the trained network and applied to the SAR image target detection task. This method overcomes the problem of insufficient SAR image samples while improving target detection accuracy and efficiency. Attached Figure Description
[0048] Figure 1 This is a flowchart illustrating a SAR image target detection method based on heterogeneous images in one embodiment;
[0049] Figure 2This is a schematic diagram of instance-level and anchor-frame-level domain-aware adaptive enhancement in one embodiment;
[0050] Figure 3 This is a schematic diagram illustrating the distance measurement between two points in a high-level semantic feature in one embodiment;
[0051] Figure 4 This is a schematic diagram illustrating the difference in feature distribution between the target domain and the source domain after using different feature alignment methods in one embodiment.
[0052] Figure 5 This is a schematic diagram of the sigmoid function curve in one embodiment;
[0053] Figure 6 This is a schematic diagram of the model training process in a SAR image detection method in one embodiment;
[0054] Figure 7 This is a structural block diagram of a SAR image target detection device based on heterogeneous images in one embodiment;
[0055] Figure 8 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation
[0056] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0057] In SAR image target detection scenarios with small sample conditions, the measurement accuracy and applicability of a single sensor are limited, failing to meet the detection needs of radar and optical uncertainties under complex electromagnetic environments or changing weather conditions. Meanwhile, efficiently interpreting massive amounts of high-resolution remote sensing imagery and fully utilizing multi-source information will become a key aspect of remote sensing information applications. Therefore, combining information acquired by multi-source heterogeneous sensors to improve the target recognition efficiency of SAR images under small sample conditions is crucial.
[0058] In the above context, such as Figure 1 As shown, a SAR image target detection method based on heterogeneous images is provided, including the following steps:
[0059] Step S100: Obtain source domain datasets and target domain datasets for the same type of target. Both source domain datasets and target domain datasets include multiple sample images.
[0060] Step S110: Divide the source domain dataset and the target domain dataset into multiple training datasets. Each training dataset includes a portion of the source domain sample images from the source domain dataset and all the target domain sample images from the target domain dataset.
[0061] Step S120: Calculate the feature distribution of sample images in each training dataset, and filter the source domain sample images in the training dataset according to the distance between the feature distribution of the target domain sample images and the feature distribution of the source domain sample images to obtain the filtered training dataset.
[0062] Step S130: The dual-stream branch feature alignment extraction network is trained using multiple selected training datasets and the target domain dataset. In the dual-stream branch feature alignment extraction network, the weight-shared hybrid domain feature extraction network and the target domain feature extraction network are used to extract features from the sample images in the selected training dataset and the target domain dataset, respectively, to obtain the corresponding hybrid image features and target domain image features. The feature alignment network is used to align the hybrid image features and the target domain image features to obtain the aligned features. The detection head then detects the target in the target domain sample image based on the aligned features.
[0063] Step S140: Optimize the adjustable parameters in the dual-stream branch feature alignment extraction network according to the total loss function for target detection until convergence, and obtain the trained dual-stream branch feature alignment extraction network. The total loss function for target detection includes a distribution distance loss function, which is calculated based on the distance between the feature distribution of the target domain sample image and the feature distribution of the source domain sample image in the training dataset.
[0064] Step S150: Extract the target domain feature extraction network and detection head from the trained dual-stream branch feature alignment extraction network to construct the SAR image target detection network. Input the SAR image to be detected into the SAR image target detection network to detect the target in the SAR image.
[0065] In this embodiment, a large number of readily available, labeled optical remote sensing images with similar targets are used to assist in target detection and recognition tasks for SAR images under small sample conditions. Secondly, a data domain-aware adaptive enhancement strategy is constructed, generating a large number of candidate samples to address the insufficiency and similarity issues of the enhanced domain-aware samples, while mitigating the distribution differences and domain shift phenomena between the visible light image data domain and the SAR image data domain. Next, to effectively utilize SAR domain knowledge, a loss branch inspired by the distribution differences of domain knowledge is designed. Then, a weight-sharing dual-stream branch feature alignment extraction network is constructed to better achieve knowledge extraction and sharing between the two domains. Finally, to better measure the distribution differences between the source and target domains, a novel and efficient plug-and-play distance metric function plugin is designed, enabling the model to adaptively and iteratively optimize the distribution of candidate images in the target domain to train a robust detector.
[0066] In step S100, the sample images in the source domain dataset are optical images, and the sample images in the target domain dataset are SAR images.
[0067] In this embodiment, different features of targets in optical and SAR images are used to train the neural network. Due to the different imaging principles of the two types of images, optical remote sensing sensors and SAR sensors each have their advantages in Earth observation. SAR sensors have extremely strong all-weather detection capabilities, able to penetrate clouds and fog, and are unaffected by haze or lighting conditions; however, their texture and ground object radiation information are not clear enough, making them difficult to identify accurately. In contrast, optical remote sensing images can more intuitively reflect information such as texture, color, and shape, but their data acquisition capabilities are limited due to the influence of sunlight and weather. Optical remote sensing images can extract rich spectral information from radiation features, which is more conducive to classification and interpretation.
[0068] However, due to the significant differences between optical and SAR images, existing optical image detection methods cannot be directly applied to SAR images. The main problems to be solved include: (1) The imaging mechanisms of SAR and optical images differ significantly, resulting in obvious differences in appearance. Unlike optical images, SAR images mainly contain texture, structure, and shape information, but not grayscale, color, or brightness information. (2) SAR and optical images have significant feature differences. For example, SAR features are highly sensitive to changes in viewpoint, parameters, etc., while optical features are the opposite. The distribution of optical and SAR image data differs significantly, and adapting knowledge learned from the optical domain to the SAR domain is a key issue. (3) Due to the huge differences between the SAR and optical domains, domain adaptation methods based on feature alignment typically lose some unique and valuable features of the SAR domain when shifting feature distributions to the labeled optical domain. Therefore, training a generalized detector to reduce the impact of the optical domain on SAR domain detection training is a challenge.
[0069] Since SAR training images are not easy to obtain, small-sample training is used when training the neural network. In step S100, the number of sample images in the target domain dataset, i.e., SAR images, is relatively small, while the number of sample images in the source domain dataset, i.e., optical images, is abundant. This effectively makes up for the problem of insufficient training data.
[0070] Meanwhile, due to the significant differences between the optical image domain and the SAR image domain, a domain adaptation problem exists. The core of solving this problem lies in how to better measure the distributional differences between the two domains and how to better learn and transfer domain knowledge. The key to the domain adaptation problem from optical to SAR images is the distributional difference between the two; how to reduce or measure the difference in distributional distance is crucial to solving the problem. Furthermore, existing methods do not consider the actual small sample size characteristic of SAR images, simply relying on domain knowledge transfer without utilizing the rich data of optical remote sensing images to assist in target detection under small sample conditions. Because the source domain optical images have a large amount of data, using the entire set of images from both the source and target domains to train the detector might seem intuitive. However, this direct data mixing training method is likely to introduce a lot of low-quality data that is unsuitable for the target domain, which will damage the model during training.
[0071] Furthermore, in response to the aforementioned domain problem, this embodiment proposes a method to eliminate effective source domain images, making the fusion of the remaining source domain images with the target images more diverse and reasonable. At the same time, it enables the neural network to adapt to the feature distribution of the target and the target more quickly and accurately during subsequent neural network training, thereby improving the target detection performance of the neural network.
[0072] To effectively mitigate the distributional differences between the visible light image data domain and the SAR image data domain, this method designs a data domain-aware adaptive enhancement strategy. It fuses and expands small-sample SAR ship target data with a sufficient amount of visible light ship target data. The source domain is assumed to be a dataset containing a sufficient number of targets. The target domain contains only a few samples in, It is the input image; Depend on and The target anchor frame coordinates and target category constitute the target domain D. s distributed With target domain D t distributed There are usually differences between them, which is due to factors such as different imaging mechanisms.
[0073] To maximize the extraction and utilization of effective information from the labeled targets in the source domain, and Extended to This method designs a domain awareness enhancement strategy to generate new data. This involves performing multi-strategy domain-aware adaptive enhancement on the source domain data at both the image level and anchor box level to approximate the distribution of the source and target domain data. For example... Figure 2 As shown, this illustrates image-level and anchor-box-level domain-aware adaptive enhancement.
[0074] Specifically, in step S110, the target domain dataset and the source domain dataset are divided to obtain multiple training datasets. Since the sample data in the source domain dataset is much larger than that in the target domain dataset, the source domain dataset can be directly divided to obtain the initial training dataset, and then all sample images from the target domain dataset are placed into each initial training dataset to obtain the training dataset.
[0075] Next, in step S120, harmful source domain sample images that are not conducive to feature learning are removed from the training dataset, which is to optimize the training dataset.
[0076] Specifically, the process involves calculating the feature distribution of sample images in each training dataset, and then filtering source domain sample images in the training dataset based on the distance between the feature distribution of target domain sample images and the feature distribution of source domain sample images. This includes: first, using a feature extraction network to extract features from sample images in the training dataset and calculating the feature distribution of each sample image; second, calculating the average feature distribution of the target domain for multiple target domain sample images in the training dataset; third, calculating the distribution distance between the feature distribution of each source domain sample image in the training dataset and the average feature distribution of the target domain; and finally, based on the calculated distribution distance and the filtering strategy, filtering out source domain sample images in the training dataset whose feature distribution is far from the average feature distribution of the target domain.
[0077] In this embodiment, by calculating the data distribution of the source domain and the data distribution of the target domain, source domain data that is closer to the target domain data when input into the network for training is selected, and source domain data that is far from the target domain data is discarded, because the mixed training of these data and target domain data will impair the training of the model.
[0078] In this embodiment, the source domain sample images in the training sample set are sorted in ascending order according to the calculated distribution distance. This way, the feature distribution of the source domain sample images that are ranked earlier is closer to that of the target domain sample images, which is more conducive to the training of the neural network. On the other hand, the source domain sample images that are ranked later are farther away from the target domain sample images, which is not conducive to training, and they are removed from the training sample set.
[0079] Specifically, assuming detector f θ =(g θ ,h θ Defined by a set of parameters θ, and derived by the feature extraction network g. θ and detection head h θ Composition, output D t middle middle The representation is as follows:
[0080]
[0081] express After feature extraction network g θ The output, express After feature extraction network g θ The output. Further, define the variable function dist. f To measure and The correlation distance between them, and the enhancement of the candidate set. Sort:
[0082]
[0083] In one embodiment, the maximum average difference can be used as a distribution distance metric:
[0084]
[0085] In formula (3), x and x' represent the feature distribution of the source domain sample image and the average feature distribution of the target domain, respectively.
[0086] In this embodiment, considering the huge distribution difference between optical images and SAR images, existing common distance measurement functions cannot effectively measure the distribution distance between the two. In order to effectively alleviate the distribution difference between the optical image data domain and the SAR image data domain, a more effective distribution distance measurement function suitable for the domain adaptation problem from optical images to SAR images is designed, starting from the measurement function.
[0087] Common optical and SAR images only represent amplitude information. However, as we know from holography and radar, in the complex domain, optical and SAR images also contain phase information; amplitude is only a part of the overall information. Especially in high-level feature spaces, measuring the distribution characteristics of phase information in optical and SAR images is even more necessary, and both exhibit a certain directionality. However, common Euclidean distance metrics or MMD distance metrics can only measure the difference in amplitude distribution between the optical and SAR image domains, not the difference in phase distribution. Cosine metrics can only measure the difference in phase direction distribution in high-level features between the two domains, not the difference in amplitude distribution. Therefore, effectively combining these two approaches is crucial. Inspired by complex functions and signal systems, a common formula for complex numbers is:
[0088] Z = A + Bi (4)
[0089] Its exponential form is expressed as:
[0090] Z = Re iθ (5)
[0091] In formulas (4) and (5), R is the modulus of Z, θ is the principal argument, A = Rcosθ, B = Rsinθ.
[0092] As can be seen from Euler's exponential form, R represents amplitude information, and therefore can be used to represent the amplitude distribution distance of the domain adaptation metric function. θ represents phase information, and can be expressed as a phase distribution distance metric. Therefore, the maximum mean difference (MMD) can be used as the amplitude component, R, and the cosine distance as the phase component, θ. The cosine distance metric function is shown below:
[0093] CSD(x,x')=cos(x,x') (6)
[0094] Here, x and x' represent two different distributions. Therefore, the distribution distance metric function proposed in this method is defined as dist. OSD From the above derivation and definition, it can be seen that dist OSD The formula is instantiated as follows:
[0095]
[0096] In this embodiment, the distribution distance metric function shown in formula (7) is used for calculation. This metric method not only considers the distance between domains in the high-dimensional feature space, but also takes into account the distribution direction between domains, thereby better aligning the relationships and knowledge between different data domains. It can be more efficient and more suitable for measuring the distribution differences between the source domain and the target domain. At the same time, it can improve the supervision of SAR target detection in heterogeneous image information fusion. Figure 3 As shown, the amplitude distance measurement and orientation angle measurement between points A and B in the high-level semantic features are displayed.
[0097] Furthermore, for filtering For the noise samples, a shrinkage rate k (0 < k ≤ 1) is designed to dynamically reduce the number of noise samples. Furthermore, based on... Construct a new optimization function φ opt To optimize The optimized extension domain is defined as follows. (i.e., screening strategy):
[0098]
[0099] In formula (8), This represents the filtered training dataset. These represent the sample images and their corresponding ground truth labels in the filtered training dataset. This indicates that the sample images are sorted in ascending order based on their distribution distance. This indicates that the perception enhancement strategy generates new data, φ opt Let n represent the new optimization function. a n b And k represents the number of samples before screening, the number of samples after screening, and the shrinkage rate, respectively.
[0100] Through φ opt n b (n b =[n a *k]) according to The increasing order starts from Select candidate To better reflect the distribution of the target domain
[0101] Next, in step S130, the dual-stream branch feature alignment extraction network is trained using multiple selected training datasets.
[0102] While source domain optical images offer a wealth of data, training the detector using a complete set of images from both the source and target domains might seem intuitive. Furthermore, using the aforementioned strategy of using domain knowledge distribution differences as inspiration and filtering candidate source domain optical image data can effectively alleviate the domain mismatch problem between optical and SAR images, eliminate harmful source domain images, and make the fusion of source and target domain images more diverse and reasonable. However, this only utilizes ample optical data from the single perspective of data augmentation, without considering the differences between the features extracted from optical and SAR images. If only source domain data is filtered before mixing and training, the uniqueness of SAR image features is not fully utilized.
[0103] To fully utilize the feature information of small sample SAR images in the target domain to guide domain knowledge learning of optical images in the source domain, a weight-sharing dual-stream branch feature alignment extraction network was constructed to better realize knowledge extraction and sharing between the two domains. A limited number of target domain instances were used to guide the extraction and transfer of domain knowledge from a sufficient number of source domain instances.
[0104] In this embodiment, in the dual-stream branch feature alignment extraction network, while training the network with a limited amount of target domain SAR data and sufficient source domain optical data, the limited target domain SAR data is also input into another feature extraction backbone identical to the mixed domain for feature extraction. The weights of the feature extraction backbones between the two domains are shared, mutually guiding learning and performing feature alignment. Finally, the corresponding loss is output for backpropagation and parameter updates, ultimately achieving the desired result. Figure 4 The domain adaptation results are shown in the following figures: Top left: Existing feature alignment-based methods cannot extract discriminative features from the limited labeled data in the target domain. Top right: Existing data augmentation-based methods cannot learn the discriminative features between the source and target domains. Bottom: The proposed method effectively combines the advantages of feature alignment and data augmentation methods, filtering source data with distributions significantly different from the target domain (left of the thick black dashed line), and learning and aligning the discriminative features between the two domains.
[0105] In this embodiment, the feature extraction network in step S120 and the feature extraction network in the dual-stream branch feature alignment extraction network have the same structure, and existing network structures with feature extraction functions can be used.
[0106] In step S140, the total target detection loss function includes the distribution distance loss function, the target confidence loss function for the target domain, and the bounding box loss function, as well as the target confidence loss function and the bounding box loss function for the source domain. Furthermore, weights are assigned to the target confidence loss function and the bounding box loss function, respectively.
[0107] Specifically, to more effectively utilize the differences between the two domains, this method proposes a loss branch, DKL, inspired by the differences in domain knowledge distribution. This branch leverages the distribution differences between the optical image domain and the SAR image domain to inspire the network to learn domain knowledge better. The original loss function is expressed as follows:
[0108]
[0109] In formula (9), This represents the class loss function of the model, which is mainly used to measure the difference between the predicted class and the actual class; The confidence loss function for the target object is used to measure the difference between the predicted target object and the actual target object. This represents the bounding box loss function, primarily used to measure the difference between the predicted and ground truth bounding boxes. Since this method only detects targets of the same category, the classification loss is 0 here, resulting in:
[0110]
[0111] In this embodiment, a distribution distance loss function is also calculated based on the distance between the feature distribution of the target domain sample images and the feature distribution of the source domain sample images in the training dataset. As mentioned above, the metric function dist is defined. f Measure each batch and The relevant distance between The distribution distance loss function is obtained by averaging the values and scaling them to between 0 and 1 using the sigmoid function.
[0112]
[0113] In formula (11), It is the activation function in neural networks, a smooth and strictly monotonic saturation function that is easy to differentiate. It is often used for the output of hidden layer neurons, and its value range is (0,1). Its function curve is shown below. Figure 5 As shown, due to the algorithm in All are values greater than 0, therefore The value range is (0.5, 1). Add this to the final loss and perform backpropagation to obtain:
[0114]
[0115] Furthermore, since the dual-stream branch feature alignment extraction network in this method extracts and aligns features from small sample SAR images, it increases the confidence and bounding box loss on the target domain SAR image. Therefore, the total target detection loss function at this time is:
[0116]
[0117] In formula (13), Let represent the target confidence loss function and the target bounding box loss function, respectively, in the target domain.
[0118] Furthermore, by assigning certain weights to the confidence scores and bounding box losses of the source and target domains, the total loss function is constructed as follows:
[0119]
[0120] In formula (14), F θ Let λ1 and λ2 represent the parameterized model, where λ1 and λ2 are the weight parameters between the confidence loss function and the target bounding box loss function for the two domains, respectively.
[0121] In this embodiment, the feature extraction network in step S120 is trained synchronously when the dual-stream branch feature alignment extraction network is trained.
[0122] In this embodiment, the SAR image detection method described herein is applied to the SAR image target detection task where the target is a ship.
[0123] like Figure 6 The diagram shows a process diagram from step S110 to step S140.
[0124] In step S150, a SAR image target detection network is constructed using the target domain feature extraction network and the detection head in the trained dual-stream branch feature alignment extraction network. The SAR target image to be detected is then input into the SAR image target detection network to achieve target detection.
[0125] The aforementioned SAR image target detection method based on heterogeneous images leverages the difference in knowledge distribution across domains. First, a loss branching mechanism is designed to effectively utilize SAR domain knowledge, derived from the observed differences in domain knowledge distribution. Second, a dual-stream branch feature alignment extraction network with weight sharing is employed. This network architecture enables better extraction and sharing of knowledge between the optical and SAR domains. Furthermore, this method develops an efficient, plug-and-play distance metric function that accurately measures the distance between features from the optical and SAR domains.
[0126] This method proposes a domain-adaptive small-sample SAR image ship detection algorithm inspired by the differences in domain knowledge distribution. It augments data by selecting optical remote sensing ship images similar to SAR target samples, thereby assisting in SAR image ship target detection under small-sample conditions. It breaks the limitation of traditional multi-source information fusion requiring paired data, achieving cross-domain learning for unpaired data and effectively improving SAR target detection performance.
[0127] To effectively utilize the differences between the two domains, this method designs a loss branch, DKL, inspired by the differences in domain knowledge distribution. It uses the distribution differences between the optical image domain and the SAR image domain to inspire the network to learn domain knowledge better.
[0128] This method aims to more effectively utilize the domain knowledge of SAR images under small sample conditions by constructing a weight-shared dual-stream branch instance-level feature alignment extraction network to better achieve knowledge extraction and sharing between the two domains.
[0129] To alleviate the difficulty of knowledge domain adaptation between optical and SAR image domains, this method proposes a novel distance metric function, OSD, for domain-adaptive target detection from optical to SAR images. It achieves more accurate distance measurement between the optical and SAR domains by constructing an efficient, plug-and-play function plugin.
[0130] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.
[0131] In one embodiment, such as Figure 7 As shown, a SAR image target detection device based on heterogeneous images is provided, comprising: a dataset acquisition module 200, a training dataset construction module 210, a training dataset optimization module 220, a neural network training module 230, a trained neural network acquisition module 240, and a SAR image target detection module 250, wherein:
[0132] The dataset acquisition module 200 is used to acquire source domain datasets and target domain datasets for the same type of target, wherein both the source domain datasets and the target domain datasets include multiple sample images;
[0133] The training dataset construction module 210 is used to divide the source domain dataset and the target domain dataset into multiple training datasets. Each training dataset includes a portion of the source domain sample images in the source domain dataset and all the target domain sample images in the target domain dataset.
[0134] The training dataset optimization module 220 is used to calculate the feature distribution of sample images in each training dataset, and to filter the source domain sample images in the training dataset according to the distance between the feature distribution of the target domain sample images and the feature distribution of the source domain sample images, so as to obtain the filtered training dataset.
[0135] The neural network training module 230 is used to train the dual-stream branch feature alignment extraction network using multiple selected training datasets and a target domain dataset. In the dual-stream branch feature alignment extraction network, a weight-shared hybrid domain feature extraction network and a target domain feature extraction network are used to extract features from the sample images in the selected training dataset and the target domain dataset, respectively, to obtain corresponding hybrid image features and target domain image features. The feature alignment network is used to align the hybrid image features and the target domain image features to obtain aligned features. The detection head detects the target in the target domain sample image based on the aligned features.
[0136] The trained neural network obtains module 240, which is used to optimize the adjustable parameters in the dual-stream branch feature alignment extraction network according to the total loss function of object detection until convergence, thereby obtaining the trained dual-stream branch feature alignment extraction network. The total loss function of object detection includes a distribution distance loss function, which is calculated based on the distance between the feature distribution of the target domain sample image and the feature distribution of the source domain sample image in the training dataset.
[0137] The SAR image target detection module 250 is used to extract the target domain feature extraction network, feature alignment network and detection head from the trained dual-stream branch feature alignment extraction network to construct a SAR image target detection network. The SAR image to be detected is input into the SAR image target detection network to detect the target in the SAR image to be detected.
[0138] Specific limitations regarding the SAR image target detection device based on heterogeneous images can be found in the limitations of the SAR image target detection method based on heterogeneous images mentioned above, and will not be repeated here. Each module in the aforementioned SAR image target detection device based on heterogeneous images can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.
[0139] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 8 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a SAR image target detection method based on heterogeneous images. The display screen can be a liquid crystal display (LCD) or an e-ink display. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.
[0140] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.
[0141] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:
[0142] Obtain source domain datasets and target domain datasets for the same type of target, wherein both the source domain datasets and target domain datasets include multiple sample images;
[0143] The source domain dataset and the target domain dataset are divided into multiple training datasets. Each training dataset includes a portion of the source domain sample images from the source domain dataset and all the target domain sample images from the target domain dataset.
[0144] Calculate the feature distribution of sample images in each of the training datasets, and filter the source domain sample images in the training datasets according to the distance between the feature distribution of the target domain sample images and the feature distribution of the source domain sample images to obtain the filtered training datasets;
[0145] The dual-stream branch feature alignment extraction network is trained using multiple selected training datasets and a target domain dataset. In the dual-stream branch feature alignment extraction network, a weight-shared hybrid domain feature extraction network and a target domain feature extraction network are used to extract features from sample images in the selected training dataset and target domain dataset, respectively, to obtain corresponding hybrid image features and target domain image features. The feature alignment network is used to align the hybrid image features and target domain image features to obtain aligned features. The detection head then detects targets in the target domain sample images based on the aligned features.
[0146] The adjustable parameters in the dual-stream branch feature alignment extraction network are optimized according to the total loss function of object detection until convergence, resulting in a trained dual-stream branch feature alignment extraction network. The total loss function of object detection includes a distribution distance loss function, which is calculated based on the distance between the feature distribution of the target domain sample image and the feature distribution of the source domain sample image in the training dataset.
[0147] The target domain feature extraction network, feature alignment network, and detection head are extracted from the trained dual-stream branch feature alignment extraction network to construct a SAR image target detection network. The SAR image to be detected is input into the SAR image target detection network to detect the target in the SAR image.
[0148] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:
[0149] Obtain source domain datasets and target domain datasets for the same type of target, wherein both the source domain datasets and target domain datasets include multiple sample images;
[0150] The source domain dataset and the target domain dataset are divided into multiple training datasets. Each training dataset includes a portion of the source domain sample images from the source domain dataset and all the target domain sample images from the target domain dataset.
[0151] Calculate the feature distribution of sample images in each of the training datasets, and filter the source domain sample images in the training datasets according to the distance between the feature distribution of the target domain sample images and the feature distribution of the source domain sample images to obtain the filtered training datasets;
[0152] The dual-stream branch feature alignment extraction network is trained using multiple selected training datasets and a target domain dataset. In the dual-stream branch feature alignment extraction network, a weight-shared hybrid domain feature extraction network and a target domain feature extraction network are used to extract features from sample images in the selected training dataset and target domain dataset, respectively, to obtain corresponding hybrid image features and target domain image features. The feature alignment network is used to align the hybrid image features and target domain image features to obtain aligned features. The detection head then detects targets in the target domain sample images based on the aligned features.
[0153] The adjustable parameters in the dual-stream branch feature alignment extraction network are optimized according to the total loss function of object detection until convergence, resulting in a trained dual-stream branch feature alignment extraction network. The total loss function of object detection includes a distribution distance loss function, which is calculated based on the distance between the feature distribution of the target domain sample images and the feature distribution of the source domain sample images in the training dataset.
[0154] The target domain feature extraction network, feature alignment network, and detection head are extracted from the trained dual-stream branch feature alignment extraction network to construct a SAR image target detection network. The SAR image to be detected is input into the SAR image target detection network to detect the target in the SAR image.
[0155] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
[0156] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0157] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.
Claims
1. A SAR image target detection method based on heterogeneous images, characterized in that, The method includes: Obtain source domain datasets and target domain datasets for the same type of target, wherein both the source domain datasets and target domain datasets include multiple sample images; The source domain dataset and the target domain dataset are divided into multiple training datasets. Each training dataset includes a portion of the source domain sample images from the source domain dataset and all the target domain sample images from the target domain dataset. A feature extraction network is used to extract features from sample images in the training dataset, and the feature distribution of each sample image is calculated. The average feature distribution of multiple target domain sample images in the training dataset is calculated, and the distribution distance between the feature distribution of each source domain sample image in the training dataset and the average feature distribution of the target domain is calculated. Based on the calculated distribution distance and a filtering strategy, source domain sample images in the training dataset whose feature distributions are far from the average feature distribution of the target domain are filtered out, resulting in a filtered training dataset. The distribution distance metric function used to calculate the distance between the feature distribution of each source domain sample image in the training dataset and the average feature distribution of the target domain is expressed as: In the above formula, , Let these represent the feature distribution of the source domain sample image and the average feature distribution of the target domain, respectively. Indicates the number of SAR image samples in the target domain; The dual-stream branch feature alignment extraction network is trained using multiple selected training datasets and a target domain dataset. In the dual-stream branch feature alignment extraction network, a weight-shared hybrid domain feature extraction network and a target domain feature extraction network are used to extract features from sample images in the selected training dataset and target domain dataset, respectively, to obtain corresponding hybrid image features and target domain image features. The feature alignment network is used to align the hybrid image features and target domain image features to obtain aligned features. The detection head then detects targets in the target domain sample images based on the aligned features. The adjustable parameters in the dual-stream branch feature alignment extraction network are optimized according to the total loss function of object detection until convergence, resulting in a trained dual-stream branch feature alignment extraction network. The total loss function of object detection includes a distribution distance loss function, which is calculated based on the distance between the feature distribution of the target domain sample image and the feature distribution of the source domain sample image in the training dataset. The target domain feature extraction network and detection head are extracted from the trained dual-stream branch feature alignment extraction network to construct a SAR image target detection network. The SAR image to be detected is input into the SAR image target detection network to detect the target in the SAR image.
2. The SAR image target detection method according to claim 1, characterized in that, The sample images in the source domain dataset are optical images, and the sample images in the target domain dataset are SAR images.
3. The SAR image target detection method according to claim 2, characterized in that, The filtering strategy is expressed as follows: In the above formula, This represents the filtered training dataset. , These represent the sample images and their corresponding ground truth labels in the filtered training dataset. This indicates that the sample images are sorted in ascending order based on their distribution distance. This indicates that the perception enhancement strategy generates new data. This represents the new optimization function. , as well as These represent the number of samples before screening, the number of samples after screening, and the shrinkage rate, respectively.
4. The SAR image target detection method according to claim 3, characterized in that, The total loss function for target detection includes the distribution distance loss function, the target confidence loss function for the target domain, and the bounding box loss function, as well as the target confidence loss function and the bounding box loss function for the source domain. Furthermore, weights are set for the target confidence loss function and the bounding box loss function, respectively.
5. The SAR image target detection method according to any one of claims 1-4, characterized in that, The SAR image detection method is applied to SAR image target detection tasks where the target is a ship.
6. A SAR image target detection device based on heterogeneous images, characterized in that, The apparatus performs the SAR image target detection method based on heterogeneous images as described in any one of claims 1-5, including: The dataset acquisition module is used to acquire source domain datasets and target domain datasets for the same type of target, wherein both the source domain datasets and the target domain datasets include multiple sample images; The training dataset construction module is used to divide the source domain dataset and the target domain dataset into multiple training datasets. Each training dataset includes a portion of the source domain sample images in the source domain dataset and all the target domain sample images in the target domain dataset. The training dataset optimization module is used to calculate the feature distribution of sample images in each training dataset, and to filter the source domain sample images in the training dataset according to the distance between the feature distribution of the target domain sample images and the feature distribution of the source domain sample images, so as to obtain the filtered training dataset. The neural network training module is used to train the dual-stream branch feature alignment extraction network using multiple selected training datasets and a target domain dataset. In the dual-stream branch feature alignment extraction network, a weight-shared hybrid domain feature extraction network and a target domain feature extraction network are used to extract features from the sample images in the selected training dataset and the target domain dataset, respectively, to obtain corresponding hybrid image features and target domain image features. The feature alignment network is used to align the hybrid image features and the target domain image features to obtain aligned features. The detection head then detects the target in the target domain sample image based on the aligned features. The trained neural network module is used to optimize the adjustable parameters in the dual-stream branch feature alignment extraction network according to the total loss function of object detection until convergence, thereby obtaining the trained dual-stream branch feature alignment extraction network. The total loss function of object detection includes a distribution distance loss function, which is calculated based on the distance between the feature distribution of the target domain sample image and the feature distribution of the source domain sample image in the training dataset. The SAR image target detection module is used to extract the target domain feature extraction network and the detection head from the trained dual-stream branch feature alignment extraction network to construct a SAR image target detection network. The SAR image to be detected is input into the SAR image target detection network to detect the target in the SAR image.
7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.
8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.
