SAR image out-of-domain data detection method and device based on dynamic multiple prototypes
Through the dynamic multi-prototype SAR image extra-domain data detection method, using clustering and prototype feature learning, combined with cross entropy and dynamic prototype distance loss function, the problem of difficulty in identifying extra-domain data in open world scenarios in the existing technology is solved. , realizes accurate identification of intra-domain and out-domain data, and improves detection performance and reliability.
Patent Information
- Application Number
- CN202411169204.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-23
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2044-08-23
AI Technical Summary
Existing SAR image target recognition models are difficult to effectively identify extradomain data in open world scenarios, resulting in new categories or data distribution changes that may not have been seen in practice, affecting detection performance.
Using the SAR image extra-domain data detection method based on dynamic multi-prototypes, the prototype feature set of known target categories is obtained through clustering, and the prototype feature learning is performed using the target recognition network, combining the cross entropy loss function and the dynamic prototype distance loss function to the network parameters Update the prototype features to achieve accurate identification of intra-domain and out-domain data.
It improves the efficiency and performance of SAR image extradomain data detection, can accurately identify in-domain targets and reject out-domain data, and enhances the reliability and security of detection.
Smart Images

Figure CN119048910A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of SAR image target recognition, and in particular to a method and device for detecting out-of-domain data of SAR images based on dynamic multiple prototypes. Background Art
[0002] Synthetic Aperture Radar (SAR) has all-weather and all-time observation capabilities and is widely used in agriculture, industry and other fields. With the improvement of SAR imaging resolution, Automatic Target Recognition (ATR) has become the most important technology in SAR image processing in the fields of airport management, military reconnaissance and maritime surveillance.
[0003] Existing deep learning models are all based on the independent and identically distributed (iid) assumption, which means that the training data and test data come from the same distribution. However, this assumption may not hold true in practice, for example, new categories that have never been seen before may appear in the test data or the data distribution may change due to changes in imaging conditions.
[0004] Therefore, it is crucial to achieve out-of-distribution (OOD) data detection for SAR targets in open-world scenarios, which can reliably identify in-distribution (ID) categories and reject OOD data. Summary of the invention
[0005] Based on this, it is necessary to provide a SAR image out-of-domain data detection method and device based on dynamic multi-prototypes, which can accurately identify in-domain targets and out-of-domain data at the same time, in order to solve the above technical problems.
[0006] A method for detecting out-of-domain data of SAR images based on dynamic multiple prototypes, the method comprising:
[0007] Acquire a training data set, wherein the training data set includes a plurality of SAR sample images related to known target categories and corresponding true value labels;
[0008] According to the features of all SAR sample images in the training data set, a prototype feature set of known target categories is obtained by clustering, wherein each known target category in the prototype feature set corresponds to at least one prototype feature;
[0009] The target recognition network is trained using the training data set and prototype feature learning is realized. The target recognition network comprises a feature extraction unit, an adaptive prototype learning unit and a classification unit in the training stage. The feature extraction unit extracts sample features of the SAR sample image. The adaptive prototype learning unit updates the prototype features according to the sample features and the relationship between the sample features and the nearest prototype features of the same type in the prototype feature set. The classification unit performs prediction according to the sample features to obtain a target category prediction result.
[0010] A joint prototype cross entropy loss function is calculated according to the target category prediction result, the prototype feature and the corresponding true value label, and the loss function includes a dynamic prototype distance loss function and a cross entropy loss function. The parameters in the feature extraction unit and the classification unit are iteratively updated according to the joint prototype cross entropy loss function, and the prototype features in the adaptive prototype learning unit are iteratively updated according to the dynamic prototype distance loss function until convergence, so as to obtain a trained target recognition network and a learned prototype feature, wherein the initialization prototype of the adaptive prototype learning unit is the mean of the corresponding cluster of the cluster;
[0011] A SAR image to be classified is obtained, and the SAR image is input into a trained target recognition network feature extraction unit, and an out-of-domain data detection result is obtained according to the distance between the sample feature and the learned prototype feature. The out-of-domain data detection result is out-of-domain data, or in-domain data. If it is in-domain data, a target recognition result with category probability is obtained through the classification unit.
[0012] In one embodiment, according to the features of all SAR sample images in the training data set, a prototype feature set of a known target category is obtained by clustering, including:
[0013] According to a preset number of prototypes, clustering is performed based on the feature data set to obtain a plurality of clusters, and the cluster center of each cluster is used as a prototype feature;
[0014] According to the obtained multiple prototype features, the prototype feature set of the known target category is constructed.
[0015] In one embodiment, clustering based on the feature data set adopts any one of the following methods: K-Means method, hierarchical clustering method, DNSCAN clustering method, mean shift clustering method or spectral clustering method.
[0016] In one embodiment, clustering the feature data set to obtain a plurality of clusters according to a preset number of prototypes includes:
[0017] Calculate according to the feature data set to obtain a corresponding distance matrix;
[0018] Perform hierarchical clustering according to the distance matrix to obtain a dendrogram;
[0019] The dendrogram is cut to obtain a plurality of clusters.
[0020] In one embodiment, a pre-trained feature extraction neural network is used to perform feature extraction on all SAR sample images in the training data set.
[0021] In one embodiment, after the SAR image is input into the feature extraction unit of the trained target recognition network:
[0022] According to the distance between the image feature of the SAR image and each of the prototype features in the updated prototype feature set, it is determined whether the SAR image is out-of-domain data, and if it is out-of-domain data, it is used as a detection result;
[0023] If the SAR image is in-domain data, the classification unit is used to perform classification according to the image features to obtain a target recognition result with category probability.
[0024] In one embodiment, the cross entropy loss function is calculated based on the target category prediction result and the corresponding true value label;
[0025] The dynamic prototype distance loss function is calculated based on the sample features and the prototype features.
[0026] In one embodiment, the joint prototype cross entropy loss function is expressed as:
[0027]
[0028] In the above formula, Represents the cross entropy loss function, where p i Represents the SAR sample image x i The target category prediction result, f(·) represents the feature extraction part of the neural network, and f(x i ) represents the features of the sample obtained after feature extraction, y i represents the corresponding true value label, N represents the number of samples in the training data set, represents the dynamic prototype distance loss function, μ j Represents the SAR sample image x i The corresponding nearest prototype feature in the class, λ represents the hyperparameter, Represents the label y i The corresponding prototype feature set.
[0029] The present application also provides a SAR image out-of-domain data detection device based on dynamic multiple prototypes, the device comprising:
[0030] A training data set acquisition module is used to acquire a training data set, wherein the training data set includes a plurality of SAR sample images related to known target categories and corresponding true value labels;
[0031] A prototype feature set obtaining module, used to obtain a prototype feature set of known target categories by clustering according to the features of all SAR sample images in the training data set, wherein each known target category in the prototype feature set corresponds to at least one prototype feature;
[0032] A target recognition network training module is used to train the target recognition network using the training data set and realize prototype feature learning. The target recognition network includes a feature extraction unit, an adaptive prototype learning unit and a classification unit in the training stage. The feature extraction unit extracts sample features of the SAR sample image. The adaptive prototype learning unit updates the prototype features according to the sample features and the relationship between the sample features and the nearest prototype features of the same type in the prototype feature set. The classification unit predicts according to the sample features to obtain a target category prediction result.
[0033] A network parameter and prototype feature update module, used to calculate a joint prototype cross entropy loss function according to the target category prediction result, the prototype feature and the corresponding true value label, the loss function includes a dynamic prototype distance loss function and a cross entropy loss function, iteratively update the parameters in the feature extraction unit and the classification unit according to the joint prototype cross entropy loss function, iteratively update the prototype features in the adaptive prototype learning unit according to the dynamic prototype distance loss function until convergence, and obtain a trained target recognition network and a learned prototype feature, wherein the initialization prototype of the adaptive prototype learning unit is the mean of the corresponding cluster of the cluster;
[0034] The SAR image out-of-domain data detection module is used to obtain a SAR image to be classified as a target, input the SAR image into a trained target recognition network feature extraction unit, and obtain an out-of-domain data detection result based on the distance between the sample feature and the learned prototype feature. The out-of-domain data detection result is out-of-domain data or in-domain data. If it is in-domain data, a target recognition result with category probability is obtained through the classification unit.
[0035] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:
[0036] Acquire a training data set, wherein the training data set includes a plurality of SAR sample images related to known target categories and corresponding true value labels;
[0037] According to the features of all SAR sample images in the training data set, a prototype feature set of known target categories is obtained by clustering, wherein each known target category in the prototype feature set corresponds to at least one prototype feature;
[0038] The target recognition network is trained using the training data set and prototype feature learning is realized. The target recognition network comprises a feature extraction unit, an adaptive prototype learning unit and a classification unit in the training stage. The feature extraction unit extracts sample features of the SAR sample image. The adaptive prototype learning unit updates the prototype features according to the sample features and the relationship between the sample features and the nearest prototype features of the same type in the prototype feature set. The classification unit performs prediction according to the sample features to obtain a target category prediction result.
[0039] A joint prototype cross entropy loss function is calculated according to the target category prediction result, the prototype feature and the corresponding true value label, and the loss function includes a dynamic prototype distance loss function and a cross entropy loss function. The parameters in the feature extraction unit and the classification unit are iteratively updated according to the joint prototype cross entropy loss function, and the prototype features in the adaptive prototype learning unit are iteratively updated according to the dynamic prototype distance loss function until convergence, so as to obtain a trained target recognition network and a learned prototype feature, wherein the initialization prototype of the adaptive prototype learning unit is the mean of the corresponding cluster of the cluster;
[0040] A SAR image to be classified is obtained, and the SAR image is input into a trained target recognition network feature extraction unit, and an out-of-domain data detection result is obtained according to the distance between the sample feature and the learned prototype feature. The out-of-domain data detection result is out-of-domain data, or in-domain data. If it is in-domain data, a target recognition result with category probability is obtained through the classification unit.
[0041] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:
[0042] Acquire a training data set, wherein the training data set includes a plurality of SAR sample images related to known target categories and corresponding true value labels;
[0043] According to the features of all SAR sample images in the training data set, a prototype feature set of known target categories is obtained by clustering, wherein each known target category in the prototype feature set corresponds to at least one prototype feature;
[0044] The target recognition network is trained using the training data set and prototype feature learning is realized. The target recognition network comprises a feature extraction unit, an adaptive prototype learning unit and a classification unit in the training stage. The feature extraction unit extracts sample features of the SAR sample image. The adaptive prototype learning unit updates the prototype features according to the sample features and the relationship between the sample features and the nearest prototype features of the same type in the prototype feature set. The classification unit performs prediction according to the sample features to obtain a target category prediction result.
[0045] A joint prototype cross entropy loss function is calculated according to the target category prediction result, the prototype feature and the corresponding true value label, and the loss function includes a dynamic prototype distance loss function and a cross entropy loss function. The parameters in the feature extraction unit and the classification unit are iteratively updated according to the joint prototype cross entropy loss function, and the prototype features in the adaptive prototype learning unit are iteratively updated according to the dynamic prototype distance loss function until convergence, so as to obtain a trained target recognition network and a learned prototype feature, wherein the initialization prototype of the adaptive prototype learning unit is the mean of the corresponding cluster of the cluster;
[0046] A SAR image to be classified is obtained, and the SAR image is input into a trained target recognition network feature extraction unit, and an out-of-domain data detection result is obtained according to the distance between the sample feature and the learned prototype feature. The out-of-domain data detection result is out-of-domain data, or in-domain data. If it is in-domain data, a target recognition result with category probability is obtained through the classification unit.
[0047] The above-mentioned out-of-domain data detection method and device for SAR images based on dynamic multiple prototypes, according to the features of all SAR sample images in the training data set, adopts clustering means to obtain a prototype feature set of known target categories, each known target category in the prototype feature set corresponds to one prototype feature or multiple prototype features, and uses the training data set to train the target recognition network. The target recognition network includes a feature extraction unit, an adaptive prototype learning unit and a classification unit in the training stage, and realizes prototype feature learning while completing the training of the feature extraction unit and the classification unit. The parameters in the feature extraction unit and the classification unit are iteratively updated according to the joint prototype cross entropy loss function, and the prototype features in the adaptive prototype learning unit are iteratively updated according to the dynamic prototype distance loss function until convergence, so as to obtain the trained target recognition network and the learned prototype features, wherein the initialization prototype of the adaptive prototype learning unit is the mean of the cluster corresponding to the clustering, and finally, the SAR image to be classified is input into the trained target recognition network to obtain the out-of-domain data detection result, which is the out-of-domain data or the in-domain data. If it is the in-domain data, the target recognition result with the category probability is obtained according to the classification network. This method can accurately identify targets within the domain and also accurately identify data outside the domain. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] Figure 1 1 is a flow chart of a method for detecting out-of-domain data of a SAR image based on dynamic multiple prototypes in one embodiment;
[0049] Figure 2 A schematic diagram of the motivation of an out-of-domain data detection method based on dynamic multiple prototypes in one embodiment;
[0050] Figure 3 A schematic diagram of a tag offset setting for SAR OOD detection in one embodiment;
[0051] Figure 4 Schematic diagram of the overall framework of out-of-domain data detection of SAR images based on dynamic multiple prototypes in one embodiment;
[0052] Figure 5 It is a structural block diagram of a SAR image out-of-domain data detection device based on dynamic multiple prototypes in one embodiment;
[0053] Figure 6 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION
[0054] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0055] In view of the problem that the prior art has low efficiency and poor detection performance when performing out-of-domain data detection on SAR images, especially for out-of-domain data detection, in this application, Figure 1 As shown, a method for detecting out-of-domain data of SAR images based on dynamic multiple prototypes is provided, comprising the following steps:
[0056] Step S100: obtaining a training data set, where the training data set includes a plurality of SAR sample images related to known target categories and corresponding true value labels.
[0057] Step S110, according to the features of all SAR sample images in the training data set, a prototype feature set of known target categories is obtained by clustering, and each known target category in the prototype feature set corresponds to at least one prototype feature.
[0058] Step S120, using the training data set to train the target recognition network and realize prototype feature learning. The target recognition network includes a feature extraction unit, an adaptive prototype learning unit and a classification unit in the training stage. The feature extraction unit extracts sample features of the SAR sample image, and the adaptive prototype learning unit updates the prototype features according to the sample features and the relationship between the sample features and the nearest prototype features of the same type in the prototype feature set. The classification unit makes predictions based on the sample features to obtain target category prediction results.
[0059] Step S130, a joint prototype cross entropy loss function is calculated based on the target category prediction result, the prototype feature and the corresponding true value label, and the loss function includes a dynamic prototype distance loss function and a cross entropy loss function. The parameters in the feature extraction unit and the classification unit are iteratively updated according to the joint prototype cross entropy loss function, and the prototype features in the adaptive prototype learning unit are iteratively updated according to the dynamic prototype distance loss function until convergence, so as to obtain a trained target recognition network and learned prototype features, wherein the initialization prototype of the adaptive prototype learning unit is the mean of the corresponding cluster of the clustering.
[0060] Step S140, obtaining a SAR image for target classification, inputting the SAR image into a trained target recognition network feature extraction unit, obtaining an out-of-domain data detection result based on the distance between the sample feature and the learned prototype feature, the out-of-domain data detection result is out-of-domain data, or in-domain data, if it is in-domain data, the target recognition result with category probability is obtained through the classification unit.
[0061] In this embodiment, the focus is on detecting unknown targets in SAR images, that is, out-of-domain data. Since in SAR target classification, generally only known categories can be identified, that is, target category features that have appeared when the algorithm is trained, however, in practice, unknown categories often appear. At this time, the existing target recognition method is used to identify unknown targets as known categories, thereby making wrong decisions. At the same time, when encountering unknown category targets, no clear warning can be given, that is, the target in the SAR image does not belong to any category of known targets. This leads to the failure to ensure its reliability and safety in many scenarios that require warnings. Therefore, the out-of-domain data detection proposed in this application is actually a process step before target recognition, which aims to detect out-of-domain data. When the detection result is out-of-domain data, subsequent target recognition is rejected. When the detection result is in-domain data, the target category recognition work is further performed.
[0062] Among the existing target detection methods, there are generally ex post methods and training methods. Among them, the ex post method aims to design an OOD (out-of-distribution) scoring function to distinguish ID (In-distribution) and OOD data. The ex post method is easy to use, for example, there is no need to modify the model. However, the OOD detection performance of the ex post method is limited by the model. Therefore, a series of training methods modify the loss function to obtain a model that can better distinguish ID and OOD data.
[0063] Among the existing SAR image target detection methods, the out-of-domain data detection method of optical images is often used in the target recognition task of SAR images. However, due to the differences between SAR images and optical images, this type of algorithm cannot achieve good recognition results on SAR images. In view of this point, re-examining the differences between optical and SAR images shows that SAR images reflect the electromagnetic scattering characteristics of the target and have a low signal-to-noise ratio. Given the characteristics of SAR images, the inter-class differences are small and the intra-class variations are large. Using a single prototype to represent each category significantly limits the representation ability of the model. However, this simplified method of using a single prototype to represent each category fails to take into account a large amount of intra-class variations, resulting in the existing detection algorithm being difficult to distinguish out-of-domain data from in-domain data, that is, known target categories and unknown target categories.
[0064] In this embodiment, a dynamic multi-prototype SAR target detection method is proposed, where dynamic means that the number of prototypes for each category is not fixed, but is determined by the characteristics of each target category, such as Figure 2 shown.
[0065] In this embodiment, clustering is first performed based on the image features of multiple known target categories, and the cluster center is used as the initial prototype feature. Then, a training method is used to make the prototype feature more accurate, so that the detection of out-of-domain data can be accurately realized. At the same time, when the detection result is in-domain data, the classification unit is used to classify the target category based on the image features.
[0066] In step S100, first, the prototype features of each known target category are obtained through the training data set. The training data set includes SAR sample images of multiple different target categories, and each target category includes multiple SAR sample images. The target category appearing in the training data set is the known target category. When in practice, the target category in the SAR image to be detected does not belong to any known target category, it is an unknown target category, that is, out-of-domain data. At the same time, in order to train the target recognition network later, the training data set also includes the true value label corresponding to each SAR sample image.
[0067] Usually, when training a deep neural network, it generally includes two stages, namely the training stage and the testing stage. The above training data set is used in the training stage. For the K-class classification task, the training data set D train ={(x i ,y i )} According to the joint data distribution P xy Independent and identically distributed extraction, where x i ∈x represents input data, x represents input space, y i ∈y represents the sample label, i.e. the true value of the label, and y=1,2,...K represents the label space. OOD means P xy ≠P x′y′ , where P x′y′ Represents the distribution of the test data set, that is, the set of SAR sample data used for testing in the test phase. Therefore, OOD detection can be divided into label shift and covariate shift, where label shift means that the test category does not overlap with the training category.
[0068] Since this method focuses on the detection of SAR out-of-domain data, in this embodiment, the target categories in the test data set and the training data set are set by label shifting, that is, the target categories in the two data sets do not overlap, such as Figure 3 shown.
[0069] In this example, the ID data is only accessible during training. For OOD testing, it can be viewed as a binary classifier that uses level set estimation to decide on the input sample x i Belongs to ID or OOD, as shown below:
[0070]
[0071] In the above formula, λ represents the threshold and S(x) represents the score.
[0072] In step S110, based on the features of all SAR sample images in the training data set, clustering is used to obtain a prototype feature set of the known target category, including: according to a preset number of prototypes, clustering is performed based on the feature data set to obtain multiple clusters, and the cluster center of each cluster is used as the prototype feature, and based on the obtained multiple prototype features, a prototype feature set of the known target category is constructed.
[0073] In this embodiment, a pre-trained feature extraction neural network is used to extract features from all SAR sample images in the training data set. The feature extraction neural network may use a neural network encoder.
[0074] It is understood that the feature extraction unit in the target recognition network can use a pre-trained neural network encoder. In one embodiment, the pre-trained feature extraction neural network can also use the feature extraction unit in the target recognition network.
[0075] In this embodiment, the means of clustering based on the feature data set may include K-Means method, hierarchical clustering method, DNSCAN clustering method, mean shift clustering method or spectral clustering method, any of which may be used to implement clustering. It is understandable that other clustering means may also be used.
[0076] Furthermore, according to a preset number of prototypes, clustering based on the feature data set to obtain multiple clusters includes: calculating according to the feature data set to obtain a corresponding distance matrix, performing hierarchical clustering according to the distance matrix to obtain a dendrogram, and cutting the dendrogram to obtain multiple clusters.
[0077] In this embodiment, since only the total number of prototypes is given and the prototypes of each target type are not specified, multiple clusters will be obtained dynamically after feature clustering. At this time, one or more corresponding clusters will be obtained according to the inter-class distinction of each target category, and the center of each cluster represents the corresponding prototype feature. At this time, the number of prototype features corresponding to each known target category is at least one.
[0078] In this embodiment, the cluster center of each cluster is calculated using the following formula:
[0079]
[0080] In the above formula, μ i represents the cluster center of the i-th cluster, C i represents the i-th cluster, z jRepresents the jth feature of the feature dataset.
[0081] Different from the idea of using the distance between training samples and test samples to detect out-of-domain data in the prior art, this method uses the distance between the test sample, i.e., the sample to be detected, and each prototype feature to detect out-of-domain data, which makes this method more efficient than the prior art. At the same time, it can be seen from the cluster center calculation method shown above that the prototype feature is the average value of the clustering cluster, which makes it more stable.
[0082] After obtaining prototype features of multiple known target categories through training data sets, they can be applied in practice to detect targets in SAR images. It should be noted here that one target category does not correspond to one prototype, but to at least one prototype, which depends on the intra-class distinction of the target.
[0083] In addition, in the cluster-based prototype initialization phase, only the total number of clusters, i.e., prototype features, is set in this method, without specifying the number of prototypes for each object class. In other words, the number of prototypes for each class is completely determined by the data characteristics, which is the origin of the "dynamic" one.
[0084] From the above, it can be seen that the representation of the prototype will greatly affect the performance of SAR OOD detection. Therefore, in order to obtain accurate category prototypes, in this method, an end-to-end dynamic multi-prototype training framework is proposed, such as Figure 4 As shown in the figure. Compared with the traditional network, the difference of DMPL lies in the adaptive prototype learning (APL) unit. APL is a plug-and-play module that automatically learns prototypes under the supervision of Dynamic Prototype Distance Loss (DPD). In addition, in order to learn the strong separation features between ID and OOD and achieve ID classification, a loss function combining cross entropy and DPD is further proposed.
[0085] In this embodiment, the prototype features obtained by clustering are used as the initial prototype in the framework, where the number of prototypes for each class is dynamic and determined by the data characteristics. Then, this method proposes an adaptive prototype learning unit, which includes a dynamic prototype distance loss (DPD) to encourage sample embedding around the nearest prototype within its class. At the same time, the cross entropy and DPD losses are combined to obtain the loss function corresponding to the entire network (target recognition network), namely the joint prototype cross entropy loss (JointPrototype-Cross-Entropy Loss), which is used to update the parameters in the feature extraction unit and the classification unit.
[0086] In this embodiment, during the training phase, the target recognition network includes a feature extraction unit, an adaptive prototype learning unit, and a classification unit connected in sequence to complete parameter training of the feature extraction unit and the classification unit, and simultaneously use the adaptive prototype learning unit to learn prototype features. During the testing and practice phases, the trained target recognition network includes a feature extraction unit and a classification unit.
[0087] Furthermore, the joint prototype cross entropy loss function includes a dynamic prototype distance loss function and a cross entropy loss function, wherein the prototype distance loss function is used to update the prototype vector in the adaptive prototype learning unit, and the joint prototype cross entropy loss function is used to update the parameters in the feature extraction unit and the classification unit.
[0088] Specifically, the adaptive prototype learning unit mainly realizes the automatic learning of prototypes, aiming to make the features of each prototype more accurate through automatic learning, in which the initialization of the prototype determines the number of prototypes in each category. To this end, a loss function DPD for adaptive prototype learning is designed. The design of DPD loss is based on the following assumptions: the high aggregation of intra-class data can improve the performance of distance-based out-of-domain data detection. This strategy stems from the large intra-class differences of SAR images, and the traditional single prototype cannot fully capture the diversity within the class. Although the clustering method can realize prototype learning, its performance is limited by the feature extractor. Therefore, the DPD strategy aims to more accurately capture the intra-class differences of each class by dynamically adjusting the prototype feature vector of each category.
[0089] Furthermore, the loss function DPD used for adaptive prototype learning requires that the sample is close to the nearest prototype within the class. Therefore, the distance between the sample and the prototype is calculated as the loss function, which is expressed as:
[0090]
[0091] In the above formula, P represents the prototype feature set, μ j Represents the SAR sample image x i The corresponding prototype feature, y i Represents the corresponding true value label.
[0092] In this embodiment, in order to realize dynamic adjustment of the prototype, the prototype is regarded as a network parameter and the optimizer is combined to realize parameter update. Based on the DPD loss function and combined with the stochastic gradient descent (SGD) optimizer, the automatic update of the prototype parameters is realized. Assume that the loss function L(P, X, Y) is about the prototype The gradient of The update formula of is as follows.
[0093]
[0094] In the above formula, η is a non-negative number, which represents the learning rate. The larger η is, the faster the prototype is updated.
[0095] Furthermore, in order to realize ID classification and prototype learning simultaneously in the end-to-end network, the joint cross entropy loss function and DPD are implemented as a joint prototype cross entropy loss (Joint Prototype-Cross-Entropy Loss, JPCL), which is expressed as:
[0096]
[0097] In the above formula, represents the cross entropy loss function, where p i Represents the SAR sample image x i The target category prediction result, f(·) represents the feature extraction part of the neural network, and f(x i ) represents the features of the sample obtained after feature extraction, y i represents the corresponding true value label, N represents the number of samples in the training data set, represents the dynamic prototype distance loss function, μ j Represents the SAR sample image x i The corresponding nearest prototype feature in the class, λ represents the hyperparameter, Represents the label y i The corresponding prototype feature set.
[0098] In this embodiment, prototype learning and classification accuracy are balanced by adjusting hyperparameters. The prototype distance and cross entropy loss functions are combined, and the prototype cross entropy loss effectively guides the model to improve its prediction accuracy for data categories while learning prototypes. This dual optimization strategy ensures that the model can not only learn features that distinguish different categories, but also have better adaptability to variations within each category.
[0099] The prototype distance part prompts the model to accurately locate the prototype features of each category, thereby minimizing the distance between the data points in the class and their corresponding prototypes. This helps capture the core features within the class, allowing the model to have better generalization capabilities for samples with high similarity within the class. The cross entropy part is directly related to the output of the model. By penalizing incorrect classification predictions, it promotes the model's predicted probability distribution in each category to be more consistent with the true label.
[0100] In addition, the combination of the two uses a hyperparameter λ to balance the importance of prototype learning and classification accuracy. In practical applications, adjusting λ can optimize model performance according to the needs of specific tasks. For example, in some tasks that require more refined intra-class differences, the weight of the prototype distance can be increased; and in scenarios where classification accuracy needs to be emphasized, the influence of cross entropy can be enhanced.
[0101] Finally, JPCL provides a flexible and powerful method to simultaneously optimize the model's feature expression ability and classification accuracy. This method can significantly improve the model's performance on complex datasets, especially in application scenarios with large intra-class differences, such as SAR out-of-domain data detection or SAR ATR.
[0102] In step S140, after the feature extraction unit extracts the features of the SAR image, it is determined whether the SAR image is out-of-domain data based on the distance between the image features of the SAR image and each prototype feature in the updated prototype feature set. If it is out-of-domain data, it is output as a detection result and the target category is no longer identified. If the SAR image is in-domain data, the classification unit classifies it according to the image features to obtain a target recognition result with a category probability.
[0103] Specifically, when performing out-of-domain data detection on a SAR image, the distance between the image features of the SAR image and all updated prototype features is calculated, and the relationship between the nearest distance and a preset threshold is determined. If the nearest distance is greater than the preset threshold, the SAR image is out-of-domain data; if the nearest distance is less than the preset threshold, the SAR image is in-domain data.
[0104] In the above-mentioned out-of-domain data detection method of SAR images based on dynamic multi-prototypes, according to the features of all SAR sample images in the training data set, a prototype feature set of known target categories is obtained by clustering, each known target category in the prototype feature set corresponds to at least one prototype feature, and the target recognition network is trained using the training data set. In the training stage, the target recognition network includes a feature extraction unit, an adaptive prototype learning unit and a classification unit. The feature extraction unit extracts sample features of the SAR sample image, and the adaptive prototype learning unit adaptively updates the prototype features based on the relationship between the prototype feature set and the sample features. The classification unit predicts according to the sample features to obtain the target category prediction result, and the cross entropy loss function is calculated according to the target category prediction result and the corresponding true value label. The dynamic prototype distance loss function is calculated according to the sample features and the prototype features. The prototype vector in the adaptive prototype learning unit is updated by the dynamic prototype distance loss function, and the parameters in the feature extraction unit and the classification unit are updated by the joint prototype cross entropy loss function, that is, the cross entropy loss function and the dynamic prototype distance loss function, until convergence, and the trained target recognition network is obtained, wherein the adaptive prototype learning unit updates each prototype feature in the prototype feature set as the initial network parameter, and finally, the SAR image to be classified is input into the trained target recognition network. At this time, the trained target recognition network includes a feature extraction unit and a classification unit. After the feature extraction unit extracts the sample feature, the updated prototype feature and the sample feature are used to obtain the out-of-domain data detection result. The out-of-domain data detection result is out-of-domain data, or in-domain data. If it is in-domain data, the target classification result with category probability is obtained. The method can accurately identify in-domain targets and out-of-domain data at the same time.
[0105] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.
[0106] In one embodiment, Figure 5As shown, a SAR image out-of-domain data detection device based on dynamic multiple prototypes is provided, comprising: a training data set acquisition module 200, a prototype feature set acquisition module 210, a target recognition network training module 220, a network parameter and prototype feature update module 230 and a SAR image out-of-domain data detection module 240, wherein:
[0107] A training data set acquisition module 200 is used to acquire a training data set, wherein the training data set includes a plurality of SAR sample images related to known target categories and corresponding true value labels;
[0108] A prototype feature set obtaining module 210 is used to obtain a prototype feature set of known target categories by clustering according to the features of all SAR sample images in the training data set, wherein each known target category in the prototype feature set corresponds to at least one prototype feature;
[0109] The target recognition network training module 220 is used to train the target recognition network using the training data set and realize prototype feature learning. The target recognition network includes a feature extraction unit, an adaptive prototype learning unit and a classification unit in the training stage. The feature extraction unit extracts the sample features of the SAR sample image. The adaptive prototype learning unit updates the prototype features according to the sample features and the relationship between the sample features and the nearest prototype features of the same type in the prototype feature set. The classification unit predicts according to the sample features to obtain a target category prediction result.
[0110] The network parameter and prototype feature update module 230 is used to calculate the joint prototype cross entropy loss function according to the target category prediction result, the prototype feature and the corresponding true value label, and the loss function includes a dynamic prototype distance loss function and a cross entropy loss function. The parameters in the feature extraction unit and the classification unit are iteratively updated according to the joint prototype cross entropy loss function, and the prototype features in the adaptive prototype learning unit are iteratively updated according to the dynamic prototype distance loss function until convergence, so as to obtain the trained target recognition network and the learned prototype features, wherein the initialization prototype of the adaptive prototype learning unit is the mean of the corresponding cluster of the cluster;
[0111] The SAR image out-of-domain data detection module 240 is used to obtain a SAR image to be classified as a target, input the SAR image into a trained target recognition network feature extraction unit, and obtain an out-of-domain data detection result based on the distance between the sample feature and the learned prototype feature. The out-of-domain data detection result is out-of-domain data or in-domain data. If it is in-domain data, the target recognition result with category probability is obtained through the classification unit.
[0112] For the specific definition of the SAR image out-of-domain data detection device based on dynamic multiple prototypes, please refer to the definition of the SAR image out-of-domain data detection method based on dynamic multiple prototypes above, which will not be repeated here. Each module in the above-mentioned SAR image out-of-domain data detection device based on dynamic multiple prototypes can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.
[0113] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 6 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for detecting out-of-domain data of a SAR image based on dynamic multiple prototypes is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a key, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.
[0114] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0115] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:
[0116] Acquire a training data set, wherein the training data set includes a plurality of SAR sample images related to known target categories and corresponding true value labels;
[0117] According to the features of all SAR sample images in the training data set, a prototype feature set of known target categories is obtained by clustering, wherein each known target category in the prototype feature set corresponds to at least one prototype feature;
[0118] The target recognition network is trained using the training data set and prototype feature learning is realized. The target recognition network comprises a feature extraction unit, an adaptive prototype learning unit and a classification unit in the training stage. The feature extraction unit extracts sample features of the SAR sample image. The adaptive prototype learning unit updates the prototype features according to the sample features and the relationship between the sample features and the nearest prototype features of the same type in the prototype feature set. The classification unit performs prediction according to the sample features to obtain a target category prediction result.
[0119] A joint prototype cross entropy loss function is calculated according to the target category prediction result, the prototype feature and the corresponding true value label, and the loss function includes a dynamic prototype distance loss function and a cross entropy loss function. The parameters in the feature extraction unit and the classification unit are iteratively updated according to the joint prototype cross entropy loss function, and the prototype features in the adaptive prototype learning unit are iteratively updated according to the dynamic prototype distance loss function until convergence, so as to obtain a trained target recognition network and a learned prototype feature, wherein the initialization prototype of the adaptive prototype learning unit is the mean of the corresponding cluster of the cluster;
[0120] A SAR image to be classified is obtained, and the SAR image is input into a trained target recognition network feature extraction unit, and an out-of-domain data detection result is obtained according to the distance between the sample feature and the learned prototype feature. The out-of-domain data detection result is out-of-domain data, or in-domain data. If it is in-domain data, a target recognition result with category probability is obtained through the classification unit.
[0121] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:
[0122] Acquire a training data set, wherein the training data set includes a plurality of SAR sample images related to known target categories and corresponding true value labels;
[0123] According to the features of all SAR sample images in the training data set, a prototype feature set of known target categories is obtained by clustering, wherein each known target category in the prototype feature set corresponds to at least one prototype feature;
[0124] The target recognition network is trained using the training data set and prototype feature learning is realized. The target recognition network comprises a feature extraction unit, an adaptive prototype learning unit and a classification unit in the training stage. The feature extraction unit extracts sample features of the SAR sample image. The adaptive prototype learning unit updates the prototype features according to the sample features and the relationship between the sample features and the nearest prototype features of the same type in the prototype feature set. The classification unit performs prediction according to the sample features to obtain a target category prediction result.
[0125] A joint prototype cross entropy loss function is calculated according to the target category prediction result, the prototype feature and the corresponding true value label, and the loss function includes a dynamic prototype distance loss function and a cross entropy loss function. The parameters in the feature extraction unit and the classification unit are iteratively updated according to the joint prototype cross entropy loss function, and the prototype features in the adaptive prototype learning unit are iteratively updated according to the dynamic prototype distance loss function until convergence, so as to obtain a trained target recognition network and a learned prototype feature, wherein the initialization prototype of the adaptive prototype learning unit is the mean of the corresponding cluster of the cluster;
[0126] A SAR image to be classified is obtained, and the SAR image is input into a trained target recognition network feature extraction unit, and an out-of-domain data detection result is obtained according to the distance between the sample feature and the learned prototype feature. The out-of-domain data detection result is out-of-domain data, or in-domain data. If it is in-domain data, a target recognition result with category probability is obtained through the classification unit.
[0127] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
[0128] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0129] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.
Claims
1. A method for detecting out-of-domain data in SAR images based on dynamic multiple prototypes, characterized in that: The method comprises: Acquire a training data set, wherein the training data set includes a plurality of SAR sample images related to known target categories and corresponding true value labels; According to the features of all SAR sample images in the training data set, a prototype feature set of known target categories is obtained by clustering, wherein each known target category in the prototype feature set corresponds to at least one prototype feature; The target recognition network is trained using the training data set and prototype feature learning is realized. The target recognition network comprises a feature extraction unit, an adaptive prototype learning unit and a classification unit in the training stage. The feature extraction unit extracts sample features of the SAR sample image. The adaptive prototype learning unit updates the prototype features according to the sample features and the relationship between the sample features and the nearest prototype features of the same type in the prototype feature set. The classification unit performs prediction according to the sample features to obtain a target category prediction result. A joint prototype cross entropy loss function is calculated according to the target category prediction result, the prototype feature and the corresponding true value label, and the loss function includes a dynamic prototype distance loss function and a cross entropy loss function. The parameters in the feature extraction unit and the classification unit are iteratively updated according to the joint prototype cross entropy loss function, and the prototype features in the adaptive prototype learning unit are iteratively updated according to the dynamic prototype distance loss function until convergence, so as to obtain a trained target recognition network and a learned prototype feature, wherein the initialization prototype of the adaptive prototype learning unit is the mean of the corresponding cluster of the cluster; A SAR image to be classified is obtained, and the SAR image is input into a trained target recognition network feature extraction unit, and an out-of-domain data detection result is obtained according to the distance between the sample feature and the learned prototype feature. The out-of-domain data detection result is out-of-domain data, or in-domain data. If it is in-domain data, a target recognition result with category probability is obtained through the classification unit.
2. The SAR image out-of-domain data detection method according to claim 1, characterized in that: According to the features of all SAR sample images in the training data set, a prototype feature set of known target categories is obtained by clustering, including: According to a preset number of prototypes, clustering is performed based on the feature data set to obtain a plurality of clusters, and the cluster center of each cluster is used as a prototype feature; According to the obtained multiple prototype features, the prototype feature set of the known target category is constructed.
3. The SAR image out-of-domain data detection method according to claim 2, characterized in that: Clustering based on the feature data set adopts any one of the following methods: K-Means method, hierarchical clustering method, DNSCAN clustering method, mean shift clustering method or spectral clustering method.
4. The SAR image out-of-domain data detection method according to claim 3, characterized in that: The step of clustering the feature data set according to the preset number of prototypes to obtain a plurality of clusters includes: Calculate according to the feature data set to obtain a corresponding distance matrix; Perform hierarchical clustering according to the distance matrix to obtain a dendrogram; The dendrogram is cut to obtain a plurality of clusters.
5. The SAR image out-of-domain data detection method according to claim 1, characterized in that: The pre-trained feature extraction neural network is used to extract features from all SAR sample images in the training data set.
6. The SAR image out-of-domain data detection method according to claim 1, characterized in that: After the SAR image is input into the feature extraction unit of the trained target recognition network: According to the distance between the image feature of the SAR image and each of the prototype features in the updated prototype feature set, it is determined whether the SAR image is out-of-domain data, and if it is out-of-domain data, it is used as a detection result; If the SAR image is in-domain data, the classification unit is used to perform classification according to the image features to obtain a target recognition result with category probability.
7. The SAR image out-of-domain data detection method according to any one of claims 1 to 6, characterized in that: The cross entropy loss function is calculated based on the target category prediction result and the corresponding true value label; The dynamic prototype distance loss function is calculated based on the sample features and the prototype features.
8. The SAR image out-of-domain data detection method according to claim 7, characterized in that: The joint prototype cross entropy loss function is expressed as: In the above formula, Represents the cross entropy loss function, where p i Represents the SAR sample image x i The target category prediction result, f(·) represents the feature extraction part of the neural network, and f(x i ) represents the features of the sample obtained after feature extraction, y i represents the corresponding true value label, N represents the number of samples in the training data set, represents the dynamic prototype distance loss function, μ j Represents the SAR sample image x i The corresponding nearest prototype feature in the class, λ represents the hyperparameter, Represents the label y i The corresponding prototype feature set.
9. A SAR image out-of-domain data detection device based on dynamic multiple prototypes, characterized in that: The device comprises: A training data set acquisition module is used to acquire a training data set, wherein the training data set includes a plurality of SAR sample images related to known target categories and corresponding true value labels; A prototype feature set obtaining module, used to obtain a prototype feature set of known target categories by clustering according to the features of all SAR sample images in the training data set, wherein each known target category in the prototype feature set corresponds to at least one prototype feature; A target recognition network training module is used to train the target recognition network using the training data set and realize prototype feature learning. The target recognition network includes a feature extraction unit, an adaptive prototype learning unit and a classification unit in the training stage. The feature extraction unit extracts sample features of the SAR sample image. The adaptive prototype learning unit updates the prototype features according to the sample features and the relationship between the sample features and the nearest prototype features of the same type in the prototype feature set. The classification unit predicts according to the sample features to obtain a target category prediction result. A network parameter and prototype feature update module, used to calculate a joint prototype cross entropy loss function according to the target category prediction result, the prototype feature and the corresponding true value label, the loss function includes a dynamic prototype distance loss function and a cross entropy loss function, iteratively update the parameters in the feature extraction unit and the classification unit according to the joint prototype cross entropy loss function, iteratively update the prototype features in the adaptive prototype learning unit according to the dynamic prototype distance loss function until convergence, and obtain a trained target recognition network and a learned prototype feature, wherein the initialization prototype of the adaptive prototype learning unit is the mean of the corresponding cluster of the cluster; The SAR image out-of-domain data detection module is used to obtain a SAR image to be classified as a target, input the SAR image into a trained target recognition network feature extraction unit, and obtain an out-of-domain data detection result based on the distance between the sample feature and the learned prototype feature. The out-of-domain data detection result is out-of-domain data or in-domain data. If it is in-domain data, a target recognition result with category probability is obtained through the classification unit.
Citation Information
Patent Citations
Open set domain adaptation method and system based on prototype comparative learning
CN116910571A
Clustering-based multi-prototype SAR image out-of-domain data post-detection method and device
CN119048909A
Method and system for unknown object detection
EP4290407A1
Systems and methods for noise-robust contrastive learning
US20210374553A1
Cited By
Medical image classification method and system based on prototype enhancement
CN120388236A
Medical image classification method and system based on class prototype enhancement
CN120388236B
