An optical image-assisted zero-shot classification method for radar scene images

Abstract

The present invention discloses an optical image-assisted zero-sample classification method for radar scene images. First, the features in the optical and radar scene images are fully extracted, and then a joint feature compatibility function is constructed to break through the compatibility bottleneck problem between the optical and radar scene image features. Finally, a superposition calibration is introduced to balance the domain offset of the visible category and the invisible category, and a one-to-many classifier is used to obtain the zero-sample classification result of the radar scene image, thereby achieving effective classification of invisible class samples. The method of the present invention overcomes the problems that the auxiliary information of other sources is difficult to fully utilize and the cross-domain features are difficult to be compatible, fully extracts the optical image feature and the radar scene image feature information, maximizes the compatibility of the extracted features in different domains, and achieves good classification performance in the radar scene image zero-sample classification task. It can classify categories without available training samples, and solves the problem that the prior art cannot classify invisible class samples.

Description

Technical Field

[0001] The present invention belongs to the field of radar scene image classification, and in particular relates to an optical image-assisted zero-sample classification method for radar scene images. Background Art

[0002] Radar remote sensing has the advantage of all-day and all-weather earth observation and has been widely used in civil fields such as terrain mapping and resource exploration. As an important application of radar image interpretation and interpretation, scene classification has attracted extensive attention and research in recent years. The existing radar scene image classification methods can effectively classify on the premise that templates or training models are derived from sufficient labeled data before classification, and cannot classify zero-sample categories with no available training samples. However, in the field of radar remote sensing, it is very likely that new categories are introduced after the training stage, or that several rare categories that are still of interest have no training samples, which becomes one of the key obstacles in the practical application of radar scene image classification.

[0003] Zero-shot learning is an effective method to solve this problem. It uses auxiliary information from other sources to associate new categories with trained categories and build classification models for new categories. It has achieved some promising results on some standard optical datasets. The paper "Song Q, Xu F. Zero-shot learning of SAR target featurespace with deep generative neural networks. IEEE Geoscience and Remote Sensing Letters, 2017, 14(12): 2245-2249" proposed a generation-based zero-shot radar target recognition framework and realized the similarity comparison between new categories and training categories. However, this method does not focus on the category prediction of zero samples, but mainly focuses on the direction prediction of new category targets. The paper "Li Y, Zhu Z, Yu JG, et al. Learning deep cross-modal embedding networks for zero-shot remote sensing image scene classification. IEEE Transactions on Geoscience and Remote Sensing, 2021, 59(12): 10590-10603." performed inter-modal and intra-modal supervision on radar scene images and semantic information in an end-to-end manner, and achieved a zero-shot classification accuracy of 43.8%. However, this method has limited auxiliary information from other sources, so its zero-shot classification accuracy still needs to be improved. Summary of the invention

[0004] In order to solve the above technical problems, the present invention proposes an optical image-assisted zero-sample classification method for radar scene images.

[0005] The technical solution adopted by the present invention is: an optical image-assisted zero-sample classification method for radar scene images, and the specific steps are as follows:

[0006] S1, collect optical and radar scene image samples and perform preprocessing;

[0007] S2, construct a zero-sample scene classification dataset of optical image-assisted radar scene images;

[0008] S3, extracting features of optical images and radar scene images;

[0009] S4, constructing a joint feature compatibility function for optical and radar scene images;

[0010] S5, build a calibration classifier;

[0011] S6. Build and train an optical image-assisted radar scene image zero-sample classification network to complete radar scene image classification.

[0012] Furthermore, the step S1 is specifically as follows:

[0013] Using the center cropping method, the collected optical image samples and radar scene image samples are cropped into image slices of the same size with the target at the center, and the sliced ​​samples are normalized:

[0014] X represents the sample data before normalization, the pixel value at the (i, j)th position is represented as X(i, j), and X′ represents the sample data after normalization.

[0015] Among them, min[X] represents the minimum value of the sample pixel value before normalization, and max[X] represents the maximum value of the sample pixel value before normalization.

[0016] Furthermore, the step S2 is specifically as follows:

[0017] The optical image samples and radar scene image samples are distinguished according to the target categories and divided into training sets and test sets; the optical image training set and data set contain all target categories, the radar scene image training set only contains visible target categories, and the test set contains all target categories.

[0018] Furthermore, the step S3 is specifically as follows:

[0019] S31, extracting latent semantic features from optical images by combining scale-invariant feature transform (SIFT) and latent Dirichlet allocation (LDA) algorithm;

[0020] First, extract feature points from each optical image: Use the SIFT algorithm to extract N feature points from each optical image. x l x Dimensional characteristics.

[0021] Then, the extracted features are clustered: the k-means algorithm is used for all the features extracted from the optical image to obtain N g Clusters are created, and a cluster index table is created to record the cluster center of each cluster and the feature points contained in the cluster.

[0022] Next, generate the corpus: According to the cluster index table established above, perform feature point frequency statistics on each category to obtain a C×N g The matrix of is used as the corpus. Where C represents the total number of categories.

[0023] Finally, the latent semantic feature representation of the target category of the optical image is obtained: the vector representing the latent semantic feature of each target category can be defined as:

[0024] w c ={θ1,θ2,…,θ l ,…,θ K} (1)

[0025] Among them, θ l represents the score of the cth category on the lth topic, K represents the total number of topics, and the potential semantic features of all categories can be extracted from the optical image W = {w1,w2,…,w c ,…,w C}.

[0026] S32. Design radar scene image feature extraction network;

[0027] The radar scene image feature extraction network is designed, which consists of five convolutional layers, ReLU function operations and maximum pooling layers arranged crosswise, and the network ends with two fully connected layers. The radar scene image convolution features v of different dimensions and containing different information are extracted from any convolutional layer and fully connected layer. conv and the fully connected feature v fc .

[0028] Furthermore, the step S4 is specifically as follows:

[0029] First, construct a fully connected feature compatible function:

[0030]

[0031] in, represents the fully connected feature compatibility score of the cth class, and T represents the vector transpose operation. represents the fully connected feature vector after dimensionality reduction, represents the field of real numbers, f w (·): Represents the latent semantic feature w c The dimension reduction mapping, and Represent q-dimensional and k-dimensional real vectors respectively, q represents w c , k represents the dimension of the feature after dimensionality reduction. v (·): Represents the dimensionality reduction mapping of radar scene image features, represents a p-dimensional real vector, p represents the fully connected feature v of the radar scene image fc Dimension.

[0032] Then, construct the convolution feature compatible function:

[0033]

[0034] in, represents the convolution feature compatibility score of the cth class, o ave (·) represents the global average pooling function, and its operation process is o ave : * represents the convolution operation, w and h represent the length and width of the feature map; represents the dimension reduction convolution weight. w ′(·): represents the latent semantic features w extracted from the optical image c The dimension reduction mapping, A represents a k′×s×s dimensional real tensor, where k′ and s represent the dimension of the convolution feature and the size of the feature map after dimensionality reduction, respectively. c,i Represents vector A c The i-th feature map in ; represents the i-th dimension-reduced convolution map feature map, d v ′(·): Represents the dimensionality reduction mapping of the radar scene feature map, and They represent J×w×h and k′×w×h dimensional real tensors respectively, J represents the number of convolutional layer feature maps before dimensionality reduction, k′<<J, represents the J feature maps of size w×h extracted from the convolutional layer of the radar scene image feature extraction network.

[0035] Finally, based on the fully connected feature compatibility function and the convolutional feature compatibility function, the fully connected feature compatibility score of the optical image latent semantics and the radar scene image is The convolution feature compatibility score is Then the joint feature compatibility function of the two is constructed as:

[0036]

[0037] in, represents the joint feature compatibility score of the cth class.

[0038] Furthermore, the step S5 is specifically as follows:

[0039] S51, classification superposition calibration introduction;

[0040] Introduce the superposition calibration formula:

[0041]

[0042] in, represents the corrected score after superposition calibration, represents the joint feature compatibility function for the input radar scene image x i Compatibility score. S represents the set of visible categories, and the superscript S represents the abbreviation of the visible class. δ(·) represents the indicator function. When the cth class is a visible class, δ(c) = 1, otherwise, δ(c) = 0. κ represents the calibration factor.

[0043] S52, build a one-to-many classifier;

[0044] Using the score of each category in formula (5) Build a one-vs-all classifier:

[0045]

[0046] in, Indicates the category with the highest score.

[0047] Furthermore, in step S6, based on steps S3-S5, an optical image-assisted radar scene image zero-sample classification network is built and then trained, as follows:

[0048] S61, radar scene image feature extraction network training;

[0049] The visible radar scene images are divided into a training set and a test set using the data set division method described in step S2, and the designed radar scene image feature extraction network is trained, combined with automatic learning rate adjustment and breakpoint continuous training.

[0050] S62, training loss function setting;

[0051] The IsLand loss function and Hinge loss function are used to optimize the network model. The IsLand loss function is expressed as:

[0052]

[0053] Among them, L IL Represents the IsLand loss function, L C represents the central loss function, μ represents the hyperparameter, c k and c j represents the feature center of the kth category and the jth category, Represents the set of feature centers.

[0054] Hinge loss expression is:

[0055]

[0056] Among them, L H represents the Hinge loss function, y represents the true value label of the radar scene image, when the image belongs to this category, y = 1, otherwise, y = -1. or L H Take the minimum value, the input radar scene image sample is classified correctly, and L H Optimize the stacking classifier.

[0057] Beneficial effects of the present invention: The method of the present invention first fully extracts features from optical and radar scene images, then constructs a joint feature compatibility function, breaks through the compatibility bottleneck problem between optical and radar scene image features, and finally introduces superposition calibration to balance the domain offset of visible and invisible categories, and uses a one-to-many classifier to obtain zero-sample classification results for radar scene images, thereby achieving effective classification of invisible class samples. The method of the present invention overcomes the problems of difficulty in fully utilizing auxiliary information from other sources and difficulty in compatibility of cross-domain features, fully extracts optical image features and radar scene image feature information, maximizes the compatibility of extracted features from different domains, achieves good classification performance in the radar scene image zero-sample classification task, and can classify categories without available training samples, solving the problem that the prior art cannot classify invisible class samples. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 The present invention is a flowchart of an optical image-assisted zero-sample classification method for radar scene images.

[0059] Figure 2 Schematic diagram of the radar scene image feature extraction network structure used in an embodiment of the present invention.

[0060] Figure 3 A diagram showing the distribution of latent semantic features of an optical image in an embodiment of the present invention.

[0061] Figure 4 A diagram of radar scene image classification results in an embodiment of the present invention. DETAILED DESCRIPTION

[0062] The present invention is mainly verified by simulation experiment, and all steps and conclusions are verified to be correct by python3.9 on Windows 10 operating system platform. To facilitate those skilled in the art to understand the technical content of the present invention, the content of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0063] like Figure 1 As shown, a flowchart of an optical image-assisted zero-sample classification method for radar scene images of the present invention is shown, and the specific steps are as follows:

[0064] S1, collect optical and radar scene image samples and perform preprocessing;

[0065] S2, construct a zero-sample scene classification dataset of optical image-assisted radar scene images;

[0066] S3, extracting features of optical images and radar scene images;

[0067] S4, constructing a joint feature compatibility function for optical and radar scene images;

[0068] S5, build a calibration classifier;

[0069] S6. Build and train an optical image-assisted radar scene image zero-sample classification network to complete radar scene image classification.

[0070] In this embodiment, the step S1 is specifically as follows:

[0071] Using the center cropping method, the collected optical image samples and radar scene image samples are cropped into image slices of the same size with the target at the center, and the sliced ​​samples are normalized:

[0072] X represents the sample data before normalization, the pixel value at the (i, j)th position is represented as X(i, j), and X′ represents the sample data after normalization.

[0073] Among them, min[X] represents the minimum value of the sample pixel value before normalization, and max[X] represents the maximum value of the sample pixel value before normalization.

[0074] In this embodiment, step S2 is specifically as follows:

[0075] The optical image samples and radar scene image samples are separated according to the target categories and divided into training sets and test sets in a ratio of 7:3. The optical image training set and data set contain all target categories, the radar scene image training set only contains visible target categories, and the test set contains all target categories.

[0076] In this embodiment, step S3 is specifically as follows:

[0077] S31. Combining the scale-invariant feature transform (SIFT) and the latent Dirichlet allocation (LDA) algorithm to extract latent semantic features from optical images.

[0078] First, feature points are extracted from each optical image: 50 128-dimensional features are extracted from each optical image using the SIFT algorithm.

[0079] Then, the extracted features are clustered: the k-means algorithm is used for all the features extracted from the optical image to obtain 100 clusters, and a cluster index table is created to record the cluster center of each cluster and the feature points contained in the cluster.

[0080] Next, generate a corpus: According to the clustering index table established above, perform frequency statistics of feature points for each category to obtain a 7×100 matrix as the corpus.

[0081] Finally, the latent semantic feature representation of the target category of the optical image is obtained: the vector representing the latent semantic feature of each target category is:

[0082] w c ={θ1,θ2,…,θ l ,…,θ K} (9)

[0083] Among them, θ l represents the score of the cth category on the lth topic, The total number of topics K is set to 120. Through the above method, the latent semantic features W = {w1, w2, ..., w c ,…,w C},w c Represents the latent semantic features of the cth class.

[0084] The first ten optical image latent semantic features extracted by this method are as follows: Figure 2As shown, it can be seen that desert, lake, coast, and farmland have significant scores on Topic 2, Topic 3, Topic 7, and Topic 10, which corresponds to their significant features on the optical image. Each category has a low score on Topic 8. The mountain category has a more uniform score on each topic than the other categories, which means that it lacks features that can be significantly distinguished from other categories.

[0085] S32. Design radar scene image feature extraction network;

[0086] The designed radar scene image feature extraction network consists of five convolutional layers, ReLU function operations and maximum pooling layers arranged crosswise, and the network ends with two fully connected layers.

[0087] Radar scene image convolution features and fully connected features of different dimensions and containing different information can be extracted from any convolution layer and fully connected layer. The radar scene image feature extraction network structure is as follows: Figure 3 Its parameters are shown in Table 1. The fully connected features of the extracted radar scene image are defined as v fc , the convolution feature of the radar scene image is v conv .

[0088] Table 1

[0089] Layer Category Layer structure Convolution layer Conv1+ReLU function Convolution kernel size: 5×5, convolution kernel size: 16 Maximum pooling layer Pool1 Pooling kernel size: 2×2 Convolutional layer Conv2+ReLU function Convolution kernel size: 5×5, convolution kernel size: 32 Maximum pooling layer Pool2 Pooling kernel size: 2×2 Convolution layer Conv3+ReLU function Convolution kernel size: 6×6, convolution kernel size: 64 Maximum pooling layer Pool3 Pooling kernel size: 2×2 Convolutional layer Conv4+ReLU function Convolution kernel size: 5×5, convolution kernel size: 64 Maximum pooling layer Pool4 Pooling kernel size: 2×2 Convolutional layer Conv5+ReLU function Convolution kernel size: 5×5, convolution kernel size: 32 Maximum pooling layer Pool5 Pooling kernel size: 2×2 Dropout Discard ratio: 0.2 Flatten - Fully connected layer Fc1 Number of units: 64 Fully connected layer Fc2 Number of units: 6

[0090] In this embodiment, step S4 is specifically as follows:

[0091] The joint feature compatibility function is constructed to maximize the compatibility of the extracted features. Since the dimensions of the extracted latent semantic features W and radar scene image features V are different, they are first mapped to a common low-dimensional feature domain. For feature diversity, convolutional layers and fully connected layers are introduced into the joint feature compatibility function to reduce dimensionality. Note that the word "joint" refers to making full use of convolutional features and fully connected features, and compatibility refers to the relationship between latent semantic features and radar scene image features.

[0092] First, construct a fully connected feature compatible function:

[0093]

[0094] in, represents the fully connected feature compatibility score of the cth class, and T represents the vector transpose operation. is the fully connected feature vector after dimensionality reduction, represents the field of real numbers, f w (·): Represents the latent semantic feature w c Dimensionality reduction mapping, 120 is w c , and 50 is the dimension after feature dimensionality reduction. v(·): Represents the dimensionality reduction mapping of radar scene image features, 512 is the fully connected feature v of the radar scene image fc The dimension of f w (·) and d v (·) The latent semantic feature w c and radar scene image features v fc Mapped to the feature domain of the same dimension, and both dimensionality reduction mapping functions are parameterized by a fully connected multi-layer neural network.

[0095] Then, construct the convolution feature compatible function:

[0096]

[0097] in, represents the convolution feature compatibility score of the cth class, o ave (·) represents the global average pooling function, and its operation process is o ave : The symbol * represents a convolution operation. represents the dimensionality reduction convolution weight, f w ′(·): represents the latent semantic features w extracted from the optical image c , 5 and 3 represent the dimension of the convolution feature after dimensionality reduction and the length and width of the feature map, respectively. c,i Represents vector A c The i-th feature map in . represents the i-th dimension-reduced convolution map feature map, d′ v (·): Represents the dimensionality reduction map of the radar scene feature map. represents 32 feature maps of size 5×5 extracted from the convolutional layer of the radar scene image feature extraction network. w (·) and d′ v (·) Parameterized by a multi-layer convolutional neural network.

[0098] Finally, based on the above fully connected feature compatibility function and convolutional feature compatibility function, the fully connected feature compatibility score of the optical image latent semantics and the radar scene image is The convolution feature compatibility score is Therefore, the joint feature compatibility function of the two is constructed as:

[0099]

[0100] in, represents the joint feature compatibility score of the cth category. The joint feature compatibility function is beneficial to increase the diversity of the extracted feature vectors and enhance the correlation between features in different fields of the same category.

[0101] In this embodiment, in step S5, the stacking calibration and one-to-many classifier are combined to eliminate the domain shift problem of the visible class and the invisible class in the zero-sample classification process of the deep network, as follows:

[0102] S51, classification superposition calibration introduction;

[0103] Since only visible class samples are input during the training phase, the visible class dominates the model. Therefore, when inputting test samples, the model tends to mark samples of the invisible class as visible class, which is called "domain shift" in zero-shot classification. Introducing superposition calibration to balance the "domain shift" problem, the superposition calibration formula is introduced as follows:

[0104]

[0105] in, represents the corrected score after superposition calibration, represents the joint feature compatibility function for the input radar scene image x i Compatibility score. S represents the set of visible categories, and the superscript S represents the abbreviation of the visible class. δ(·) represents the indicator function. When the cth class is a visible class, δ(c) = 1, otherwise, δ(c) = 0. κ represents the calibration factor.

[0106] In fact, the score of the seen category output by the joint feature compatibility function is higher than that of the unseen category. Therefore, the purpose of introducing the calibration function is to balance the scores of the seen and unseen categories by adjusting the calibration factor κ, thereby reducing the model's judgment tendency on the unseen category.

[0107] S52, build a one-to-many classifier;

[0108] Using the score of each category in formula (13) Build a one-vs-all classifier:

[0109]

[0110] in, It indicates the category with the highest score. From formula (14), it can be seen that the modified compatibility score of each category is equivalent to a binary classifier. Therefore, the c-th category radar scene image multi-classifier can be realized by combining C binary classifiers.

[0111] In this embodiment, in step S6, based on steps S3-S5, an optical image-assisted radar scene image zero-sample classification network is built and then trained, as follows:

[0112] S61, radar scene image feature extraction network training;

[0113] The visible radar scene image is divided into a training set and a test set in a ratio of 7:3 using the data set division method described in step S2, and the designed radar scene image feature extraction network is trained, and the recognition rate of the designed network for the radar scene category is continuously improved by combining the training method of automatic learning rate adjustment and breakpoint continuous training, so that the designed network obtains better feature extraction capabilities. In this embodiment, during the training process, the initial learning rate is set to 0.001, the batchsize is set to 32, and the Adam optimizer is used for training acceleration optimization.

[0114] S62, training loss function setting;

[0115] Two loss functions are used to optimize the network model: IsLand loss function and Hinge loss function. IsLand loss function can make the distance between classes larger and the features within the class more compact. Its expression formula is:

[0116]

[0117] Among them, L IL Represents the IsLand loss function, L C represents the central loss function, μ represents the hyperparameter, c k and c j represents the feature center of the kth category and the jth category, Represents the set of feature centers. Using L IL The radar scene classification network is optimized to enable it to extract more discriminative radar scene image features and improve classification accuracy.

[0118] Hinge loss is a commonly used function in maximum margin optimization, and its expression is:

[0119]

[0120] Among them, L H represents the Hinge loss function, y represents the true value label of the radar scene image, when the image belongs to this category, y = 1, otherwise, y = -1. or L H Take the minimum value, the input radar scene image sample is classified correctly, and L H Optimize the stacking classifier.

[0121] Figure 4is the confusion matrix of the classification results of the method of the present invention. It can be seen that the method of the present invention not only achieves a recognition rate of 62% for the zero-sample category mountain peak, but also can achieve recognition rates of 87%, 99%, 86%, 84%, 86% and 75% for visible categories such as cities, coasts, deserts, farmlands, lakes and rivers, respectively, and its average recognition rate of visible categories can reach 86%. Therefore, the method of the present invention can maintain a high recognition rate of visible categories while effectively identifying the zero-sample category, verifying the effectiveness of the method of the present invention. In addition, it can be seen from the last row of the confusion matrix that 17% and 15% of "mountains" are predicted as "farmlands" and "deserts" because the semantic features of these two categories are similar, which increases the difficulty of classification, but the method of the present invention can still achieve good zero-sample classification performance of radar scene images.

[0122] In summary, the method of the present invention overcomes the problems that it is difficult to fully utilize auxiliary information from other sources and that cross-domain features are difficult to be compatible, fully extracts the optical image features and radar scene image feature information, maximizes the compatibility of the extracted features from different domains, and achieves good classification performance in the zero-sample classification task of radar scene images. It can classify categories without available training samples, and solves the problem that the prior art cannot classify invisible class samples.

[0123] Those skilled in the art should be aware that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the scope of protection of the present invention is not limited to such specific statements and embodiments. For those skilled in the art, the present invention may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of the claims of the present invention.

Claims

1. An optical image-assisted zero-shot classification method for radar scene images, the specific steps are as follows: S1, collect optical and radar scene image samples and perform preprocessing; S2, construct a zero-sample scene classification dataset of optical image-assisted radar scene images; S3, extracting features of optical images and radar scene images; S4, constructing a joint feature compatibility function for optical and radar scene images; S5, build a calibration classifier; S6. Build and train an optical image-assisted radar scene image zero-shot classification network to complete radar scene image classification; The step S3 is specifically as follows: S31, extracting latent semantic features from optical images by combining scale-invariant feature transform (SIFT) and latent Dirichlet allocation (LDA) algorithm; First, extract feature points from each optical image: Use the SIFT algorithm to extract N feature points from each optical image. x l x Characteristics of dimensions; Then, the extracted features are clustered: The k-means algorithm is used for all features extracted from the optical image to obtain N g Clusters, create a cluster index table, record the cluster center of each cluster and the feature points contained in the cluster; Next, generate the corpus: According to the cluster index table established above, perform feature point frequency statistics on each category to obtain a C×N g The matrix is ​​used as the corpus; where C represents the total number of categories; Finally, the latent semantic feature representation of the target category of the optical image is obtained: the vector representing the latent semantic feature of each target category is defined as: w c ={θ1,θ2,…,θ l ,…,θ K } (1) Among them, θ l represents the score of the cth category on the lth topic, K represents the total number of topics, and the potential semantic features of all categories can be extracted from the optical image W = {w1,w2,…,w c ,…,w C }; S32. Design radar scene image feature extraction network; Design a radar scene image feature extraction network, which consists of five convolutional layers, ReLU function operations and maximum pooling layers arranged crosswise, and the network ends with two fully connected layers; extract radar scene image convolution features v of different dimensions and containing different information from any convolutional layer and fully connected layer conv and the fully connected feature v fc ; The step S4 is specifically as follows: First, construct a fully connected feature compatible function: in, represents the fully connected feature compatibility score of the cth class, and T represents the vector transposition operation; represents the fully connected feature vector after dimensionality reduction, represents the field of real numbers, Represents the latent semantic feature w c The dimension reduction mapping, and Represent q-dimensional and k-dimensional real vectors respectively, q represents w c The dimension of , k represents the dimension after feature dimensionality reduction; Represents the dimensionality reduction mapping of radar scene image features, represents a p-dimensional real vector, p represents the fully connected feature v of the radar scene image fc Dimensions; Then, construct the convolution feature compatible function: in, represents the convolution feature compatibility score of the cth class, o ave (·) represents the global average pooling function, and its operation process is: * represents the convolution operation, w and h represent the length and width of the feature map; represents the dimensionality reduction convolution weight; represents the latent semantic features w extracted from the optical image c The dimension reduction mapping, A represents a k′×s×s dimensional real tensor, where k′ and s represent the dimension of the convolutional feature and the size of the feature map after dimensionality reduction, respectively; c,i Represents vector A c The i-th feature map in ; represents the i-th dimension-reduced convolution map feature map, Represents the dimensionality reduction mapping of the radar scene feature map, and They represent J×w×h and k′×w×h dimensional real tensors respectively, J represents the number of convolutional layer feature maps before dimensionality reduction, k′<<J, represents J feature maps of size w×h extracted from the convolutional layer of the radar scene image feature extraction network; Finally, based on the fully connected feature compatibility function and the convolutional feature compatibility function, the fully connected feature compatibility score of the optical image latent semantics and the radar scene image is The convolution feature compatibility score is Then the joint feature compatibility function of the two is constructed as: in, represents the joint feature compatibility score of the cth class; The step S5 is specifically as follows: S51, classification superposition calibration introduction; Introduce the superposition calibration formula: in, represents the corrected score after stacking calibration, represents the joint feature compatibility function for the input radar scene image x i Compatibility score; S represents the set of visible categories, the superscript S represents the abbreviation of the visible class; δ(·) represents the indicator function, when the cth class is a visible class, δ(c) = 1, otherwise, δ(c) = 0, κ represents the calibration factor; S52, build a one-to-many classifier; Using the score of each category in formula (5) Build a one-vs-all classifier: in, indicates the category with the highest score; In step S6, based on steps S3-S5, an optical image-assisted radar scene image zero-sample classification network is built and then trained, as follows: S61, radar scene image feature extraction network training; The visible radar scene images are divided into a training set and a test set using the data set division method described in step S2, and the designed radar scene image feature extraction network is trained, and the training is performed in combination with automatic learning rate adjustment and breakpoint continuous training; S62, training loss function setting; The IsLand loss function and Hinge loss function are used to optimize the network model; the IsLand loss function expression formula is: Among them, L IL Represents the IsLand loss function, L C represents the central loss function, μ represents the hyperparameter, c k and c j represents the feature center of the kth category and the jth category, represents the set of feature centers; Hinge loss expression is: Among them, L H represents the Hinge loss function, y represents the true value label of the radar scene image, when the image belongs to this category, y = 1, otherwise, y = -1; when or L H Take the minimum value, the input radar scene image sample is classified correctly, and L H Optimize the stacking classifier.

2. The optical image-assisted zero-shot classification method for radar scene images according to claim 1, characterized in that: The step S1 is specifically as follows: Using the center cropping method, the collected optical image samples and radar scene image samples are cropped into image slices of the same size with the target at the center, and the sliced ​​samples are normalized: X represents the sample data before normalization, the pixel value at the (i, j)th position is represented as X(i, j), and X′ represents the sample data after normalization. Among them, min[X] represents the minimum value of the sample pixel value before normalization, and max[X] represents the maximum value of the sample pixel value before normalization.

3. The optical image-assisted zero-shot classification method for radar scene images according to claim 1, characterized in that: The step S2 is specifically as follows: The optical image samples and radar scene image samples are distinguished according to the target categories and divided into training sets and test sets; the optical image training set and data set contain all target categories, the radar scene image training set only contains visible target categories, and the test set contains all target categories.

Images