A zero-shot SAR image recognition method based on dual-branch autoencoder

Abstract

The invention provides a zero-shot SAR image recognition method based on a dual-branch autoencoder, which belongs to the technical field of image processing. The method comprises: constructing a dual-branch generation network G‑Net to extract semantic features and visual features of images in a training set; inputting fused features into a classification network C‑Net and a conversion network T‑Net, wherein the classification network C‑Net outputs target classification of the SAR image, and the conversion network T‑Net outputs a reconstructed image of the SAR image, and the positions of all known targets in a feature space are obtained; and back-propagation gradient optimization is performed on the network by calculating known target classification and image reconstruction losses to predict the positions of unknown targets in the feature space. The invention realizes accurate recognition of known targets and effective distinction of unknown targets, and solves the problem of distribution concentration in zero-shot SAR image recognition tasks.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing, and in particular relates to a zero-sample SAR image recognition method based on a dual-branch automatic encoder. Background Art

[0002] Synthetic Aperture Radar (SAR) is an active sensor technology used to obtain radar images of the earth's surface. SAR images are obtained by installing a radar antenna on a radar platform, emitting microwave signals and receiving the reflected signals. Unlike optical sensors, SAR is not limited by sunlight, clouds and atmospheric conditions and can be observed in any weather conditions. SAR images have the following characteristics: 1) High resolution: SAR can achieve high spatial resolution image acquisition, with a typical resolution of several meters or even sub-meter level. 2) All-weather capability: SAR is not limited by lighting conditions and can acquire images during the day, at night and under cloud cover. 3) Independence: SAR can be independent of the optical reflectivity of surface features, so it is suitable for monitoring different surface types, such as land, ocean, ice and snow covered areas, etc. 4) Rich information: SAR images can provide physical and geometric information of the surface, including surface morphology, texture, moving targets, etc. Due to the above characteristics of SAR images, it has a wide range of applications in earth observation, environmental monitoring, military reconnaissance, disaster monitoring and other fields.

[0003] Automatic target recognition is one of the most challenging tasks in synthetic aperture radar (SAR) applications. Although existing deep learning methods have achieved considerable success in automatic target recognition tasks for SAR, existing methods are unable to recognize objects that are not present in the training set. Summary of the invention

[0004] In view of the above-mentioned deficiencies in the prior art, the present invention proposes a new zero-shot SAR image recognition method based on a dual-branch autoencoder, which aims to expand the recognition ability of existing methods for SAR images, reduce data acquisition and annotation costs, achieve rapid adaptation to unknown targets, help respond to emergencies and new threats, and improve security monitoring and protection capabilities.

[0005] The present invention provides a zero-sample SAR image recognition method based on a dual-branch autoencoder, comprising:

[0006] Obtain target classification semantic information of known SAR images as training sets;

[0007] Construct a dual-branch generative network G-Net to extract the semantic features and visual features of the images in the training set;

[0008] Fuse semantic features and visual features into fused features;

[0009] The fused features are input into the classification network C-Net and the conversion network T-Net. The classification network C-Net outputs the target classification of the SAR image, and the conversion network T-Net outputs the reconstructed image of the SAR image to obtain the positions of all known targets in the feature space. The network is back-propagated and gradient optimized by calculating the known target classification and image reconstruction losses to predict the positions of unknown targets in the feature space.

[0010] The classification loss of known targets is used as the target to retrain the classification network C-Net and the conversion network T-Net. The conversion network T-Net maps the target image to the feature space and inputs it into the classification network C-Net to obtain the reconstructed image classification result. The network is optimized by back-propagating the known target classification loss to enhance the accuracy of the conversion network T-Net in predicting the positioning of the image in the feature space.

[0011] The process of zero-shot target recognition is as follows: by reversely inputting the SAR image into the transformation network T-Net, the position of the input image in the feature space is obtained. For known targets, the classification network C-Net network can be input to predict the category. For unknown targets, they are distinguished from known categories based on their position in the feature space.

[0012] Furthermore, the semantic feature extraction branch of the dual-branch generation network G-Net uses a multi-layer perceptron network to extract features from the semantic information of the input target; the visual feature extraction branch of the dual-branch generation network G-Net uses a multi-layer convolutional neural network to extract features from the input SAR image;

[0013] The semantic feature extraction branch includes a three-layer fully connected neural network FCN, which outputs semantic features;

[0014] The visual feature extraction branch includes 5 convolutional neural network residual modules; each residual module includes a convolution module, a MaxPooling layer and a LeakyReLU activation function layer connected in series, each residual module outputs a feature map and serves as the input of the next residual module, and finally outputs the visual feature.

[0015] Furthermore, the fusion of the semantic features and the visual features into fusion features includes:

[0016] Expand the visual features and connect them with the semantic features to obtain the connection features;

[0017] The connection features are input into a feature fusion network to obtain fused features, wherein the feature fusion network is a fully connected network consisting of three fully connected layers.

[0018] Furthermore, the classification network C-Net includes three fully connected layers, and obtains the prediction probability of each category through the ReLU activation function layer and the SoftMax layer; the conversion network T-Net includes a deconvolution module corresponding to the visual feature extraction branch, which is used to obtain a reconstructed image of the input image according to the input fusion features;

[0019] Furthermore, the calculation of known target classification and image reconstruction loss includes:

[0020] The known target classification loss L is calculated according to the following cl , and by minimizing L cl Adapting trainable parameters :

[0021]

[0022] in, is the i-th SAR image in the input training set, is the category label corresponding to the image, A S is the semantic feature of the image, N S is the number of training samples, S is the number of visible categories of the image, and p() is the probability distribution function;

[0023] The mixed loss L of reconstruction and classification is calculated according to the following formula hy ;

[0024]

[0025] Among them, QUOTE To reconstruct the image, For Category At the center of the feature space, represents a custom loss weight, where is the reconstruction and classification error term of the SAR image, which is used to expand the inter-class clustering variance in the feature space. is the central error term, which is used to reduce the cluster variance within the class.

[0026] Furthermore, the classification loss of the known target is used as a target to retrain the classification network C-Net and the conversion network T-Net, including:

[0027] First training phase: minimization Conduct model training, , , To customize the loss weights, disable the central error term at the beginning of training , only the reconstruction and classification error terms are applied; after the feature space distribution reaches a fairly stable state, the central error term is enabled;

[0028] Second training phase: by minimizing the known target classification loss The conversion network T-Net is trained to locate unknown targets. The conversion network T-Net takes the SAR image as input and outputs the position of the target in the feature space. C-Net accepts the output of the conversion network T-Net as input and outputs the class label corresponding to the input SAR image.

[0029] The beneficial effects of the present invention are as follows: the present invention proposes a new method for zero-shot SAR image recognition based on a dual-branch auto-encoding network, which can realize accurate recognition of known targets and effectively distinguish unknown targets, solving the problem of distribution concentration in zero-shot SAR image recognition tasks; constructing a stable feature space through visual and semantic information, and further increasing the inter-class variance of the feature space by fine-tuning the classification network; realizing rapid adaptation to unknown targets, and performing recognition and classification, helping to respond to emergencies and new threats, and improving security monitoring and protection capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0031] Figure 1 It is a specific flow chart of the zero-sample SAR image recognition method provided by the present invention.

[0032] Figure 2 This is a neural network architecture diagram proposed in the present invention. DETAILED DESCRIPTION

[0033] In order to enable those skilled in the art to better understand the technical solutions in the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0034] like Figure 1 As shown, the method includes:

[0035] Step 101: Collect and label training data.

[0036] Obtain target classification semantic information of known SAR images as training sets.

[0037] Step 102: Extracting semantic features and visual features.

[0038] A dual-branch generative network G-Net is constructed to extract the semantic features and visual features of the images in the training set.

[0039] Step 103: Multimodal feature fusion.

[0040] The semantic features and visual features are fused into fused features.

[0041] Step 104: Feature space construction.

[0042] The fused features are input into the classification network C-Net and the conversion network T-Net. The classification network C-Net outputs the target classification of the SAR image, and the conversion network T-Net outputs the reconstructed image of the SAR image to obtain the positions of all known targets in the feature space. The network is back-propagated and gradient optimized by calculating the known target classification and image reconstruction losses to predict the positions of unknown targets in the feature space.

[0043] Step 105: Neural network fine-tuning.

[0044] The classification loss of known targets is used as the target to retrain the classification network C-Net and the conversion network T-Net. The conversion network T-Net maps the target image to the feature space and inputs it into the classification network C-Net to obtain the reconstructed image classification result. The network is optimized by back-propagating the known target classification loss to enhance the accuracy of the conversion network T-Net in predicting the positioning of the image in the feature space.

[0045] Step 106: Zero-shot target recognition.

[0046] By reversely inputting the SAR image into the transformation network T-Net, the position of the input image in the feature space is obtained. For known targets, the classification network C-Net network can be input to predict the category. For unknown targets, they are distinguished from known categories based on their position in the feature space.

[0047] The present invention proposes a zero-shot SAR image recognition method based on a dual-branch autoencoder. Figure 2 The neural network shown in the figure can expand the ability of existing recognition methods for SAR images through this method. By learning the feature representation of known target categories and migrating them to unknown categories, it can realize the recognition and classification of new targets and reduce the cost of data collection and annotation. At the same time, it can realize rapid adaptation to unknown targets, and perform recognition and classification, help respond to emergencies and new threats, and improve the ability of security monitoring and protection.

[0048] Optionally, as an embodiment of the present invention, the training data collection and labeling specifically include the following contents:

[0049] The model is trained using the MASTAR dataset, which is a classic SAR image target recognition dataset, which contains images of ground targets such as BTR70 (armored transport vehicle), BMP2 (infantry fighting vehicle), T72 (tank), 2S1 (self-propelled howitzer), BRDM2 (armored reconnaissance vehicle), BTR60 (armored transport vehicle), D7 (bulldozer), T62 (tank), ZIL131 (freight truck), and ZSU234 (self-propelled anti-aircraft gun). The present invention needs to re-label the MASTAR dataset, create a semantic attribute matrix for each type of target, and record and summarize ten different attributes to describe the equipment status of the target, including whether the target has armor, barrel, auxiliary machine gun, hatch, track, periscope, etc.

[0050] This embodiment uses a simple text data mining method to capture the relevant semantic information of known SAR targets. By crawling web pages, a large amount of text data is collected. Then, an open source toolkit is used to extract high-frequency words to describe the attributes of each type of target. Ten different attributes are summarized to describe the equipment status of the target, including whether the target has armor, barrel, auxiliary machine gun, hatch, track, periscope, amphibious, the number of wheels, and whether there are two side doors. A binary vector, 1 is true and 0 is false, is used to represent the equipment status of the target.

[0051] Optionally, as an embodiment of the present invention, the extraction of semantic features and visual features includes the following: using a generative network G-Net composed of a dual-branch network to extract the semantic features and visual features of the image respectively, and using a multimodal feature fusion network to perform feature fusion. The semantic feature extraction network branch uses a multi-layer perceptron network to extract features from the semantic attribute matrix of the input target. The visual feature extraction network branch uses a multi-layer convolutional neural network to extract features from the input SAR image. Since SAR targets have a high degree of visual similarity, the use of semantic attributes can increase the separability between classes.

[0052] About semantic feature extraction network:

[0053] The semantic attribute matrix of a given target , input to the semantic feature extraction network branch Na for feature extraction. Na is composed of a three-layer fully connected neural network FCN. Its first layer network l1 consists of 32 neurons and outputs a vector of size 32; its second layer network l1 consists of 64 neurons and outputs a vector of size 64; its third layer is the output layer, which consists of 128 neurons and finally outputs a semantic feature vector F of size 128.a .

[0054] About the visual feature extraction network branch:

[0055] Given a dataset image I of size 640×640×3, input it to the visual feature extraction network branch N v . N v It consists of 5 convolutional neural network residual modules, including {L1, L2, L3, L4, L5}. Each residual module is composed of a convolution module, a MaxPooling layer and a LeakyReLU activation function layer with a slope of 0.1 in series. The MaxPooling layer mainly downsamples the input data to reduce the amount of data and the number of parameters, improve computational efficiency, and help enhance the model's ability to learn translation invariance. The LeakyReLU activation function layer has certain advantages in gradient propagation and sparsity, and is widely used in convolutional neural networks and other deep learning models to help improve the performance and generalization ability of the model.

[0056] Specific L 1 The convolution module is composed of a convolution layer with a convolution kernel of 64, a step size of 2, a size of 3×3, and a valid padding, and a convolution block in series. The convolution block is composed of a convolution layer with a convolution kernel of 64, a step size of 1, a size of 1×1, and the same padding, and a convolution layer with a convolution kernel of 64, a step size of 1, a size of 3×3, and the same padding, and the input and output of the convolution block are connected through residuals (the convolution blocks involved in the present invention all implement residual connections). Residual connections can solve the problem of gradient disappearance, improve the fitting ability of the model, accelerate model convergence, and achieve a deeper network structure. L 1 The input of the residual module is a dataset image I of size 640×640×3, and its output is a feature map F1 of size 320×320×64. 2 The convolution module is composed of a convolution layer with a kernel of 128, a stride of 2, a size of 3×3, and a valid padding, and a convolution block repeated twice in series. The convolution block is composed of a convolution layer with a kernel of 128, a stride of 1, a size of 1×1, and the same padding, and a convolution layer with a kernel of 128, a stride of 1, a size of 3×3, and the same padding in series. L 2 The input of the residual module is a feature map F1 of size 320×320×64, and its output is a feature map F2 of size 160×160×128. 3The convolution module is composed of a convolution layer with a kernel size of 256, a stride of 2, a size of 3×3, and a valid padding, and a convolution block repeated eight times in series. The convolution block is composed of a convolution layer with a kernel size of 256, a stride of 1, a size of 1×1, and the same padding, and a convolution layer with a kernel size of 256, a stride of 1, a size of 3×3, and the same padding in series. L 3 The input of the residual module is a feature map F2 of size 160×160×128, and its output is a feature map F3 of size 80×80×256. 4 The residual module is composed of a convolutional layer with a kernel size of 256, a stride of 2, a size of 3×3, and a valid padding, and a convolutional block repeated eight times in series. The convolutional block is composed of a convolutional layer with a kernel size of 256, a stride of 1, a size of 1×1, and the same padding, and a convolutional layer with a kernel size of 256, a stride of 1, a size of 3×3, and the same padding in series. 4 The input of the convolution module is a feature map F3 of size 80×80×256, and its output is a feature map F4 of size 40×40×256. 5 The convolution module is composed of a convolution layer with a kernel of 128, a stride of 2, a size of 3×3, and a valid padding, and a convolution block repeated twice in series. The convolution block is composed of a convolution layer with a kernel of 128, a stride of 1, a size of 1×1, and the same padding, and a convolution layer with a kernel of 128, a stride of 1, a size of 3×3, and the same padding in series. L 5 The input of the residual module is a feature map F4 of size 40×40×256, and its output is a feature map of size 20×20×128, which is the visual feature F5.

[0057] Optionally, as an embodiment of the present invention, multimodal feature fusion fuses semantic features and visual features to achieve mapping of the target's visual and semantic information into a unified feature space.

[0058] About feature fusion module:

[0059] This step combines the visual feature F5 with the semantic feature F a First, the visual features are expanded to obtain a vector F with a vector size of 400×128. flat , then the vector F flat and semantic feature F a Connect and get the connection feature F con, with a size of 401 × 128. Next, feature fusion is performed through the feature fusion network, which is a fully connected network consisting of three fully connected layers. Specifically, the first layer of the network consists of 4096 neurons, and its output is a feature vector of size 4096. The second layer of the network consists of 1024 neurons, and its output is a feature vector of size 1024. The third layer of the network consists of 256 neurons, and its output is the final fusion feature F of size 256.

[0060] Optionally, as an embodiment of the present invention, the feature space construction specifically includes the following contents:

[0061] The training set data is input into the generative network G-Net to extract and fuse semantic and visual features. The obtained feature vector is input into the classification network C-Net and the conversion network T-Net. The classification network outputs the classification result of the image, and the conversion network outputs the reconstruction of the SAR image. The network is back-propagated through target classification and image reconstruction losses to optimize the gradient and construct a feature space with good inter-class separability. All known targets can find their positions in the feature space.

[0062] Specifically, given a training set , including N s training samples

[0063] , the corresponding label is The training set category label is represented by y i ={1,2,3,…S}, with a total of S visible categories, and the corresponding semantic information matrix is ​​represented by A S ={a1,a2,...a S}. In order to obtain a feature space with sufficient discriminability, the classification loss L is used cl and the mixed loss L hy Train the neural network.

[0064] About Classification Network:

[0065] The classification network C-Net is used to predict the category of the input image. Specifically, the network includes three fully connected layers with 64, 32, and 10 neurons respectively. The output vector of size 10 is obtained. After passing through the ReLU activation function layer and the SoftMax layer, the predicted probability of each category is obtained. The classification loss L is calculated according to the following formula cl ,C-Net maximizes the inter-class variance of the target by learning auxiliary classification information to achieve the purpose of zero-shot learning.

[0066]

[0067] By minimizing Lcl , will adapt the trainable parameters To increase the probability of the correct class label.

[0068] About switching networks:

[0069] The input image is reconstructed using the conversion network T-Net. Specifically, the network includes 5 deconvolution modules corresponding to the visual feature extraction network in G-Net. With the input fusion feature F of size 256, T-Net can obtain the input image Reconstructed image of QUOTE .

[0070] The mixing loss L is calculated according to the following formula hy .

[0071]

[0072] For Category At the center of the feature space, Represents a custom loss weight, where the first term is the reconstruction loss of the SAR image, which is used to expand the inter-class clustering variance in the feature space, and the second term is the center loss, which is used to reduce the intra-class clustering variance.

[0073] Optionally, as an embodiment of the present invention, neural network fine-tuning specifically includes the following contents:

[0074] The classification loss of known targets is used as the target to retrain T-Net and C-Net in the fine-tuning stage. T-Net maps the target image to the feature space, and C-Net, as an auxiliary network, takes the output of T-Net as input to obtain the image classification result. The network is optimized by back-propagating the classification loss to enhance the positioning accuracy of T-Net for image features. After fine-tuning, T-Net will be able to accurately predict the location of unknown targets in the feature space. In the testing phase, there is no need to fine-tune the neural network.

[0075] The model training process is divided into two stages. In the first stage, by minimizing . Disable the central error term at the beginning of training To avoid significant fluctuations in the model at the beginning of training, only the reconstruction and classification error terms are applied. After the feature space distribution reaches a fairly stable state, the central error term is enabled.

[0076]

[0077] After the first stage, the known target classification loss is minimized Train T-Net to interpret (or locate) unknown targets. The architecture of T-Net is inversely symmetric to the visual extraction branch of G-Net. T-Net takes SAR images as input and outputs the position of the target in the feature space. C-Net accepts the output of T-Net as input and outputs the class label corresponding to the input SAR image. By minimizing the known target classification loss To improve the positioning accuracy of T-Net. In this training phase, G-Net does not need to be trained again.

[0078] Optionally, as an embodiment of the present invention, zero-sample target recognition specifically includes the following contents:

[0079] By reversely inputting the SAR image into the T-Net network, the position of the input image in the feature space is obtained. For known targets in the training set, the output of T-Net is used as the input of the classification network C-Net to obtain the predicted category. For position targets that do not exist in the training set, they are distinguished from other known categories based on their position in the feature space, thereby realizing the function of zero-sample SAR image recognition.

[0080] Although the present invention has been described in detail by referring to the accompanying drawings and in combination with the preferred embodiments, the present invention is not limited thereto. Without departing from the spirit and essence of the present invention, a person of ordinary skill in the art may make various equivalent modifications or substitutions to the embodiments of the present invention, and these modifications or substitutions shall be within the scope of the present invention or any person skilled in the art may easily think of changes or substitutions within the technical scope disclosed by the present invention, and shall be within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A zero-shot SAR image recognition method based on a dual-branch autoencoder, characterized in that: include: Obtain target classification semantic information of known SAR images as training sets; Construct a dual-branch generative network G-Net to extract the semantic features and visual features of the images in the training set; The semantic features and visual features are fused to obtain fused features; The fused features are input into the classification network C-Net and the conversion network T-Net. The classification network C-Net outputs the target classification of the SAR image, and the conversion network T-Net outputs the reconstructed image of the SAR image to obtain the positions of all known targets in the feature space. The network is back-propagated and gradient optimized by calculating the known target classification and image reconstruction losses to predict the positions of unknown targets in the feature space. The classification loss of known targets is used as the target to retrain the classification network C-Net and the conversion network T-Net. The conversion network T-Net maps the target image to the feature space and inputs it into the classification network C-Net to obtain the reconstructed image classification result. The network is optimized by back-propagating the known target classification loss to enhance the accuracy of the conversion network T-Net in predicting the positioning of the image in the feature space. The process of zero-shot target recognition is as follows: the SAR image is input into the conversion network T-Net to obtain the position of the input image in the feature space. For known targets, the classification network C-Net network can be input to predict the category. For unknown targets, they are distinguished from known categories based on their position in the feature space. The semantic feature extraction branch of the dual-branch generation network G-Net uses a multi-layer perceptron network to extract features from the semantic information of the input target; the visual feature extraction branch of the dual-branch generation network G-Net uses a multi-layer convolutional neural network to extract features from the input SAR image; The semantic feature extraction branch includes a three-layer fully connected neural network FCN, which outputs semantic features; The visual feature extraction branch includes 5 convolutional neural network residual modules; each residual module includes a convolution module, a MaxPooling layer and a LeakyReLU activation function layer connected in series, each residual module outputs a feature map and serves as the input of the next residual module, and finally outputs the visual feature; The classification network C-Net includes three fully connected layers, and obtains the prediction probability of each category through the ReLU activation function layer and the SoftMax layer; the conversion network T-Net includes a deconvolution module corresponding to the visual feature extraction branch, which is used to obtain a reconstructed image of the input image according to the input fusion features; The calculation of known target classification and image reconstruction loss includes: The known target classification loss L is calculated according to the following cl , and by minimizing L cl Adapting trainable parameters : in, is the i-th SAR image in the input training set, is the category label corresponding to the image, A S is the semantic feature of the image, N S is the number of training samples, S is the number of visible categories of the image, and p() is the probability distribution function; The mixed loss L of reconstruction and classification is calculated according to the following formula hy ; in, To reconstruct the image, For Category At the center of the feature space, represents a custom loss weight, where is the reconstruction and classification error term of the SAR image, which is used to expand the inter-class clustering variance in the feature space. is the central error term, which is used to reduce the cluster variance within the class; The classification loss of the known target is used as the target to retrain the classification network C-Net and the conversion network T-Net, including: First training phase: minimization Conduct model training, , , To customize the loss weights, disable the central error term at the beginning of training , only the reconstruction and classification error terms are applied; after the feature space distribution reaches a fairly stable state, the central error term is enabled; Second training phase: by minimizing the known target classification loss The conversion network T-Net is trained to locate unknown targets. The conversion network T-Net takes the SAR image as input and outputs the position of the target in the feature space. C-Net accepts the output of the conversion network T-Net as input and outputs the class label corresponding to the input SAR image.

2. The method according to claim 1, characterized in that The step of fusing the semantic features and the visual features to obtain the fused features includes: Expand the visual features and connect them with the semantic features to obtain the connection features; The connection features are input into a feature fusion network to obtain fused features, wherein the feature fusion network is a fully connected network consisting of three fully connected layers.

Images