A method and device for training image classification model based on SAR images

Abstract

The present invention provides a method and device for training an image classification model based on SAR images. Multi-source images, including SAR, infrared, and visible light images and corresponding image classification data, are acquired at the same angle for the same target. The "infrared-visible light features" obtained by training an auxiliary neural network based on infrared and visible light images are used to assist in training an image classification model based on SAR images. Infrared and visible light images are only used as auxiliary modalities during the training process, and the input of the network in actual applications is SAR single-modal images. The present invention uses a multi-task learning method to improve the accuracy of image classification based on SAR images, and provides a method for solving the limitations of single-modal SAR images in practical applications.

Description

Technical Field

[0001] The present invention relates to the field of images and deep learning, and in particular to a method and device for training an image classification model based on SAR images. Background Art

[0002] Synthetic Aperture Radar (SAR) is an active imaging method that uses the Doppler shift theory and radar coherence principle. SAR has strong penetration and can effectively detect camouflaged targets. Its imaging is not limited by light, climate, and fog. Therefore, it has high practical application value in military reconnaissance, geographic mapping, disaster monitoring, and other fields. However, due to the resolution limit, the single-modal image classification network based on SAR images has certain limitations.

[0003] Infrared images can distinguish targets from backgrounds and maintain good imaging effects throughout the day. Visible light images have high spatial resolution and can provide more delicate texture details. By fusing the advantages of the two images, we can combine the thermal radiation information in infrared images and the delicate information in visible light images. The fusion of infrared and visible images performs well in image processing. Dual-modal images can obtain multi-faceted scene information and extract rich target image information. Therefore, the multi-modal fusion image classification network based on infrared and visible light modalities can show superior performance in image classification tasks.

[0004] Convolutional Neural Network (CNN) is a feedforward neural network. Due to its network structure, it has excellent performance in image processing, especially large-scale image processing. Therefore, CNN has been widely used in applications such as image recognition and target detection. CNN has obvious advantages in terms of computational complexity compared to other network structures, and thus has been widely used.

[0005] Infrared, visible light images and SAR images belong to different modalities, and their imaging mechanisms are very different, so their applications are also different. SAR can provide rich target information and is almost unaffected by weather, but single-mode SAR images are limited by resolution in practical applications. The fusion image classification network using infrared and visible light dual modalities has better performance in classification accuracy. Therefore, it is necessary to develop an image classification network based on SAR images that can perform image classification tasks with higher accuracy, with the assistance of infrared and visible light modalities. It can improve the accuracy of single-modal image classification networks based on SAR images to a certain extent, and also provide ideas for other applications based on single-modal SAR images to achieve performance improvement. Summary of the invention

[0006] In view of the shortcomings of the prior art, the present invention provides a method and device for training an image classification model based on SAR images. In view of the limitations of single-modal SAR images in practical applications, infrared-visible light features are used as prior knowledge to improve the single-modal image classification network based on SAR images. In practical applications, the network only needs to input SAR images without the participation of other modalities. The present invention uses a multi-task learning method to enable the image classification network based on SAR images to learn richer image information to improve the network classification accuracy.

[0007] The objective of the present invention is achieved through the following technical solutions:

[0008] The method comprises the following steps:

[0009] Step S1: acquiring multi-source image data for the same target, including SAR, infrared, visible light image data and image classification data;

[0010] Step S2: construct and train an auxiliary neural network using the acquired infrared and visible light images. The auxiliary neural network takes the preprocessed infrared and visible light images as input and outputs the predicted image classification results and infrared-visible light features. The specific steps are as follows:

[0011] Step S21: construct an auxiliary neural network, use the main convolution layer of the ResNet-50 network structure to extract infrared and visible light single modal features respectively, use the convolution layer to fuse the above infrared and visible light single modal features to obtain infrared-visible light features, and use the prediction layer of the ResNet-50 network structure to predict the image classification result with the above infrared-visible light features as input;

[0012] Step S22: using the acquired infrared and visible light images and image classification data to train the auxiliary neural network;

[0013] Step S23: extracting and saving the infrared-visible light features of the auxiliary neural network trained in step S22;

[0014] Step S3: Using the acquired SAR image and the infrared-visible light features saved in step S23, a target network is constructed and trained. The target network only needs the SAR single-modal image as input, and the output is the image classification result and the fitting of the infrared-visible light features. The specific steps are as follows:

[0015] Step S31: construct a target network using the DenseNet-121 network structure;

[0016] Step S32: The target network is trained using the acquired SAR image, image classification data and the infrared-visible light features saved in step S23. The two target outputs of the network are the image classification result and the infrared-visible light features. The errors between the two target outputs and the corresponding true values ​​constitute the actual error of the network. The loss is expressed as:

[0017] Loss = loss1 + loss2 * φ

[0018] Among them, loss1 is the loss between the classification result predicted by the network and the actual classification result, loss2 is the loss between the infrared-visible light features fitted by the network and the infrared-visible light features, and φ is the hyperparameter that balances loss1 and loss2; the two losses jointly determine the weight update of the target network, thereby obtaining an image classification model based on SAR images.

[0019] Preferably, the classification accuracy of the auxiliary neural network is required to be greater than or equal to 0.95;

[0020] The auxiliary neural network requires that the image classification accuracy be two percentage points greater than that of the SAR image single-modal network.

[0021] Preferably, in the target network loss function, loss1 and loss2 use a cross entropy loss function and a mean square error loss function respectively.

[0022] Preferably, the target network is a multi-task target network.

[0023] Preferably, the objective function of the multi-task target network is:

[0024]

[0025] sU T =I

[0026] Where m is the number of target network tasks, n i is the number of training samples, is the label of sample j of task i, is the loss function, b=(b1,...b m ) T represents the offset compensation of task i, U∈R d×d It includes the weight parameters of i tasks, d is the parameter dimension, ||A|| 2 2,1 is the L2 regularization matrix, a irepresents the weight parameter of the task, I is the unit matrix, and λ is the regularization parameter; the first half of the formula (1) shown represents all the losses of the i tasks, and the second half uses L2 regularization to ensure the sparse rows of the solution and the orthogonalization of the constraint matrix U, which can be transformed into:

[0027]

[0028] stD≥0,tr(D)≤1

[0029] in is the first half of formula (1), tr(W T D -1 W) is the trace of the matrix, W i =Ua i is the weight parameter of task i, and D ≥ 0 specifies that the D matrix is ​​a semi-positive definite matrix. By solving the covariance matrix D, the multi-task problem of the target network is decoupled to achieve the purpose of parallel computing, that is, to optimize the multi-task target network.

[0030] A training device for an image classification model training method based on SAR images, specifically comprising:

[0031] A multi-source image acquisition unit is used to acquire multi-source images of the same target at the same angle, including SAR images, infrared images and visible light images, and to mark the images with classification labels to obtain image classification data;

[0032] An infrared-visible light feature acquisition unit, used for acquiring infrared-visible light features through an auxiliary neural network;

[0033] The target network construction unit is used to take the SAR single-modal image as input, and the predicted infrared-visible light features and image classification results as output. The image classification model based on the SAR image is trained by using the acquired SAR image, the classification result data and the infrared-visible light features extracted by the auxiliary neural network;

[0034] The multi-source image preprocessing unit is used to process the corresponding SAR images, infrared images and visible light images into pictures of the same size.

[0035] The trained SAR image-based image classification model only needs SAR single-modality image data to obtain rich image information, thus showing better performance. Specifically:

[0036] An image classification device based on SAR images, comprising:

[0037] Image data acquisition module, used to acquire SAR single-mode image data for image classification tasks;

[0038] The SAR image classification module is used to input the SAR image into the SAR image-based image classification model trained by any of the above training methods to obtain an image classification result.

[0039] The present invention uses a multi-task learning method to assist in training an image classification model based on SAR images through a fusion image classification neural network based on infrared and visible light images. The method aims to assist in the training of SAR image classification models through infrared-visible light features to improve the image classification accuracy based on SAR images. At the same time, infrared and visible light images are only used as prior knowledge in the training process and do not need to be used as network input in practical applications. The present invention provides a method for solving the limitations of single-modality SAR images in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 The figure is a flowchart of an image classification model based on SAR images.

[0041] Figure 2 This is a diagram of the neural network structure of an image classification model based on SAR images. DETAILED DESCRIPTION

[0042] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0043] As attached Figure 1 , Attachment Figure 2 The flowchart and neural network structure diagram of the image classification model based on SAR images are shown. The present invention provides an image classification model training method that only requires SAR single-modality images as input, which specifically includes the following steps:

[0044] Step S1: Obtain a multi-source image dataset for image classification tasks. The specific steps are as follows:

[0045] Step S11: acquiring multi-source images of the same angle for the same target, including SAR images, infrared images and visible light images, dividing the training set and the validation set into a certain proportion, and establishing a labeling file for storing image classification data;

[0046] Step S12: The obtained multi-source images are preprocessed. In this embodiment, all image data sizes are cut to 224*224 to meet the requirements of the attached Figure 1 , Attachment Figure 2 The network input size requirements are shown. After preprocessing, a multimodal dataset for image classification tasks is obtained;

[0047] Step S2: Using the acquired infrared and visible light image data and image classification data, construct and train an auxiliary neural network. The specific steps are as follows:

[0048] Step S21: construct an auxiliary neural network, as shown in the attached Figure 1 As shown, the single-modal feature extraction network adopts the main convolution layer of the ResNet-50 structure; the fusion layer is a convolution layer, and this embodiment requires that the infrared-visible light feature size and the number of channels are consistent with the single-modal feature map; the prediction layer adopts the prediction layer of the ResNet-50 structure, that is, the prediction layer composed of the pooling layer and the fully connected layer at the end of the network; the specific structure of ResNet is shown in Table 1:

[0049] Table 1: ResNet network structure

[0050]

[0051] Step S22: training the auxiliary neural network, taking the acquired infrared and visible light image data and image classification data as input, training the auxiliary neural network, and this embodiment requires that the classification accuracy of the auxiliary neural network is greater than or equal to 0.95;

[0052] Step S23: Extract the infrared-visible light features of the auxiliary neural network trained in step S22, as shown in the attached Figure 2 The fusion layer output shown, the saved features need to be normalized using the Sigmoid function.

[0053] Step S3: Using the acquired SAR image data, the infrared-visible light features saved in step S23, and the image classification data, the target network is constructed and trained. The specific steps are as follows:

[0054] Step S31: construct the target network, as shown in the attached Figure 1 As shown, in this embodiment, the DenseNet-121 network structure is used to construct the target network; wherein, the specific structure of DenseNet is shown in Table 2:

[0055] Table 2: DenseNet network structure

[0056]

[0057]

[0058] Step S32: Train the target network, as shown in the attached Figure 2 As shown, the network takes SAR single-mode images as input, the main task is image classification, and the auxiliary task is the fitting of the infrared-visible light features saved in step S23. Therefore, the network includes two outputs, namely the SAR image classification result and the fitting output of the infrared-visible light features. The error of the network is composed of the two above-mentioned two outputs and their corresponding true values, and the loss is expressed as:

[0059] Loss = loss1 + loss2 * φ

[0060] Among them, loss1 and loss2 are the loss between the classification result predicted by the network and the actual classification result and the loss between the feature map obtained by the main convolution layer of the SAR image and the infrared-visible light feature, respectively. φ is a hyperparameter used to balance loss1 and loss2. The loss1 adopts the cross entropy loss function, and the loss2 adopts the mean square error loss function; the two losses jointly determine the weight update of the target network, thereby training an image classification model based on SAR images.

[0061] Based on the above-mentioned training method of the image classification model based on SAR images, the present invention provides a training device of the image classification model based on SAR images, which specifically includes the following units:

[0062] A multi-source image acquisition unit is used to acquire multi-source images of the same target at the same angle, including SAR images, infrared images and visible light images, and to mark the images with classification labels to obtain image classification data;

[0063] The infrared-visible light feature acquisition unit is used to acquire infrared-visible light features through an auxiliary neural network.

[0064] The target network construction unit is used to take the SAR single-modal image as input and the predicted infrared-visible light features and image classification results as output. The image classification model based on SAR images is trained by using the acquired SAR images, classification result data and infrared-visible light features extracted by the auxiliary neural network.

[0065] The multi-source image preprocessing unit is used to process the corresponding SAR images, infrared images and visible light images into pictures of the same size.

[0066] The above embodiments are not limitations of the present invention, and the present invention is not limited to the above embodiments. As long as the requirements of the present invention are met, they belong to the protection scope of the present invention.

[0067] The contents not described in detail in the specification of the present invention belong to the prior art known to the professional and technical personnel in this field.

Claims

1. A method for training an image classification model based on SAR images, characterized in that: The method comprises the following steps: Step S1: acquiring multi-source image data for the same target, including SAR, infrared, visible light image data and image classification data; Step S2: construct and train an auxiliary neural network using the acquired infrared and visible light images. The auxiliary neural network takes the preprocessed infrared and visible light images as input and outputs the predicted image classification results and infrared-visible light features. The specific steps are as follows: Step S21: construct an auxiliary neural network, use the main convolution layer of the ResNet-50 network structure to extract infrared and visible light single modal features respectively, use the convolution layer to fuse the above infrared and visible light single modal features to obtain infrared-visible light features, and use the prediction layer of the ResNet-50 network structure to predict the image classification result with the above infrared-visible light features as input; Step S22: using the acquired infrared and visible light images and image classification data to train the auxiliary neural network; Step S23: extracting and saving the infrared-visible light features of the auxiliary neural network trained in step S22; Step S3: using the acquired SAR image and the infrared-visible light features saved in step S23, constructing and training a target network, wherein the target network only needs the SAR single-modal image as input, and outputs the image classification result and the fitting of the infrared-visible light features; The specific steps are as follows: Step S31: construct a target network using the DenseNet-121 network structure; Step S32: Use the acquired SAR image, image classification data and infrared-visible light features saved in step S23 to train the target network. The two target outputs of the network are the image classification result and the infrared-visible light features. The errors between the two target outputs and the corresponding true values ​​constitute the actual error of the network. The loss is expressed as: Loss = loss1 + loss2 * φ Among them, loss1 is the loss between the classification result predicted by the network and the actual classification result, loss2 is the loss between the infrared-visible light features fitted by the network and the infrared-visible light features, loss1 and loss2 use the cross entropy loss function and mean square error loss function respectively; φ is the hyperparameter that balances loss1 and loss2; the two losses jointly determine the weight update of the target network, thereby obtaining an image classification model based on SAR images.

2. The image classification model training method based on SAR images according to claim 1 is characterized in that: The classification accuracy of the auxiliary neural network is required to be greater than or equal to 0.

95.

3. The image classification model training method based on SAR images according to claim 1, characterized in that: The target network is a multi-task target network.

4. The image classification model training method based on SAR images according to claim 3 is characterized in that: The objective function of the multi-task target network is: stUU T =I Where m is the number of target network tasks, n i is the number of training samples, is the label of sample j of task i, is the loss function, b=(b1,...b m ) T represents the offset compensation of task i, U∈R d×d Including the weight parameters of i tasks, d is the parameter dimension, is the L2 regularization matrix, a i represents the weight parameter of the task, I is the unit matrix, and λ is the regularization parameter; the first half of the formula (1) shown represents all the losses of the i tasks, and the second half uses L2 regularization to ensure the sparse rows of the solution and the orthogonalization of the constraint matrix U, which can be transformed into: stD≥0,tr(D)≤1 in is the first half of formula (1), tr(W T D -1 W) is the trace of the matrix, W i =Ua i is the weight parameter of task i, and D≥0 specifies that the D matrix is ​​a semi-positive definite matrix.

5. A training device for an image classification model training method based on SAR images, specifically comprising: A multi-source image acquisition unit is used to acquire multi-source images of the same target at the same angle, including SAR images, infrared images and visible light images, and to mark the images with classification labels to obtain image classification data; An infrared-visible light feature acquisition unit, used for acquiring infrared-visible light features through an auxiliary neural network; The auxiliary neural network takes the preprocessed infrared and visible light images as input, uses the main convolution layer of the ResNet-50 network structure to extract infrared and visible light single modal features respectively, uses the convolution layer to fuse the above infrared and visible light single modal features to obtain infrared-visible light features, and uses the prediction layer of the ResNet-50 network structure to predict the image classification result with the above infrared-visible light features as input; The target network construction unit uses the DenseNet-121 network structure to build the target network, with SAR single-modal images as input, and the predicted infrared-visible light features and image classification results as output. The error between the two target outputs and the corresponding true values ​​constitutes the actual error of the network, and the loss is expressed as: Loss = loss1 + loss2 * φ Among them, loss1 is the loss between the classification result predicted by the network and the actual classification result, loss2 is the loss between the infrared-visible light feature fitted by the network and the infrared-visible light feature, loss1 and loss2 use the cross entropy loss function and mean square error loss function respectively; φ is the hyperparameter for balancing loss1 and loss2; the image classification model based on SAR images is trained using the acquired SAR images, classification result data and infrared-visible light features extracted by the auxiliary neural network; The multi-source image preprocessing unit is used to process the corresponding SAR images, infrared images and visible light images into pictures of the same size.

Images