Robust multimodal remote sensing image target detection method

Abstract

The present invention belongs to the technical field of image information processing, and provides a robust multimodal remote sensing image target detection method. Aiming at the problem that three modal images cannot be obtained in the same scene, a robust multimodal remote sensing image target detection method based on knowledge distillation is proposed. It can be applied to different types of input source images, and can obtain good prediction results. It summarizes and synthesizes the knowledge of a series of target detectors specially designed for different modal images, and forms a robust single network. The method of the present invention can obtain a robust target detection network for multiple modal remote sensing images by only using unaligned multimodal images.

Description

Technical Field

[0001] The present invention belongs to the technical field of image information processing, and in particular relates to a method for multimodal image target detection. Background Art

[0002] At present, the methods related to this patent include two aspects: the first is the target detection algorithm based on deep learning; the second is the deep learning algorithm of knowledge distillation.

[0003] The target detection methods based on deep learning are mainly divided into two categories: two-stage target detection algorithms, which mostly design specific network structures to extract areas that may be targets. The network proposed by Ren et al. in the paper "Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks" designs an RPN network for extracting areas that may be targets, and then classifies these areas. With the assistance of specific areas, the accuracy is high, but there is also a problem of slow detection speed; the one-stage target detection network does not have a network structure for extracting regions of interest. In order to improve the accuracy of the network, a large number of scholars have conducted numerous studies. Redmon et al. in the paper "You Only Look Once: Unified, Real-Time Object Detection" adopted the method of dividing the image into blocks and assigned the task of target detection to the image blocks. Lin et al. solved the problem of sample imbalance in the one-stage network in "Focal Loss for Dense Object Detection" and improved the detection accuracy of the network. Since the one-stage network abandons the calculation process of extracting the region of interest, it has a faster target detection speed than the two-stage network, but abandoning the extraction of the region of interest inevitably leads to a decrease in the accuracy of target detection.

[0004] The deep learning algorithm of knowledge distillation was proposed by Hinton et al. in the paper "Distilling the Knowledge in a Neural Network". The purpose of this method is to compress the complex teacher model knowledge into a simple and lightweight student model, thereby improving the performance of the lightweight model. That is, the student model is trained by sending data into a pre-trained teacher model to generate soft labels. Although the original idea was to compress complex models into simpler models through matching predictions, distillation further extends this approach to matching intermediate layer features, inter-domain knowledge transfer and many other aspects.

[0005] A common method to solve the problem of multimodal image object detection is to use image fusion to fuse images of different modalities into one image and send it to the network for detection. This requires that images of three modalities exist at the same time in the same scene, which is difficult to meet in most situations, so it is also difficult to use this method for detection. Summary of the invention

[0006] Aiming at the problem that three modal images cannot be obtained in the same scene, a robust multimodal remote sensing image target detection method based on knowledge distillation is proposed. It can be applied to different types of input source images and can obtain good prediction results. It summarizes the knowledge of a series of target detectors specially designed for different modal images and forms a robust single network.

[0007] The technical solution of the present invention:

[0008] A robust multimodal remote sensing image target detection method, the steps are as follows:

[0009] Three object detectors are trained using images of three modalities respectively;

[0010] ResNet50 is used as the backbone network of three target detectors to extract features for target detection;

[0011] During the network training phase, the loss functions of the three target detectors are the same, and the difference lies in the input data. The loss function of any target detector is explained as follows: Since there are only a small number of targets in an image, directly using cross entropy to calculate the loss function of each position will cause the loss value of the positive sample to be "swamped" by the loss value of the negative sample, affecting the network's learning of the positive sample features. Therefore, we use the Facel Loss loss function to balance the loss function values ​​corresponding to the positive and negative samples:

[0012]

[0013] Among them, N1 is the number of targets in a training batch, Y xyc is the true value of the image, The network prediction value α and β are two hyperparameters of the network used to determine the balance between positive and negative samples in the anchor box classification process. The anchor box regression branch actually predicts the position and size difference between the artificial anchor box and the true value; the true value anchor box is now assumed to be (x, y, h, w), and the four values ​​represent the horizontal coordinate, vertical coordinate, height and width of the anchor box position; the artificial anchor box is The network needs to predict:

[0014]

[0015] The following loss function is used to supervise the network:

[0016]

[0017]

[0018]

[0019] Where N2 is the number of predicted boxes whose overlap (IOU) with the jth true value box is greater than the threshold; is the network's prediction value for the i-th anchor box that meets the requirements (the overlap (IOU) with the j-th true value box is greater than the threshold), t mj is the corresponding true value, (x j ,y j ,h j ,w j ) is the parameter of the jth true value anchor box, The parameters of the anchor box are manually specified for the i-th person who meets the requirements. The total loss function of the network is:

[0020] L=L class +εL loc (1-4)

[0021] Among them, ε is used to balance the losses of the two tasks, L class and L loc is the loss function specified in equations (1-1) and (1-3).

[0022] Three target detectors perform knowledge distillation to obtain a robust network

[0023] We propose an inductive method for multimodal remote sensing target detection tasks based on knowledge distillation, which integrates the knowledge of different target detection networks specially trained on multiple source datasets. It is a model enhanced by multiple target detectors from multiple sources. Therefore, we can complete the prediction of any input source image using only a single modality target detection network. The purpose of knowledge distillation is to compress the complex teacher model knowledge into a simple and lightweight student model. They train the student model by feeding the data into a pre-trained teacher model to generate soft labels. Although the original idea was to compress complex models into simpler models by matching predictions, distillation further extends this approach to matching intermediate layer features, inter-domain knowledge transfer, and many other aspects.

[0024] The student network selects the corresponding teacher network for learning based on the input. The process of image knowledge distillation for the three modalities is similar, so we only explain one modality. Here we assume that we have a dataset {a i ,b i}, i = 1, 2, ..., n, where a is the input image and b is the classification label. t ,P s The classification prediction results of the teacher and student models are as follows. The classification task optimization objectives of our student detection model during training are as follows:

[0025] L cls =μL hard (P s ,b)+(1-μ)L soft (P s ,P t )(1-5)

[0026] Among them, L hard is the hard loss between the student model prediction and the true value, L soft is the soft loss between the predictions of the student model and the predictions of the teacher model. μ is a hyperparameter used to balance the soft and hard losses. The soft labels contain the potential relationships between targets of different categories discovered by the teacher model. By learning soft labels, the student network inherits this hidden information. However, unlike simple classification problems, detection problems need to deal with severe imbalances between different categories, that is, the background dominates. Therefore, we use class-weighted cross entropy as the classification distillation loss:

[0027] L soft (P s ,P t )=-∑w c P t logP s (1-6)

[0028] where w c To balance the weights, we use larger weights for the target class and smaller weights for the background class; P s and P t These are the target classification prediction results of the student model and the teacher model respectively. For the target detection network, there is another branch used to locate and regress the target bounding box. Similar to the classification branch, we also use the true value and the prediction of the teacher model to calculate the bounding box regression loss.

[0029] L reg =L sL1 (R s ,y reg )+νL t (R s ,R t ,y reg )(1-7)

[0030] Among them, y reg is the true coordinate point vector of the bounding box, Rs ,R t are the regression prediction results of the student and teacher models, respectively. ν is a hyperparameter used to balance the loss between the student model prediction results and the teacher model prediction results. L sL1 is the smooth L1 loss. For the loss, we describe it as follows:

[0031]

[0032] That is, we believe that when the prediction error of the student and teacher models exceeds a certain range, we use L2 loss for the true value. The prediction of the teacher model here is used as a judgment criterion rather than directly used. Among them, m is the discrimination boundary parameter. Using the teacher's intermediate performance can help the training process of the student model and improve the student's final performance. Therefore, we constrain the features of the intermediate and terminal layers of the student network respectively. We use the L2 distance between the student and teacher models for constraints, and the feature constraint loss is described as follows:

[0033] L f (F s ,F t )=||F s -F t ||2(1-9)

[0034] Among them, F s and F t They are the feature maps of the teacher network (target detector) and the student network respectively. The network loss function in the final distillation process is:

[0035] L d =λ1L cls +λ2L reg +λ3L f (1-10)

[0036] Among them, λ1, λ2, and λ3 are used to balance the losses of the two tasks.

[0037] Beneficial effects of the present invention: The multimodal remote sensing image target detection method of the present invention utilizes the knowledge distillation method. The current common image fusion algorithm requires a set of multimodal images in the same scene during training and inference, and this requirement is generally difficult to meet. However, the method of the present invention does not have this problem. Only by using unaligned (multiple modal images in the same scene do not need to exist at the same time) multimodal images can a robust target detection network for multimodal remote sensing images be obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 Flowchart of the training method for three target detectors.

[0039] Figure 2This is a flowchart of the knowledge distillation process after the three target detectors are trained. Teacher 1, Teacher 2, and Teacher 3 represent Figure 1 Three object detectors trained in .

[0040] Figure 3 Schematic diagram of the network structure of three target detectors.

[0041] Figure 4 This is a schematic diagram of the knowledge distillation process. Since the knowledge distillation process of modalities two and three is similar to that of modality one, for the sake of simplicity, the figure only shows the image distillation process of modality one. DETAILED DESCRIPTION

[0042] The specific implementation of the present invention is further described below in conjunction with the accompanying drawings and technical solutions.

[0043] Figure 1 The training method flow chart of the three target detectors is as follows: First, three detectors with the same structure (such as Figure 3 As shown in the figure, we first train a target detector for each modality 1, 2, and 3, and then train three corresponding target detectors on the data of modalities 1, 2, and 3: target detector 1, target detector 2, and target detector 3. The three target detectors are supervised using equation (1-4) as the loss function. When the loss function of a target detector converges, the training of the target detector ends. When the training of the three target detectors ends, this process ends. After this process, we can obtain three target detectors (teacher networks) with good performance on the corresponding modality data.

[0044] Figure 2 The figure is a flow chart of the knowledge distillation process. For the data of network training, we first determine its modality: when the training image comes from modality 1, the student network follows teacher 1 (target detector 1) to learn; when the training image comes from modality 2, the student network follows teacher 2 (target detector 2) to learn; when the training image comes from modality 3, the student network follows teacher 3 (target detector 3) to learn; since the learning process of the three modalities is the same, we take modality 1 as an example to illustrate the specific process. When the input is a remote sensing image of modality 1, it is input into the teacher network (target detector 1) and the student network at the same time. Formula (1-9) is used to make the student network output features similar to those of the teacher network. (1-7) and (1-8) are used to constrain the prediction results of the student network with the results of the teacher network and the true value respectively. When the above losses converge on the remote sensing images of the three modalities, that is, when formula (1-10) converges on the remote sensing images of the three modalities, this process ends, and the obtained student network is a model that is robust to the three modalities.

[0045] In general, this method designs a robust multimodal remote sensing image target detection method based on knowledge distillation, which can effectively perform target detection on multimodal remote sensing images.

Claims

1. A robust multimodal remote sensing image target detection method, characterized in that: Here are the steps: Three object detectors are trained using images of three modalities respectively; ResNet50 is used as the backbone network of three target detectors to extract features for target detection; During the network training phase, the loss functions of the three target detectors are the same. The difference lies in the input data. The loss function of any target detector is described as follows: The Facel Loss loss function is used to balance the loss function values ​​corresponding to positive and negative samples: Among them, N1 is the number of targets in a training batch, Y xyc is the true value of the image, is the network prediction value, γ and β are two hyperparameters of the network used to define the balance between positive and negative samples in the anchor box classification process; the anchor box regression branch actually predicts the position and size difference between the artificially defined anchor box and the true value; the true value anchor box is now assumed to be (x, y, h, w), and the four values ​​represent the horizontal coordinate, vertical coordinate, height and width of the anchor box position respectively; the artificially defined anchor box is The network needs to predict: The following loss function is used to supervise the network: Among them, N2 is the number of predicted boxes whose overlap with the jth true value box is greater than the threshold; is the network's prediction value for the i-th anchor box that meets the requirements. Meeting the requirements means that the overlap between the i-th and j-th true value boxes is greater than the threshold; t mj is the corresponding true value, (x j ,y j ,h j ,w j ) is the parameter of the jth true value anchor box, The parameters of the anchor box are artificially specified for the i-th person who meets the requirements; the total loss function of the network is: L=L class +εL loc (1-4) Among them, ε is used to balance the losses of the two tasks, L class and L loc is the loss function specified in equation (1-1) and equation (1-3); Three target detectors perform knowledge distillation to obtain a robust network A knowledge distillation-based induction method for multimodal remote sensing target detection tasks is proposed. It integrates the knowledge of different target detection networks trained on multiple source datasets and is a model enhanced by multiple target detectors from multiple sources. Therefore, only a single-modality target detection network is used to complete the prediction of any input source image. The purpose of knowledge distillation is to compress the complex teacher model knowledge into a simple and lightweight student model. The student model is trained by feeding the data into a pre-trained teacher model to generate soft labels. The student network selects the corresponding teacher network for learning based on the input. The process of image knowledge distillation for the three modalities is the same. The description for any modality is as follows: Assume that you have a dataset {a i ,b i }, i = 1, 2, ..., n, where a is the input image and b is the classification label; P t ,P s These are the classification prediction results of the teacher and student models respectively. The classification task optimization objectives of the student detection model during training are as follows: L cls =μL hard (P s ,b)+(1-μ)L soft (P s ,P t )(1-5) Among them, L hard is the hard loss between the student model prediction and the true value, L soft is the soft loss between the predictions of the student model and the teacher model; μ is a hyperparameter used to balance the soft and hard losses; the soft labels contain the potential relationships between different categories of targets discovered by the teacher model; by learning the soft labels, the student network inherits this hidden information; the detection problem needs to deal with the serious imbalance between different categories; therefore, class-weighted cross entropy is used as the classification distillation loss: L soft (P s ,P t )=-Σw c P t logP s (1-6) Among them, w c To balance the weights, a large weight is used for the target class and a small weight is used for the background class; P s and P t These are the target classification prediction results of the student model and the teacher model respectively; for the target detection network, there is also a branch used to locate and regress the target bounding box, that is, the classification branch, which also uses the true value and the prediction of the teacher model to calculate the border regression loss: L reg =L sL1 (R s , the reg ) + νL t (R s , R t , the reg )(1-7) Among them, y reg is the true coordinate point vector of the bounding box, R s ,R t are the regression prediction results of the student and teacher models respectively; ν is a hyperparameter used to balance the loss between the student model prediction results and the true value and the teacher model prediction results; L sL1 is the smoothed L1 loss; for the loss L t , described as follows: That is, it is considered that when the prediction error of the student and teacher models exceeds the discrimination boundary, the L2 loss is used for the true value; the prediction of the teacher model here is used as a judgment criterion rather than directly; where m is the discrimination boundary parameter; the teacher's intermediate performance is used to help the training process of the student model and improve the student's final performance; therefore, the features of the intermediate and terminal layers of the student network are constrained respectively; the L2 distance between the student and teacher models is used for constraint, and the feature constraint loss is described as follows: L f (F s ,F t )=||F s -F t ||2 (1-9) Among them, F s and F t The network loss function in the final distillation process of the feature graphs of the teacher network and the student network is: L d =λ1L cls +λ2L reg +λ3L f (F s ,F t )(1-10) Among them, λ1, λ2, and λ3 are used to balance the losses of the two tasks.

Images