A cross-modal image rotation object recognition method and system based on knowledge distillation
The cross-modal image rotation target recognition method addresses the challenge of missing modalities in remote sensing by using knowledge distillation to enhance the student network's detection accuracy and enable simultaneous infrared and visible light modality detection.
Patent Information
- Application Number
- CN202411259450.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-10
- Publication Date
- 2024-12-27
- Estimated Expiration
- 2044-09-10
AI Technical Summary
Multimodal remote sensing target detection systems face challenges in handling missing modalities due to hardware damage, data transmission failures, and electromagnetic interference, leading to increased complexity and difficulty in optimizing multiple classification heads when multiple modalities are missing.
A cross-modal image rotation target recognition method using knowledge distillation, where a complete multimodal teacher network model guides an incomplete multimodal student network model to learn key information, bridging the semantic gap between modalities and improving detection accuracy through feature and relationship distillation.
The method enhances the detection accuracy of the student network model by learning complete multimodal data distribution and key information, enabling efficient inference and cross-modal detection functions, particularly in scenarios with missing infrared or visible light modalities.
Smart Images
Figure CN118781331B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image processing, and in particular to a cross-modal image rotation target recognition method and system based on knowledge distillation. Background Art
[0002] With the development of multimodal technology, multimodal remote sensing target detection has gradually attracted widespread attention. Although multimodal remote sensing target detection technology has shown significant advantages in improving detection accuracy and robustness, it still faces many challenges in practical applications. For example, in the case of hardware damage, data transmission failure, and electromagnetic interference of the sensor, it is easy to cause a certain mode to be missing, making it impossible for the multimodal model to perform fusion detection. Regarding the problem of missing modes, it is mainly divided into the following three types: missing training set modes, missing test set modes, and missing training set and test set modes. In the actual deployment scenario of remote sensing target detection, there will be a situation where the test set mode is missing. For the situation where a certain mode is missing in complex and changeable scenarios, the existing technical solutions are: based on the generation method, using the existing modal information to generate the missing modal information, and then using the fusion of multiple modalities for detection. However, this method requires training a generation network for each missing mode. When the number of missing modes increases, its complexity also increases accordingly. The multi-task learning method uses multiple classification heads to distinguish the missing modalities and enhances the robustness of the model through multi-task learning. However, this method is difficult to optimize multiple classification heads when the number of missing modalities increases. Summary of the invention
[0003] Technical problem to be solved by the present invention: In view of the above-mentioned problems in the prior art, a cross-modal image rotation target recognition method and system based on knowledge distillation are provided. The present invention aims to use a complete multimodal teacher network model based on the knowledge distillation method to guide an incomplete multimodal student network model to learn the knowledge of the teacher model and improve the accuracy of the student network model. In the testing phase, only the student model is required for inference, with high computational efficiency, and can learn the complete multimodal data distribution and key information, thereby improving the detection accuracy of the student network model and having cross-modal detection function.
[0004] In order to solve the above technical problems, the technical solution adopted by the present invention is:
[0005] A cross-modal image rotation object recognition method based on knowledge distillation, comprising:
[0006] S101, constructing and training a teacher network model with infrared modality images and visible light modality images as input, wherein the teacher network model includes a multi-modal backbone network, a multi-scale aggregation layer, and a teacher network detection head, and freezing the network parameters of the teacher network model after the training is completed; constructing a student network model with visible light modality images / infrared modality images as input, wherein the student network model includes a single-modality backbone network, a multi-scale aggregation layer, and a student network detection head;
[0007] S102, using samples of visible light modal images to train the student network model, and at the same time inputting samples of visible light modal images and their corresponding infrared modal images into the teacher network model, using the multimodal backbone network and the features extracted from the multi-scale aggregation layer in the teacher network model to perform knowledge distillation and calculate the feature and relationship distillation losses to train and optimize the network parameters of the student network model, until the training of the student network model is completed for use in rotating target recognition of the input visible light modal image.
[0008] Optionally, in step S102, when performing knowledge distillation using the features extracted from the multimodal backbone network and the multiscale aggregation layer in the teacher network model and calculating the feature and relationship distillation loss, a loss function consisting of the sum of the channel distillation loss and the spatial distillation loss is used to optimize the network parameters of the single-modal backbone network of the student network model, and the function expression of the channel distillation loss is:
[0009] ,
[0010] The functional expression of the spatial distillation loss is:
[0011] ,
[0012] in, and are channel distillation loss and spatial distillation loss, MSE and KL are mean square error loss and Kullback-Leibler divergence, respectively. and are the channel distillation features of the teacher network model and the student network model, respectively. The channel distillation features are obtained by performing average pooling and normalization operations on the features of the corresponding scales in sequence; and They are the spatial distillation features of the teacher network model and the student network model, and the spatial distillation features are obtained by summing the features of the corresponding scales in the channel dimension and reshaping the tensors. The tensor and normalization operation of is the number of images in batch processing, is the number of channels, and before performing channel distillation features and spatial distillation features, including the features for the student network model, The convolution maps its channel dimension to the same dimension as the features of each scale output by each fusion module in the teacher network model.
[0013] Optionally, in step S102, when performing knowledge distillation using the features extracted from the multimodal backbone network and the multiscale aggregation layer in the teacher network model and calculating the feature and relationship distillation loss, a loss function composed of cosine similarity loss is used to optimize the network parameters of the multiscale aggregation layer of the student network model, and the function expression of the cosine similarity loss is:
[0014] ,
[0015] in, is the cosine similarity loss, To vectorize the expansion operation, and are the Gram matrices of the instance features of the teacher network model and the student network model, respectively, and have:
[0016] ,
[0017] ,
[0018] in, is a fused feature vector of the teacher network model, wherein the fused feature vector is a feature map of the region of interest obtained by extracting instance-level region of interest features from features of each scale output by each fusion module in the teacher network model, and expanding the region of interest feature map along the spatial dimension; is the feature vector of the student network model, which is a feature map of the region of interest obtained by extracting instance-level region of interest features from the feature output of each scale of the unimodal backbone network of the student network model, and expanding the region of interest feature map along the spatial dimension.
[0019] Optionally, extracting the instance-level region of interest features is extracting the instance-level region of interest features using a precise rotation region of interest pooling operator PRROI Pooling, including:
[0020] S201, dividing the rotation region of interest ROI in the features extracted from the multi-scale aggregation layer into The size of the block microelement, where is the side length of the preset block element;
[0021] S202, for Each block in the large and small blocks performs a local coordinate system traversal, and calculates the contribution of the four corner points in each block element based on the local coordinates of the block element:
[0022] ,
[0023] In the above formula, is the contribution of the i-th corner point, i=1,2,3,4, and are the local upper left corner coordinates and upper right corner coordinates of each block element, and are the continuous coordinates inside the infinitesimal element;
[0024] S203, converting the coordinates of the four corner points of each block element into the global coordinate system to obtain the global coordinates according to the following formula:
[0025] ,
[0026] In the above formula, ( are the global coordinates of the corner points, To rotate the ROI, are the local coordinates of the corner points, is the global coordinate center point of the rotation region of interest RO;
[0027] S204, calculate the characteristic output of each block element according to the following formula:
[0028] ,
[0029] In the above formula, is the characteristic output of the block element, is the global feature of the four corner points of the cube element, Represents the contribution of each corner point of the block element; and the average characteristics of the block element are obtained according to the following formula:
[0030] ,
[0031] In the above formula, is the average characteristic of the micro-element of the block, is the area of each square microelement;
[0032] S205, combining the average features of the block elements to obtain instance-level region of interest features.
[0033] Optionally, when training and optimizing the network parameters of the student network model in step S102, it also includes adopting a back propagation process of a precise rotation region of interest pooling operator PRROI Pooling and a coordinate back propagation process to make the extracted instance-level region of interest more accurate, wherein the back propagation process of the precise rotation region of interest pooling operator PRROI Pooling includes:
[0034] S301, dividing the rotation region of interest ROI in the features extracted from the multi-scale aggregation layer into The size of the block microelement, where is the side length of the preset block element;
[0035] S302, for Each block in the large and small blocks is traversed in the local coordinate system, and the contribution of the four corner points in each block element is calculated according to the local coordinates of the block element;
[0036] S303, converting the coordinates of the four corner points of each block element into the global coordinate system to obtain the global coordinates according to the following formula:
[0037] S304, multiplying the gradient of the loss function with respect to the features in the block element by the contribution of the corner point and then back-propagating to the global coordinates of the block element.
[0038] The coordinate back propagation process of the precise rotation region of interest pooling operator PRROI Pooling includes:
[0039] S401, dividing the rotation region of interest ROI in the features extracted from the multi-scale aggregation layer into The size of the block microelement, where is the side length of the preset block element;
[0040] S402, calculate partial derivatives in the Y-axis direction and the X-axis direction of the block element according to the following formula:
[0041] ,
[0042] ,
[0043] ,
[0044] ,
[0045] In the above formula, Represented in the feature map Upward square microelement Pooling of features within and are the local upper left corner coordinates and upper right corner coordinates of the block element, Extract the microelement features at a certain coordinate. represents a bilinear interpolation operation, and for any continuous local coordinates The expression for bilinear interpolation operation of the feature is:
[0046] ,
[0047] ,
[0048] In the above formula, is a continuous local coordinate The result of bilinear interpolation operation on the feature, Represented in continuous local coordinates Discrete coordinates near The interpolation coefficients calculated on To obtain the maximum value, In continuous local coordinates Nearby discrete coordinate points The characteristic response value of
[0049] S403, calculating each coordinate component of the rotating region of interest ROI according to the following formula The gradient contribution of is:
[0050] ,
[0051] ,
[0052] ,
[0053] ,
[0054] ,
[0055] In the above formula, are the coordinates of the block element, They are the coordinate components of the rotation region of interest ROI respectively The gradient contribution of and Represent the feature size after pooling respectively; according to the coordinate components of the rotating region of interest ROI The gradient contribution of the rotation region of interest ROI coordinate components center point, size and angle Perform gradient backpropagation, where is the center point of the coordinate components of the rotated region of interest ROI, To rotate the width and height of the region of interest ROI, is the angle to rotate the region of interest ROI.
[0056] Optionally, the multimodal backbone network includes an infrared backbone network for performing multi-scale feature extraction on infrared modality images, a visible light backbone network for performing multi-scale feature extraction on visible light modality images, and fusion modules for fusing features of the same scale extracted by the infrared backbone network and the visible light backbone network, respectively. The fusion module fuses the features of the same scale extracted by the infrared backbone network and the visible light backbone network, including: performing element-wise addition of features of corresponding scales extracted by the infrared backbone network and the visible light backbone network; and then performing global pooling, fully connected FC, sigmoid activation function activation, and multiplying the fused features obtained by element-wise addition. , extract the attention feature map through the attention operation, and activate it through convolution and sigmoid activation function to obtain the attention weight x; then multiply the attention weight x and the fused feature obtained by element-level addition to obtain the infrared missing feature, and multiply 1-x and the fused feature obtained by element-level addition to obtain the visible light missing feature; finally, add the infrared missing feature and the features of the corresponding scale extracted by the infrared backbone network at the element level, and add the visible light missing feature and the features of the corresponding scale extracted by the visible light backbone network at the element level, and then add the two added features and activate them through convolution and Relu activation function to obtain the fusion layer of the input first multi-scale aggregation layer.
[0057] Optionally, the teacher network detection head includes two cascaded fully connected layers and a classification branch and a positioning branch located at the back end of the two fully connected layers, wherein the classification branch is A fully connected layer of size is the hidden layer parameter, C is the number of categories, and the positioning branch is The student network detection head includes a classification branch and a positioning branch, the classification branch includes 4 convolutional layers, and the positioning branch includes 4 convolutional layers.
[0058] In addition, the present invention also provides a cross-modal image rotation target recognition system based on knowledge distillation, including a microprocessor and a memory connected to each other, and the microprocessor is programmed or configured to execute the cross-modal image rotation target recognition method based on knowledge distillation.
[0059] In addition, the present invention also provides a computer-readable storage medium, which stores a computer program / instruction, and the computer program / instruction is programmed or configured to execute the cross-modal image rotation target recognition method based on knowledge distillation through a processor.
[0060] In addition, the present invention also provides a computer program product, including a computer program / instruction, which is programmed or configured to execute the cross-modal image rotation target recognition method based on knowledge distillation through a processor.
[0061] Compared with the prior art, the present invention mainly has the following advantages: in view of the lack of infrared modality in remote sensing images, the cross-modal image rotation target recognition method based on knowledge distillation of the present invention includes a method based on knowledge distillation, using a complete multimodal teacher network model to guide an incomplete multimodal student network model to learn the knowledge of the teacher model, improve the accuracy of the student network model, and only need the student model to perform inference in the test phase, with high computational efficiency, and can learn the data distribution and key information of the complete multimodality, make up for the semantic gap between the fusion features of infrared and visible light modalities and the features of visible light modalities, improve the detection accuracy of the student network model, and have a cross-modal detection function. For example, when a student network model with a visible light modality image as input is constructed, the detection accuracy of the visible light modality image of the student network model can be improved, and the infrared modality detection function can be simultaneously possessed; when a student network model with an infrared modality image as input is constructed, the detection accuracy of the infrared modality image of the student network model can be improved, and the visible light modality detection function can be simultaneously possessed. BRIEF DESCRIPTION OF THE DRAWINGS
[0062] Figure 1 Schematic diagram of the basic process flow across visible light modalities of the method according to an embodiment of the present invention.
[0063] Figure 2 Schematic diagram of the structure of the teacher network model in an embodiment of the present invention.
[0064] Figure 3 Schematic diagram of the structure of the student network model in the embodiment of the present invention.
[0065] Figure 4 Schematic diagram of the structure of the fusion module in the teacher network model in an embodiment of the present invention.
[0066] Figure 5 Schematic diagram of the knowledge distillation principle when training the student network model in an embodiment of the present invention.
[0067] Figure 6 Schematic diagram of the extraction process of channel distillation features in an embodiment of the present invention.
[0068] Figure 7 Schematic diagram of the extraction process of spatial distillation features in an embodiment of the present invention.
[0069] Figure 8 Schematic diagram of the principle of the precise rotation region of interest pooling operator PRROI Pooling in an embodiment of the present invention, wherein (a) is a schematic diagram of dividing the rotation region of interest ROI into square elements, and (b) is a schematic diagram of the coordinates of the upper left corner and the upper right corner of the square element in the local coordinate system.
[0070] Fig. 9The figure is a schematic diagram of the principle of constructing the cosine similarity loss in an embodiment of the present invention. DETAILED DESCRIPTION
[0071] like Figure 1 As shown, the cross-modal image rotation target recognition method based on knowledge distillation in this embodiment includes:
[0072] S101, construct and train a teacher network model with infrared modality images and visible light modality images as input, such as Figure 2 As shown in , the teacher network model includes a multimodal backbone network, a multi-scale aggregation layer, and a teacher network detection head. After training, the network parameters of the teacher network model are frozen; a student network model with visible light modality images / infrared modality images as input is constructed, as shown in Figure 1 As shown, in this embodiment, a student network model is specifically constructed with a visible light modality image as input. In addition, a student network model with an infrared modality image as input can also be constructed, such as Figure 3 As shown, the student network model includes a single-modal backbone network, a multi-scale aggregation layer, and a student network detection head;
[0073] S102, using samples of visible light modal images to train the student network model, and at the same time inputting samples of visible light modal images and their corresponding infrared modal images into the teacher network model, using the multimodal backbone network and the features extracted from the multi-scale aggregation layer in the teacher network model to perform knowledge distillation and calculate the feature and relationship distillation losses to train and optimize the network parameters of the student network model, until the training of the student network model is completed for use in rotating target recognition of the input visible light modal image.
[0074] like Figure 2 As shown, the multimodal backbone network includes an infrared backbone network for extracting multi-scale features from infrared modality images, a visible light backbone network for extracting multi-scale features from visible light modality images, and a fusion module for fusing features of the same scale extracted from the infrared backbone network and the visible light backbone network. Figure 2 As shown in Figure 2, the teacher network detection head includes two cascaded fully connected layers and a classification branch and a localization branch located at the back end of the two fully connected layers. The classification branch is A fully connected layer of size is the hidden layer parameter, C is the number of categories, and the positioning branch is The fully connected layer has a size of , which includes 5 positioning parameters.
[0075] In this embodiment, the RetinaNet-O or Focus-O network model based on the anchor frame mechanism or anchor point mechanism is used as the student network model, and the visible light modality is used as input. Figure 3As shown in the figure, the student network detection head includes a classification branch and a localization branch. The classification branch contains 4 convolutional layers, and the localization branch includes 4 convolutional layers.
[0076] The teacher network model uses LSKNet with a dual-branch architecture as the feature extractor for infrared and visible light to generate features for each modality. Figure 4 As shown, the fusion module fuses the features of the same scale extracted by the infrared backbone network and the visible light backbone network, including: adding the features of the corresponding scales extracted by the infrared backbone network and the visible light backbone network at the element level; then sequentially undergoing global pooling, fully connected FC, sigmoid activation function activation, multiplying the fused features obtained by element-level addition, extracting the attention feature map through the attention operation, and activating the sigmoid activation function to obtain the attention weight x; then multiplying the attention weight x and the fused features obtained by element-level addition to obtain the infrared missing feature, and multiplying 1-x and the fused features obtained by element-level addition to obtain the visible light missing feature; finally, adding the infrared missing feature and the features of the corresponding scale extracted by the infrared backbone network at the element level, adding the visible light missing feature and the features of the corresponding scale extracted by the visible light backbone network at the element level, and then adding the two added features and activating them through convolution and Relu activation function to obtain the fusion layer input into the first multi-scale aggregation layer. The fusion module of multi-modal fusion mainly performs progressive fusion of the features of visible light and infrared modalities. The specific fusion operations are as follows: first, the visible light modal features are element-wise added with the infrared modal features; then, the feature is subjected to an attention operation similar to SENet to obtain the attention weight w, and the attention weight w is multiplied with the fused feature to obtain the infrared missing feature, and the 1-w attention weight is multiplied with the fused feature to obtain the visible light missing feature; finally, each modality makes up for the missing features, and the final fused feature is obtained by adding the features of the two complete modalities.
[0077] like Figure 5 As shown, in step S102 of this embodiment, when using the multimodal backbone network in the teacher network model and the features extracted from the multi-scale aggregation layer to perform knowledge distillation and calculate the feature and relationship distillation losses, the loss function consisting of the sum of the channel distillation loss and the spatial distillation loss is used to optimize the network parameters of the single-modal backbone network of the student network model. The function expression of the channel distillation loss is:
[0078] ,
[0079] The functional expression of the spatial distillation loss is:
[0080] ,
[0081] in, and are channel distillation loss and spatial distillation loss, MSE and KL are mean square error loss and Kullback-Leibler divergence, respectively. and They are the channel distillation features of the teacher network model and the student network model, respectively. Figure 6 As shown, the channel distillation feature is obtained by performing average pooling and normalization operations on the features of the corresponding scales in sequence; and The spatial distillation features of the teacher network model and the student network model are respectively Figure 7 As shown in the figure, the spatial distillation feature is obtained by summing the features of the corresponding scales in the channel dimension and reshaping the tensor. The tensor and normalization operation of is the number of images in batch processing, is the number of channels, and before performing channel distillation features and spatial distillation features, including the features for the student network model, The convolution maps its channel dimension to the same dimension as the features of each scale output by each fusion module in the teacher network model. The teacher network model adopts a dual-branch dual-stage network structure. The fusion module of the multimodal backbone network can extract the teacher fusion features of infrared and visible light images, which are recorded as The student network model adopts a single-stage network structure, and the single-mode backbone network can extract student features. ; Implement feature distillation on the teacher-student network and convert the student features pass Convolution maps its channel dimension to fusion features with the teacher The same dimension is used to perform feature distillation of channel dimension and spatial dimension respectively.
[0082] In addition, if Figure 5 and Fig. 9 As shown, in step S102 of this embodiment, when using the multimodal backbone network and the features extracted from the multi-scale aggregation layer in the teacher network model to perform knowledge distillation and calculate the feature and relationship distillation loss, it includes using the loss function composed of cosine similarity loss to optimize the network parameters of the multi-scale aggregation layer of the student network model. The function expression of the cosine similarity loss is:
[0083] ,
[0084] in, is the cosine similarity loss, To vectorize the expansion operation, and are the Gram matrices of the instance features of the teacher network model and the student network model, respectively, and have:
[0085] ,
[0086] ,
[0087] in, is a fused feature vector of the teacher network model, wherein the fused feature vector is a feature map of the region of interest obtained by extracting instance-level region of interest features from features of each scale output by each fusion module in the teacher network model, and expanding the region of interest feature map along the spatial dimension; is the feature vector of the student network model, which is a feature map of the region of interest obtained by extracting instance-level region of interest features from the feature output of each scale of the unimodal backbone network of the student network model, and expanding the region of interest feature map along the spatial dimension.
[0088] In this embodiment, the instance-level ROI features are extracted by using the precise rotation ROI pooling operator PRROI Pooling to extract the instance-level ROI features, and the precise rotation ROI pooling operator PRROIPooling is used to perform instance-level relationship distillation on the backbone network features of the teacher and student network models. and Extract instance-level target features, calculate the Gram matrix of the teacher network and the student network, and construct the cosine similarity loss to optimize the student network. Specifically, Figure 8 As shown in the figure, the precise rotation region of interest pooling operator PRROIPooling is used to extract instance-level region of interest features including:
[0089] S201, dividing the rotation region of interest ROI in the features extracted from the multi-scale aggregation layer into The size of the block microelement, where is the side length of the preset block element, such as Figure 8 As shown in (a);
[0090] S202, for Each block in the large and small blocks performs a local coordinate system traversal, and calculates the contribution of the four corner points in each block element based on the local coordinates of the block element:
[0091] ,
[0092] In the above formula, is the contribution of the i-th corner point, i=1,2,3,4, and are the local upper left corner coordinates and upper right corner coordinates of each block element, and are the continuous coordinates inside the infinitesimal element;
[0093] S203, converting the coordinates of the four corner points of each block element into the global coordinate system to obtain the global coordinates according to the following formula:
[0094] ,
[0095] In the above formula, ( are the global coordinates of the corner points, To rotate the ROI, are the local coordinates of the corner points, The global coordinate center point of the rotated region of interest ROI;
[0096] S204, calculate the characteristic output of each block element according to the following formula:
[0097] ,
[0098] In the above formula, is the characteristic output of the block element, is the global feature of the four corner points of the cube element, Represents the contribution of each corner point of the block element; and the average characteristics of the block element are obtained according to the following formula:
[0099] ,
[0100] In the above formula, is the average characteristic of the micro-element of the block, is the area of each square microelement;
[0101] S205, combining the average features of the block elements to obtain instance-level region of interest features.
[0102] When training and optimizing the network parameters of the student network model in step S102 of this embodiment, it also includes using the back propagation process of the precise rotation region of interest pooling operator PRROI Pooling and the coordinate back propagation process to make the extracted instance-level region of interest more accurate, wherein the back propagation process of the precise rotation region of interest pooling operator PRROI Pooling includes:
[0103] S301, dividing the rotation region of interest ROI in the features extracted from the multi-scale aggregation layer into The size of the block microelement, where is the side length of the preset block element;
[0104] S302, for Each block in the large and small blocks is traversed in the local coordinate system, and the contribution of the four corner points in each block element is calculated according to the local coordinates of the block element;
[0105] S303, converting the coordinates of the four corner points of each block element into the global coordinate system to obtain the global coordinates according to the following formula:
[0106] S304, multiplying the gradient of the loss function with respect to the features in the block element by the contribution of the corner point and then back-propagating to the global coordinates of the block element.
[0107] The coordinate back propagation process of the precise rotation region of interest pooling operator PRROI Pooling includes:
[0108] S401, dividing the rotation region of interest ROI in the features extracted from the multi-scale aggregation layer into The size of the block microelement, where is the side length of the preset block element;
[0109] S402, calculate partial derivatives in the Y-axis direction and the X-axis direction of the block element according to the following formula:
[0110] ,
[0111] ,
[0112] ,
[0113] ,
[0114] In the above formula, Represented in the feature map Upward square microelement Pooling of features within and are the local upper left corner coordinates and upper right corner coordinates of the block element, such as Figure 8 As shown in (b), Extract the microelement features at a certain coordinate. represents a bilinear interpolation operation, and for any continuous local coordinates The expression for bilinear interpolation operation of the feature is:
[0115] ,
[0116] ,
[0117] In the above formula, is a continuous local coordinate The result of bilinear interpolation operation on the feature, Represented in continuous local coordinates Discrete coordinates near The interpolation coefficients calculated on To obtain the maximum value, In continuous local coordinates Nearby discrete coordinate points The characteristic response value of
[0118] S403, calculating each coordinate component of the rotating region of interest ROI according to the following formula The gradient contribution of is:
[0119] ,
[0120] ,
[0121] ,
[0122] ,
[0123] ,
[0124] In the above formula, are the coordinates of the block element, They are the coordinate components of the rotation region of interest ROI respectively The gradient contribution of and Represent the feature size after pooling respectively; according to the coordinate components of the rotating region of interest ROI The gradient contribution of the rotation region of interest ROI coordinate components center point, size and angle Perform gradient backpropagation, where is the center point of the coordinate components of the rotated region of interest ROI, To rotate the width and height of the region of interest ROI, is the angle to rotate the region of interest ROI.
[0125] In order to verify the effectiveness of the cross-modal image rotation target recognition method based on knowledge distillation in this embodiment, experiments were conducted on the VEDAI dataset in this embodiment, and the VEDAI dataset, a public dataset released by the University of Caen in 2015, was used for training and testing, and compared with the Focus-O anchor mechanism and RetinaNet-O anchor frame mechanism methods. The VEDAI dataset includes two size data of 512×512 and 1024×1024, each of which contains two images of visible light and near infrared, including a total of 9 categories, all of which are rotated frame annotations. This experiment uses the AdamW optimizer, with an initial learning rate of 0.0001, a batch size of 2, 36 rounds of training, a learning rate of 0.00001 in the 25th round, and a learning rate of 0.000001 in the 34th round. The single-stage method Fcos-O (Fully Convolutional One-Stage Object Detection) and RetinaNet-O (Focal Loss for Dense Object Detection) are selected for experimental comparison. The teacher network model has a multimodal test accuracy of 83.30% on the VEDAI dataset.
[0126] Table 1 Effectiveness of the method in this embodiment on the single-stage anchor mechanism detector RetinaNet-O and anchor mechanism detector Fcos-O on the VEDAI dataset (%)
[0127]
[0128] Table 1 compares the recognition accuracy and mAP50 index of the single-stage anchor mechanism detector RetinaNet-O and the anchor mechanism detector Fcos-O in 9 categories under the cross-modal knowledge distillation scheme, where RGB is the original single-modal detection method for visible light modality images, IR is the original single-modal detection method for infrared modality images, and Cross-RGB and Cross-IR are student network models constructed in this embodiment with visible light modality images and infrared modality images as input. As shown in Table 1, Cross-RGB and Cross-IR based on the method of this embodiment are both improved in mAP50 compared with the original model RGB and IR modalities. RetinaNet-O improved by 4.20% and 5.14%, and Fcos-O improved by 2.35% and 3.11%, proving the superiority of the cross-modal image rotation target recognition method based on knowledge distillation in this embodiment.
[0129] In summary, in order to solve the problem of missing infrared modalities in remote sensing images, the cross-modal image rotation target recognition method based on knowledge distillation in this embodiment bridges the semantic gap between the fusion features of infrared and visible light modalities and the features of visible light modalities by realizing the feature distillation of the backbone network and the instance-level relationship distillation between the teacher and student network models, as well as the instance-level feature distillation of the feature graphs with the same resolution. At the same time, the student network learns the data distribution and instance-level relationship information of the fusion features of the teacher network, improves the detection accuracy of the student network, and enables the student network to detect infrared and visible light modalities at the same time.
[0130] In addition, this embodiment also provides a cross-modal image rotation target recognition system based on knowledge distillation, including a microprocessor and a memory connected to each other, and the microprocessor is programmed or configured to execute the cross-modal image rotation target recognition method based on knowledge distillation.
[0131] In addition, this embodiment also provides a computer-readable storage medium, which stores a computer program / instruction, and the computer program / instruction is programmed or configured to execute the cross-modal image rotation target recognition method based on knowledge distillation through a processor.
[0132] In addition, this embodiment also provides a computer program product, including a computer program / instruction, which is programmed or configured to execute the cross-modal image rotation target recognition method based on knowledge distillation through a processor.
[0133] Those skilled in the art should understand that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application can take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the process Figure 1 A process or multiple processes and / or boxes Figure 1These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0134] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.
Claims
1. A cross-modal image rotation object recognition method based on knowledge distillation, characterized in that: include: S101, constructing and training a teacher network model with infrared modality images and visible light modality images as input, wherein the teacher network model includes a multi-modal backbone network, a multi-scale aggregation layer, and a teacher network detection head, and freezing the network parameters of the teacher network model after the training is completed; constructing a student network model with visible light modality images / infrared modality images as input, wherein the student network model includes a single-modality backbone network, a multi-scale aggregation layer, and a student network detection head; S102, using samples of visible light modal images to train the student network model, and at the same time inputting samples of visible light modal images and their corresponding infrared modal images into the teacher network model, using the features extracted from the multimodal backbone network and the multiscale aggregation layer in the teacher network model to perform knowledge distillation and calculate the feature and relationship distillation losses to train and optimize the network parameters of the student network model, until the training of the student network model is completed for use in rotating target recognition of the input visible light modal image; In step S102, when the multimodal backbone network and the features extracted from the multiscale aggregation layer in the teacher network model are used to perform knowledge distillation and calculate the feature and relationship distillation losses, a loss function consisting of the sum of the channel distillation loss and the spatial distillation loss is used to optimize the network parameters of the single-modal backbone network of the student network model. The function expression of the channel distillation loss is: ， The functional expression of the spatial distillation loss is: ， in, and are channel distillation loss and spatial distillation loss, MSE and KL are mean square error loss and Kullback-Leibler divergence, respectively. and are the channel distillation features of the teacher network model and the student network model, respectively. The channel distillation features are obtained by performing average pooling and normalization operations on the features of the corresponding scales in sequence; and They are the spatial distillation features of the teacher network model and the student network model, and the spatial distillation features are obtained by summing the features of the corresponding scales in the channel dimension and reshaping the tensors. The tensor and normalization operation of is the number of images in batch processing, is the number of channels, and before performing channel distillation features and spatial distillation features, including the features for the student network model, The convolution maps its channel dimension to the same dimension as the features of each scale output by each fusion module in the teacher network model; In step S102, when the features extracted from the multimodal backbone network and the multiscale aggregation layer in the teacher network model are used for knowledge distillation and the feature and relationship distillation losses are calculated, the network parameters of the multiscale aggregation layer of the student network model are optimized using a loss function composed of cosine similarity loss. The function expression of the cosine similarity loss is: ， in, is the cosine similarity loss, To vectorize the expansion operation, and are the Gram matrices of the instance features of the teacher network model and the student network model, respectively, and have: ， ， in, is a fused feature vector of the teacher network model, wherein the fused feature vector is a feature map of the region of interest obtained by extracting instance-level region of interest features from features of each scale output by each fusion module in the teacher network model, and expanding the region of interest feature map along the spatial dimension; is a feature vector of the student network model, which is a feature map of the region of interest obtained by extracting instance-level region of interest features from the features of each scale of the unimodal backbone network output of the student network model, and expanding the region of interest feature map along the spatial dimension; The method of extracting instance-level region of interest features is to use a precise rotation region of interest pooling operator PRROI Pooling to extract instance-level region of interest features, including: S201, dividing the rotation region of interest ROI in the features extracted from the multi-scale aggregation layer into The size of the block microelement, where is the side length of the preset block element; S202, for Each block in the large and small blocks performs a local coordinate system traversal, and calculates the contribution of the four corner points in each block element based on the local coordinates of the block element: ， In the above formula, is the contribution of the i-th corner point, i=1,2,3,4, and are the local upper left corner coordinates and upper right corner coordinates of each block element, and are the continuous coordinates inside the infinitesimal element; S203, converting the coordinates of the four corner points of each block element into the global coordinate system to obtain the global coordinates according to the following formula: ， In the above formula, ( are the global coordinates of the corner points, To rotate the ROI, are the local coordinates of the corner points, The global coordinate center point of the rotated region of interest ROI; S204, calculate the characteristic output of each block element according to the following formula: ， In the above formula, is the characteristic output of the block element, is the global feature of the four corner points of the cube element, Represents the contribution of each corner point of the block element; and the average characteristics of the block element are obtained according to the following formula: ， In the above formula, is the average characteristic of the micro-element of the block, is the area of each square microelement; S205, combining the average features of the block elements to obtain instance-level region of interest features.
2. The cross-modal image rotation object recognition method based on knowledge distillation according to claim 1, characterized in that: When training and optimizing the network parameters of the student network model in step S102, the back propagation process of the precise rotation region of interest pooling operator PRROI Pooling and the coordinate back propagation process are also included to make the extracted instance-level region of interest more accurate, wherein the back propagation process of the precise rotation region of interest pooling operator PRROI Pooling includes: S301, dividing the rotation region of interest ROI in the features extracted from the multi-scale aggregation layer into The size of the block microelement, where is the side length of the preset block element; S302, for Each block in the large and small blocks is traversed in the local coordinate system, and the contribution of the four corner points in each block element is calculated according to the local coordinates of the block element; S303, converting the coordinates of the four corner points of each block element into the global coordinate system to obtain the global coordinates according to the following formula: S304, multiplying the gradient of the loss function with respect to the feature in the block element by the contribution of the corner point and then back-propagating to the global coordinates of the block element; The coordinate back propagation process of the precise rotation region of interest pooling operator PRROI Pooling includes: S401, dividing the rotation region of interest ROI in the features extracted from the multi-scale aggregation layer into The size of the block microelement, where is the side length of the preset block element; S402, calculate partial derivatives in the Y-axis direction and the X-axis direction of the block element according to the following formula: ， ， ， ， In the above formula, Represented in the feature map Upward square microelement Pooling of features within and are the local upper left corner coordinates and upper right corner coordinates of the block element, Extract the microelement features at a certain coordinate. represents a bilinear interpolation operation, and for any continuous local coordinates The expression for bilinear interpolation operation of the feature is: ， ， In the above formula, is a continuous local coordinate The result of bilinear interpolation operation on the feature, Represented in continuous local coordinates Discrete coordinates near The interpolation coefficients calculated on To obtain the maximum value, In continuous local coordinates Nearby discrete coordinate points The characteristic response value of S403, calculating each coordinate component of the rotating region of interest ROI according to the following formula Gradient contribution of: ， ， ， ， ， In the above formula, are the coordinates of the block element, They are the coordinate components of the rotation region of interest ROI respectively The gradient contribution of and Represent the feature size after pooling respectively; according to the coordinate components of the rotating region of interest ROI The gradient contribution of the rotation region of interest ROI coordinate components center point, size and angle Perform gradient backpropagation, where is the center point of the coordinate components of the rotated region of interest ROI, To rotate the width and height of the region of interest ROI, is the angle to rotate the region of interest ROI.
3. The cross-modal image rotation object recognition method based on knowledge distillation according to claim 1 is characterized in that: The multimodal backbone network includes an infrared backbone network for extracting multi-scale features from infrared modal images, a visible light backbone network for extracting multi-scale features from visible light modal images, and fusion modules for fusing features of the same scale extracted from the infrared backbone network and the visible light backbone network, respectively. The fusion module fuses the features of the same scale extracted from the infrared backbone network and the visible light backbone network by: adding the features of the corresponding scales extracted from the infrared backbone network and the visible light backbone network at the element level; then performing global pooling, fully connected FC, sigmoid activation function activation, multiplying the fused features obtained by element-wise addition, and passing through The attention feature map is extracted through the attention operation, and the attention weight x is obtained by activation through convolution and sigmoid activation function; the infrared missing feature is multiplied by the fused feature obtained by element-wise addition of the attention weight x, and the visible light missing feature is multiplied by the fused feature obtained by element-wise addition of 1-x; finally, the infrared missing feature and the features of the corresponding scale extracted by the infrared backbone network are added element-wise, and the visible light missing feature and the features of the corresponding scale extracted by the visible light backbone network are added element-wise, and the two added features are added and activated by convolution and Relu activation function to obtain the fusion layer of the first multi-scale aggregation layer.
4. The cross-modal image rotation object recognition method based on knowledge distillation according to claim 3 is characterized in that: The teacher network detection head includes two cascaded fully connected layers and a classification branch and a positioning branch located at the back end of the two fully connected layers. The classification branch is A fully connected layer of size is the hidden layer parameter, C is the number of categories, and the positioning branch is The student network detection head includes a classification branch and a positioning branch, the classification branch includes 4 convolutional layers, and the positioning branch includes 4 convolutional layers.
5. A cross-modal image rotation object recognition system based on knowledge distillation, comprising a microprocessor and a memory connected to each other, characterized in that: The microprocessor is programmed or configured to execute the cross-modal image rotation target recognition method based on knowledge distillation as described in any one of claims 1 to 4.
6. A computer-readable storage medium having a computer program / instruction stored therein, characterized in that: The computer program / instruction is programmed or configured to execute the cross-modal image rotation target recognition method based on knowledge distillation as described in any one of claims 1 to 4 through a processor.
7. A computer program product comprising a computer program / instructions, characterized in that The computer program / instruction is programmed or configured to execute the cross-modal image rotation target recognition method based on knowledge distillation as described in any one of claims 1 to 4 through a processor.