A multimodal vehicle target detection framework based on mutual distillation attribute fusion network

Abstract

The present invention belongs to the field of computer vision technology, and discloses a multimodal vehicle target detection framework based on a mutual distillation attribute fusion network. First, a cross-modal mutual distillation module based on optimal transmission theory is proposed, which utilizes the complementary characteristics of infrared images and visible light images, and suppresses multimodal noise interference through cross-modal spatial knowledge mutual distillation; secondly, an attribute decomposition module is proposed, which utilizes the attribute routing mechanism to decompose the mixed multimodal global features into independent attribute features, and ensures the diversity of attribute features by maximizing attribute entropy; finally, an attribute graph fusion module is proposed, which independently fuses the corresponding attribute features of different modalities, and uses a personalized attribute graph customized for each category in the feature fusion process to model the association between various target attribute features, thereby improving the adaptability of the model to targets of different categories.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular to a multimodal vehicle target detection framework based on a mutual distillation attribute fusion network. Background Art

[0002] As one of the core tasks of autonomous driving, vehicle target detection is the key to the intelligent scene perception of the autonomous driving system. Visible light images can obtain rich details and texture information of the target, which can help the car perceive the surrounding vehicles and personnel in real time, and provide visual support for the autonomous driving strategy. However, visible light images often cannot provide sufficient perception information for the vehicle under low light conditions, so relying only on a single visible light image will lead to safety problems. Inspired by human multimodal perception, researchers began to focus on multimodal vehicle target detection. Among them, infrared images can greatly improve the adaptability of autonomous driving systems to different environments by combining them with visible light images due to their robust perception capabilities in different environments. Therefore, how to use infrared images and visible light images to improve the scene perception capabilities of autonomous driving systems has become the focus of researchers.

[0003] In order to make full use of the complementary properties of infrared images and visible light images, the current mainstream solution is multimodal feature fusion, that is, to establish feature interactions between images of different modalities in the feature space to achieve multimodal feature fusion, and under the guidance of supervisory information, encourage the model to learn a multimodal complementary fusion mode that can minimize the loss of target recognition and target positioning. However, the global coarse weighting strategy ignores the fine local spatial structure of multimodal features. Therefore, some researchers proposed a cross-modal feature fusion and interaction module to more fully capture the complex associations between multimodal features. Inspired by Transformer, researchers designed a multimodal cross-attention module, which captures multimodal complementary associations by learning cross-modal global attention, thereby improving the expressive power of multimodal fusion features.

[0004] Although the above methods have achieved impressive performance, the existence of multimodal noise interference and attribute mixing problems hinders the further improvement of the performance of these methods. Summary of the invention

[0005] The technical problem to be solved by the present invention is the multimodal noise interference problem and the attribute mixing problem; the present invention proposes a multimodal vehicle target detection framework based on a mutual distillation attribute fusion network, including a backbone network, a classifier, a feature decoder, an attribute decomposition module, an attribute graph fusion module and a prediction head;

[0006] Input the infrared image and visible light image into the backbone network to obtain the feature maps of the two modalities;

[0007] Input the classifier to get the bimodal attention map Among them, c is the number of categories, h is the height of the graph, and w is the width of the graph;

[0008] For the two-modal attention map M v and M I Perform cross-modal spatial knowledge mutual distillation to obtain distilled infrared features and visible light features, which are then input into the feature decoder to obtain the query Token Q;

[0009] The query Token Q is then input into the attribute decomposition module to obtain the infrared image attribute features and the visible light image attribute features;

[0010] The infrared image attribute features and the visible light image attribute features are fused in the attribute graph fusion module to obtain the attribute fusion feature A F ,

[0011] Fusion of attributes into features A F As a node set of the attribute graph, obtain the multimodal query sequence M F ;

[0012] The multimodal query sequence M F Input the prediction head to obtain visible light image prediction and infrared image prediction.

[0013] Furthermore, the multimodal vehicle target detection framework based on the mutual distillation attribute fusion network of the present invention also includes a feature encoder; used to perform feature enhancement on the feature maps of the two modes output by the backbone network to obtain a feature enhancement sequence and represents the visible image enhancement feature enhancement sequence, represents the infrared image enhancement feature enhancement sequence;

[0014] The enhanced feature sequences of the two modalities and Then input it into the classifier for classification.

[0015] Furthermore, by using the optimal transfer theory, we can quantize the attention map To the attention map The minimum transmission cost of the two modal attention maps is described, and the transmission matrix T = [t ij ] and the distance matrix D = [d ij ]; where t ij Indicates from arrive The transmission weight, d ij Quantified and The distance between Less than , the visible light mode has a stronger representation ability at this point, so at this time d ij is larger, the model does not encourage the attention map of the visible light modality at this point to approach the infrared modality. Greater than , the infrared mode has a stronger characterization ability at this point. ij The model encourages the attention map of the visible light modality at this point to be close to the infrared modality. In this way, the student modality can learn the spatial knowledge of the teacher modality during the knowledge distillation process, ensuring that the model captures cross-modal complementary knowledge while filtering out inter-modal information interference.

[0016] The inter-distillation loss L is designed MD , infrared and visible light images are used as teacher modalities and student modalities respectively, and the model's learning ability of cross-modal complementary knowledge is improved through cross-modal knowledge mutual distillation. Considering that the importance of the visible light modality often changes with scene changes, a scene-aware weight generation module is designed to adaptively adjust the loss value according to the current scene.

[0017] L MD =OT(M V ,M I )+γOT(M I ,M V )

[0018] Among them, γ is the weight coefficient, and OT(·) is the optimal transmission distance for distillation.

[0019] γ=WAM(P)

[0020] Where P is the scene perception vector, which is obtained by global pooling of the feature map of the visible light image, and WAM(·) is the weighted perception module.

[0021] Furthermore, the attribute decomposition module includes K capsule mapping layers, L attribute capsules S l and L groups of dynamic routing weights;

[0022] L attribute capsule S l Corresponding to the dynamic routing weight of group L;

[0023] For the attribute capsule S l Initialize a set of dynamic routing weights And use the following rules to weight dynamic routing and property capsule S l To update, the update process of the tth time step is expressed as:

[0024]

[0025] Among them, Squash(·) is the activation function, and its main purpose is to normalize the output of the attribute capsule. is the multimodal attribute feature at the t-th time step, is the weight obtained by Softmax of the dynamic routing weight at the tth time step, and ⊙ is the scalar product. The bottom-level capsule passes its output to the upper-level attribute capsule through the weight matrix. This process essentially projects the output of the bottom-level capsule into different attribute spaces. When the output of the bottom-level capsule has a large scalar product with the output of a certain attribute capsule, it means that the output of the bottom-level capsule has a strong representation ability in the attribute space.

[0026] Furthermore, in order to enhance the diversity of attribute capsules, the concept of attribute entropy is introduced to encourage each attribute capsule to learn different features and improve its feature expression ability. The attribute entropy H(S) is defined as:

[0027]

[0028] Among them, p(S l,j ) indicates S l In the probability distribution of the jth dimension, in the present invention, Softmax is used to obtain S l The probability distribution p(S l ).

[0029] The K capsule mapping layers map the query Token Q to K underlying capsules, where the kth underlying capsule is represented as:

[0030] C k =CML k (Q)

[0031] Among them, CML k (·) is the kth capsule mapping layer;

[0032] Furthermore, the attribute is fused with feature A F As a node set of the attribute graph, obtain the multimodal query sequence M F , specifically:

[0033] Set a learnable edge structure for each class based on the pseudo-label of the query Token Q to build an attribute graph;

[0034] The attribute graph of the cth class is represented as in, represents the edge set, Represents the attribute fusion feature of the cth category;

[0035] Using a set of learnable vectors Opposite edge set Row representation, learnable vector W E It encodes the association between attributes, and D is the edge feature dimension;

[0036] Afterwards, the information transfer mechanism of the graph neural network is used to update the attribute graph nodes by capturing the associations between the attributes;

[0037] Use global average pooling to integrate the node features of the attribute graph to obtain multimodal fusion queries Multimodal Fusion Query is a multimodal query sequence M F The elements in .

[0038] Furthermore, the multimodal query sequence M F Input the prediction head, which includes the target recognition head and the target positioning head, to obtain the query sequence M F The category prediction and position prediction corresponding to each element in are obtained to obtain the visible light image prediction and infrared image prediction.

[0039] The predicted target recognition loss is calculated based on the category prediction and the true class label, and Focal Loss is used as the target recognition loss to strengthen the model's attention to difficult targets;

[0040] L1 loss and GIoU loss are used as target positioning losses to enhance the detection performance of the model in complex scenes. The GIoU metric improves the IoU by considering the minimum closure area of ​​the predicted box and the true box, which can provide a more stable gradient for model learning.

[0041] Beneficial effects: The present invention provides a multimodal vehicle target detection framework based on a mutual distillation attribute fusion network; first, a cross-modal mutual distillation module based on optimal transmission theory is proposed, which utilizes the complementary characteristics of infrared images and visible light images, and suppresses multimodal noise interference through cross-modal spatial knowledge mutual distillation. Secondly, an attribute decomposition module is proposed, which utilizes the attribute routing mechanism to decompose the mixed multimodal global features into independent attribute features, and ensures the diversity of attribute features by maximizing attribute entropy. Finally, an attribute graph fusion module is proposed, which independently fuses the corresponding attribute features of different modalities, and uses a personalized attribute graph customized for each category in the feature fusion process to model the association between various types of target attribute features, thereby improving the adaptability of the model to targets of different categories. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a principle block diagram of the method of the present invention.

[0043] Figure 2 It is a schematic diagram of the attribute decomposition module. DETAILED DESCRIPTION

[0044] The present invention is further described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] Step 1: First, in order to obtain a multimodal attention map, the infrared image and visible light image of the vehicle's surrounding environment are input into the backbone network to obtain the feature maps of the two modalities. The feature encoder is used to enhance the feature maps to obtain a feature enhancement sequence and The enhanced feature sequences of the two modalities and Input to the classifier to get the bimodal attention map Among them, c is the number of categories, h is the height of the graph, and w is the width of the graph.

[0046] Then, cross-modal spatial knowledge distillation is performed. It is proposed to use the optimal transfer theory to quantize the attention map To the attention map The minimum transmission cost is represents the i-th point of the visible light attention map, represents the i-th point of the infrared attention map, describes the difference between the two modal attention maps, and defines the transfer matrix T = [t ij ] and the distance matrix D = [d ij ].

[0047] where t ij Indicates from arrive The transmission weight, d ij Quantified and The distance between Less than , the visible light mode has a stronger representation ability at this point, so at this time d ij The cross-modal spatial knowledge mutual distillation model does not encourage the attention map of the visible light modality at this point to approach the infrared modality. Greater than , the infrared mode has a stronger characterization ability at this point. ij The cross-modal spatial knowledge mutual distillation model encourages the attention map of the visible light modality at this point to be close to the infrared modality. In this way, the student modality can learn the spatial knowledge of the teacher modality during the knowledge distillation process, ensuring that the model captures cross-modal complementary knowledge while filtering out inter-modal information interference;

[0048] Finally, the inter-distillation loss L is designedMD , infrared and visible light images are used as teacher modality and student modality respectively, and the model's learning ability of cross-modal complementary knowledge is improved through cross-modal knowledge mutual distillation:

[0049] L MD =OT(M V ,M I )+γOT(M I ,M V )

[0050] Among them, γ is the weight coefficient, and OT(·) is the optimal transmission distance for distillation. Considering that the importance of the visible light modality often changes with scene changes, a weight generation module based on scene perception is designed to adaptively adjust the value of γ according to the current scene:

[0051] γ=WAM(P)

[0052] Where P is the scene perception vector, which can be obtained by global pooling of the feature map of the visible light image. WAM(·) is the weighted perception module, which consists of a multi-layer perceptron.

[0053] Step 2: Input the infrared features and visible features obtained through cross-modal spatial knowledge distillation into the feature decoder to obtain a set of query tokens Q. The query tokens Q are then input into the attribute decomposition module based on the attribute routing mechanism inspired by the capsule network [S. Sabour, N. Frosstand G. Hinton, "Matrix capsules with em routing", Proc. Int. Conf. Learn. Represent., pp. 1-15, 2018.] to obtain multimodal attribute features. and The details are as follows:

[0054] First, K capsule mapping layers are used to map the query Token Q into K underlying capsules. The kth underlying capsule can be expressed as:

[0055] C k =CML k (Q)

[0056] Among them, CML k (·) is the kth capsule mapping layer;

[0057] Then, initialize L attribute capsules S l , L attribute capsules S l Corresponding to L groups of dynamic routing weights. For attribute capsule S l , initialize a set of dynamic routing weights for it And use the following rules to weight dynamic routing and property capsule S l To update, the update process of the tth time step is expressed as:

[0058]

[0059] Among them, Squash(·) is the activation function, and its main purpose is to normalize the output of the attribute capsule. is the multimodal attribute feature at the t-th time step, is the weight obtained by Softmax of the dynamic routing weight at the t-th time step, and ⊙ is the scalar product.

[0060] After the last time step, we get the multimodal attribute features and The number of time steps t is a pre-set hyperparameter;

[0061] In the attribute routing process, the bottom-level capsule transmits its output to all the attribute capsules in the upper layer through the weight matrix. This process essentially projects the output of the bottom-level capsule into different attribute spaces. When the output of the bottom-level capsule has a large scalar product with the output of a certain attribute capsule, it means that the output of the bottom-level capsule has a strong representation ability in the attribute space. The attribute routing refers to multiplying the bottom-level capsule and the dynamic routing weight to project the output of the bottom-level capsule into different attribute spaces.

[0062] In addition, in order to enhance the diversity of attribute capsules, the concept of attribute entropy is introduced to improve its feature expression ability by encouraging each attribute capsule to learn different features. Specifically, the attribute entropy H(S) is defined as:

[0063]

[0064] Among them, p(S l,j ) indicates S l In the probability distribution of the jth dimension, in the present invention, Softmax is used to obtain S l The probability distribution p(S l ).

[0065] Step 3: First, for the multimodal attribute features of infrared and visible light modes and Input into the attribute fusion module, use the cross-attention mechanism to fuse the corresponding attributes of the two modalities, and obtain the attribute fusion feature

[0066] Then, A FAs a node set of the attribute graph, a learnable edge structure is set for each class based on the pseudo-label of the query Token Q to construct the attribute graph. The attribute graph of the cth class can be expressed as in, Represents an edge set, and in the present invention, a set of learnable vectors is used Opposite edge set Row representation, learnable vector W E It encodes the association between attributes, and D is the edge feature dimension;

[0067] Afterwards, the information transfer mechanism of the graph neural network is used to update the attribute graph nodes by capturing the associations between the attributes;

[0068] Use global average pooling to integrate the node features of the attribute graph to obtain multimodal fusion queries Multimodal query sequence is a multimodal query sequence M F The elements in .

[0069] The multimodal query sequence M F Input the prediction head, which includes the target recognition head and the target positioning head, to obtain the query sequence M F The category prediction and position prediction corresponding to each element in can obtain the visible light image prediction and infrared image prediction.

[0070] After the prediction is obtained, the predicted target recognition loss is calculated based on the category prediction and the true class label. In the present invention, considering that there are often a large number of difficult targets in the vehicle target detection task, such as small vehicles and occluded vehicles, Focal Loss is used as the target recognition loss to strengthen the model's attention to difficult targets;

[0071] After that, the predicted target positioning loss is calculated based on the position prediction and the real box label. In the present invention, considering that in vehicle target detection, due to the complexity of the detection scene, occluded targets and small targets are widely present, and the traditional IoU loss is difficult to provide an effective gradient for the model when the border overlap is small. Therefore, L1 loss and GIoU loss are used as target positioning losses to enhance the detection performance of the model in complex scenes;

[0072] The GIoU metric improves the IoU by considering the minimum enclosing area of ​​the predicted box and the true box, which can provide a more stable gradient for model learning.

[0073] In summary, the present invention has developed a mutual distillation attribute fusion network for multimodal vehicle target detection, which improves the detection performance of the model in complex real-world scenarios through cross-modal spatial knowledge mutual distillation, attribute decomposition and attribute fusion. Specifically, on the one hand, the present invention introduces the concept of "attribute fusion" in multimodal target detection for the first time, decomposing independent and diverse attribute features from mixed global features, and then alleviates the mutual interference between potential attribute features in the multimodal feature fusion process by fusing the attribute features corresponding to different modalities respectively. On the other hand, the present invention customizes attribute graphs for targets of different categories, alleviating the problem of the model sacrificing its adaptability to category targets in the process of feature fusion due to the pursuit of universality for most categories. In addition, the present invention deeply mines multimodal complementary information through cross-modal mutual distillation, thereby improving the model's ability to resist interference with multimodal noise information.

Claims

1. A multimodal vehicle target detection framework, characterized in that: It includes backbone network, classifier, feature decoder, attribute decomposition module, attribute graph fusion module and prediction head; Input the infrared image and visible light image into the backbone network to obtain the feature maps of the two modalities; The classifier obtains the bimodal attention map based on the feature maps of the two modalities. Among them, c is the number of categories, h is the height of the graph, and w is the width of the graph; For the two-modal attention map M v and M I Perform cross-modal spatial knowledge mutual distillation to obtain distilled infrared features and visible light features, which are then input into the feature decoder to obtain the query Token Q; The query Token Q is then input into the attribute decomposition module to obtain the infrared image attribute features and the visible light image attribute features; The infrared image attribute features and the visible light image attribute features are fused in the attribute graph fusion module to obtain the attribute fusion feature A F , Fusion of attributes into features A F As a node set of the attribute graph, obtain the multimodal query sequence M F ; The multimodal query sequence M F Input the prediction head to obtain visible light image prediction and infrared image prediction.

2. A multimodal vehicle target detection framework according to claim 1, characterized in that: Also includes a feature encoder; The feature encoder is used to enhance the feature maps of the two modes output by the backbone network to obtain a feature enhancement sequence and represents the visible image enhancement feature enhancement sequence, represents the infrared image enhancement feature enhancement sequence; The enhanced feature sequences of the two modalities and Then input it into the classifier for classification.

3. A multimodal vehicle target detection framework according to claim 1, characterized in that: Using optimal transfer theory, by quantizing the attention map To the attention map The minimum transmission cost of the two modal attention maps is described, and the transmission matrix T = [t ij ] and the distance matrix D = [d ij ]; where t ij Indicates from arrive The transmission weight, d ij Quantified and The distance between.

4. A multimodal vehicle target detection framework according to claim 1, characterized in that: The inter-distillation loss L is designed MD , taking infrared and visible light images as teacher and student modalities respectively: L MD =OT(M V ,M I )+γOT(M I ,M V ) Among them, γ is the weight coefficient, OT(·) is the optimal transmission distance for distillation; γ=WAM(P) Where P is the scene perception vector, which is obtained by global pooling of the feature map of the visible light image, and WAM(·) is the weighted perception module.

5. A multimodal vehicle target detection framework according to claim 1, characterized in that: The attribute decomposition module includes K capsule mapping layers, L attribute capsules S l and L groups of dynamic routing weights; L attribute capsule S l Corresponding to the dynamic routing weight of group L; For the attribute capsule S l Initialize a set of dynamic routing weights And use the following rules to adjust the dynamic routing weight and property capsule S l To update, the update process of the tth time step is expressed as: Among them, Squash(·) is the activation function, the main purpose of which is to normalize the output of the attribute capsule; is the multimodal attribute feature at the t-th time step, is the weight obtained by Softmax after the dynamic routing weight of the t-th time step, ⊙ is the scalar product, C k represents the kth bottom-level capsule.

6. A multimodal vehicle target detection framework according to claim 5, characterized in that: Introduce attribute entropy H(S), which is defined as: Among them, p(S l,j ) indicates S l In the probability distribution of the jth dimension, use Softmax to get S l The probability distribution p(S l ).

7. A multimodal vehicle target detection framework according to claim 5, characterized in that: The K capsule mapping layers map the query Token Q to K underlying capsules, where the kth underlying capsule is represented as: C k =CML k (Q) Among them, CML k (·) is the kth capsule mapping layer.

8. A multimodal vehicle target detection framework according to claim 1, characterized in that: Fusion of attributes into features A F As a node set of the attribute graph, obtain the multimodal query sequence M F , specifically: Set a learnable edge structure for each class based on the pseudo-label of the query Token Q to build an attribute graph; The attribute graph of the cth class is represented as in, represents the edge set, Represents the attribute fusion feature of the cth category; Using a set of learnable vectors Opposite edge set Row representation, learnable vector W E It encodes the association between attributes, and D is the edge feature dimension; Afterwards, the information transfer mechanism of the graph neural network is used to update the attribute graph nodes by capturing the associations between the attributes; Use global average pooling to integrate the node features of the attribute graph to obtain multimodal fusion queries Multimodal Fusion Query is a multimodal query sequence M F The elements in .

9. A multimodal vehicle target detection framework according to claim 1, characterized in that: The multimodal query sequence M F Input the prediction head, which includes the target recognition head and the target positioning head, to obtain the query sequence M F The category prediction and position prediction corresponding to each element in are obtained to obtain the visible light image prediction and infrared image prediction.

10. A multimodal vehicle target detection framework according to claim 9, characterized in that: Focal Loss is used as the target recognition loss, and L1 loss and GIoU loss are used as the target positioning loss.

Images