A multimodal UAV target detection framework based on scene language perception guidance

Abstract

The present invention belongs to the field of computer vision technology, and discloses a multimodal UAV target detection framework based on scene language perception guidance. First, a visual-language pre-training model is embedded in the multimodal target detection framework, and a scene language perception module is proposed. The visual-language pre-training model is used to provide the model with scene intelligent perception and understanding capabilities. Secondly, the conditional parameter generation module dynamically generates fusion parameters according to the current scene and target features, and the model's adaptability to scene changes is improved by customizing personalized fusion modes for different targets. Finally, a multimodal dynamic decoder is designed, and a dynamic symmetric fusion mechanism is established. Multimodal feature fusion is achieved by dynamically mining the complex complementary associations between multimodal data, and the importance of different modalities is adjusted in real time according to the current scene. The decoding mechanism of DETR is extended from single modality to multimodality, providing a new paradigm for multimodal UAV target detection without the need for a priori frames.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular relates to a multimodal unmanned aerial vehicle target detection framework based on scene language perception guidance. Background Art

[0002] In recent years, drones have been widely used in key areas such as traffic monitoring, remote sensing observation, and disaster relief due to their strong maneuverability, small size, and low operating costs. As one of the core tasks in the field of computer vision and artificial intelligence, target detection algorithms can significantly improve drones' ability to perceive and analyze complex environments, playing a vital role in the above applications.

[0003] However, existing drone target detection methods mainly rely on single-modal data, which limits the drone's perception capabilities in complex scenes. Especially in low-visibility nighttime environments, relying solely on visible light images often cannot achieve comprehensive scene perception. Some researchers have begun to explore the application of multimodal data to enhance drones' target detection capabilities in complex environments.

[0004] Although existing multimodal drone target detection methods based on infrared and visible light fusion have achieved impressive performance. However, most of these methods are based on the detection paradigm of "prior box + non-maximum suppression", which easily leads to missed detection and false detection of targets. Especially in the application scenarios of drone target detection, due to its unique wide shooting field of view, it is often necessary to process images with complex backgrounds and dense targets. The occlusion and overlap between targets in such scenes are particularly serious.

[0005] DETR provides a solution to this problem. The model is designed for unimodal target detection tasks, and how to extend it to multimodal target detection tasks is still a key issue to be solved. In addition, existing multimodal feature fusion methods often use fixed fusion parameters, resulting in targets sharing the same fusion mode in different scenarios. This one-size-fits-all fusion strategy shows obvious limitations in dynamically changing practical application scenarios, especially in drone target detection that needs to adapt to changing environmental conditions. Specifically, drones often need to cope with drastically changing lighting and weather conditions during missions. In this case, an ideal multimodal fusion model should be able to dynamically adjust its fusion strategy based on real-time scene information. In addition, fixed fusion parameter models tend to optimize the global average performance, while neglecting the personalized customization of multimodal feature fusion modes in specific scenarios. This design may lead to suboptimal performance of the model. Summary of the invention

[0006] Purpose of the invention: In view of the problems existing in the above-mentioned background technology, the present invention proposes a multimodal UAV target detection method based on scene language perception guidance, which specifically includes the following steps.

[0007] Step 1: Input the infrared and visible light images into the backbone network to obtain a multimodal feature map, and inject position codes into each element of the multimodal feature map to obtain a visible light feature sequence. and infrared characteristic sequences

[0008] At the same time, the infrared and visible light images are input into the scene language perception module respectively to obtain the scene language perception embedding of the two modalities. and in, scene-language-aware embeddings representing the visible light modality, Representing scene language-aware embeddings for infrared light modality.

[0009] Step 2: transform the feature sequences of the two modes and Input into the Transformer encoder to capture the spatial dependencies between elements in the feature sequence and obtain the feature enhancement sequence of the two modes and

[0010] Next, the feature enhancement sequences of the two modalities are and Input the classifier and get the classification graph C of the two modes V and C I .

[0011] Step 3: Classification diagram C based on two modes V and C I , select the topN elements of the classification probability score to construct the query sequence E of the two modalities V and E I .

[0012] Step 4: Query sequence E of two modalities V and E I Update to get the updated two-modal query sequence and , and the updated query sequence and and scene language-aware embedding and Input the condition parameter generation module together, which is the query sequence E V and E I Each element in customizes the personalized fusion parameter W.

[0013] Step 5: Based on the characteristic parameter W, the query sequence and Fusion is performed to obtain the fusion query sequence E F ;

[0014] Among them, the fusion parameter W is provided by the conditional parameter generation module, and the fusion weight of the multimodal features is obtained based on scene-aware embedding using the modal trade-off module.

[0015] Step 6: Fusion query sequence E F Input the prediction head to obtain the target's position prediction and category prediction, completing the multimodal UAV target detection task.

[0016] Furthermore, in step 1, the infrared and visible light images are input into the scene language perception module respectively to obtain the scene language perception embedding of the two modalities. and The details are as follows:

[0017] Step 1.1, dividing the image into a plurality of image blocks, wherein the image refers to a visible image and an infrared image;

[0018] Step 1.2: Input the image block into the image encoder based on the VIT architecture to obtain the image embedding F p .

[0019] Step 1.3, initialize a set of text token sequences F TS , in the token sequence F TS Add a decoding tag to the beginning of , and use the causal self-attention mechanism to update the text token sequence F TS Among them, the role of the causal self-attention mechanism is to ensure that the generation process of the current text token only depends on the text token of the previous image block, so as to ensure the causality of the input sequence. Based on the causal self-attention mechanism, the text token sequence F can be obtained. TS The causal embedding sequence It is expressed as:

[0020]

[0021] Where Q, K, and V represent the learnable query mapping, key mapping, and value mapping, respectively. k is the vector dimension, M is the mask matrix;

[0022] Step 1.4: Use the cross-attention layer to integrate the image embedding F p and causal embedding sequence That is, integrating the multimodal information of images and texts to embed the causal sequence To update:

[0023]

[0024] in, denote the learnable query mapping, key mapping, and value mapping in the criss-cross attention layer, respectively.

[0025] Afterwards, the updated causal embedding sequence Input is a feed-forward layer composed of MLP, outputs the word with the highest probability in the VoCap vocabulary, and obtains the scene language-aware text T corresponding to the current text token.

[0026]

[0027] Among them, FFL(·) is the feed-forward layer.

[0028] Step 1.5: Input the next image block into the image encoder based on the VIT architecture, go to step 1.2, and repeat the above process until a terminator is generated or all image blocks are traversed or all text tokens are traversed, and the complete scene language-aware text T corresponding to the image is obtained.

[0029] Finally, the language text is input into the text encoder, and the text encoder is used to map T to the language embedding space to obtain the scene language-aware embedding The scene language-aware embedding Context-aware embeddings including two modalities and

[0030] Furthermore, the query sequence of the two modes in step 4 and and scene language-aware embedding and The conditional parameter generation module includes a perception fusion layer and a conditional parameter generation network. The conditional parameter generation module customizes a personalized fusion parameter W for each element in the query sequence. The specific steps are as follows:

[0031] Step 4.1, construct the perception fusion layer to integrate scene language perception and target feature information as follows:

[0032] The query sequence and and scene language-aware embedding and Input to the perceptual fusion layer to obtain the conditional fusion vector C F , including the visible light condition fusion vector C F V and infrared condition fusion vector C F I ;

[0033] Step 4.2: Conditional fusion vector C F Input to the conditional parameter generation network for dynamic parameter generation, expressed as:

[0034] W=CPGN(C F )

[0035] Among them, CPGN(·) is a conditional parameter generation network, which is composed of multi-layer perceptrons;

[0036] W is the dynamic fusion parameter corresponding to the query sequence, including the dynamic fusion parameter W of the visible light modality V = {Q V ,K V ,V V} and the dynamic fusion parameter W of the infrared modality I = {Q I ,K I ,V I}.

[0037] Further, in step 5, based on the characteristic parameter W, the query sequence and Fusion is performed to obtain the fusion query sequence E F , the steps are as follows:

[0038] In order to capture the complementary task-related information in the visible light modality and the infrared modality, a scene perception-guided dynamic symmetric fusion mechanism is designed. After integrating the query sequences of the two modalities, the modal trade-off coefficient generated by the modal trade-off module is used to obtain the fused query sequence. This process is a dynamic symmetric fusion mechanism. Dynamic means that the value of the modal trade-off coefficient is not fixed, but can be dynamically adjusted according to the specific visual scene to integrate the query sequences of the two modalities to obtain the fused query sequence E. F . Fusion query sequence E F It can be expressed as:

[0039]

[0040] Among them, λ1 and λ2 are modal trade-off coefficients, which are obtained by the modal trade-off module based on language-aware embedding.

[0041] The modal trade-off coefficients λ1 and λ2 are obtained as follows:

[0042] Embedding scene language awareness and Input into the modal trade-off module to obtain the fusion weight of multimodal features, namely the modal trade-off coefficients λ1 and λ2;

[0043]

[0044] Where MTM(·) represents the modal trade-off module.

[0045] Furthermore, the prediction head in step 6 is a prediction head of Hungarian matching;

[0046] Fusion query sequence E F Input the target recognition head and the target location head to obtain the category prediction corresponding to each element in the query sequence and location prediction

[0047]

[0048] Among them, ORH(·) and ORL(·) are the object recognition head and object localization head, respectively, which are composed of simple linear layers.

[0049] Beneficial effects:

[0050] 1. Enhanced perception capability in complex environments: By adopting multimodal data, the present invention can improve the target detection capability of UAVs in complex scenes, especially at night or in low visibility environments.

[0051] 2. Improved target detection accuracy: In view of the problems of missed and false detection of targets in existing methods, the present invention reduces the occurrence of these problems by improving the detection paradigm.

[0052] 3. Dynamic adjustment of fusion strategy: The present invention can dynamically adjust the multimodal feature fusion strategy according to real-time scene information to adapt to changing environmental conditions, such as changes in lighting and weather.

[0053] 4. Solving the limitations of the DETR model in multimodal tasks: This invention explores how to expand the DETR model from unimodal target detection tasks to multimodal target detection tasks, solving a key problem in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is a principle block diagram of the method of the present invention. DETAILED DESCRIPTION

[0055] The present invention is further described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0056] A multimodal UAV target detection framework based on scene language perception guidance of the present invention includes five key modules, namely: backbone network, Transformer encoder, scene language perception module, conditional parameter generation module, multimodal dynamic fusion decoder;

[0057] The scene language perception module uses a vision-language pre-trained model to convert vision into text descriptions, and obtains scene language perception embedding from text descriptions to provide intelligent scene perception for the dynamic fusion mechanism;

[0058] The conditional parameter generation module combines scene information and target characteristics to customize personalized fusion parameters for each target; it dynamically generates fusion parameters based on global scene language perception and the local context of the target, improving the model's ability to adapt to complex scene changes.

[0059] The multimodal dynamic fusion decoder captures multimodal complementary information through multimodal feature fusion and dynamically adjusts the fusion mode between multimodal data according to scene and target characteristics;

[0060] Based on the multimodal drone target detection framework of the present invention, the detection method is as follows:

[0061] Step 1: Input the infrared and visible light images into the backbone network to obtain the multimodal feature map M and the visible light feature map M. V and infrared characteristic map M I ; Inject position coding into each element of the multimodal feature map M to obtain a visible light feature sequence and infrared characteristic sequences Specifically:

[0062] Input the image into the backbone network to obtain the feature map Among them, h, w and c represent the length, width and number of channels of the feature map respectively. Then, for each element in the feature map M Inject position encoding to form feature sequence

[0063]

[0064] in, Indicates m i,j The position code.

[0065] The scene language perception module of the present invention is constructed based on the vision-language pre-training model BLIP. At the same time, the infrared and visible light images are respectively input into the scene language perception module to obtain the scene language perception embedding of the two modalities. and The scene language perception module processes images as follows:

[0066] Step 1.1: Divide the image into several image blocks and input them into the image encoder based on the VIT (Vision Transformer) architecture to obtain the image embedding F p =VIT(x1+p1,...,x i +p i ,...,x N +p N );

[0067] Among them, x i represents the image block, p i Represents the position encoding of the image block;

[0068] Step 1.2, initialize a set of text token sequences F TS , add a decoding tag at the beginning of the token sequence, and use the causal self-attention mechanism to update the text token sequence. The role of the causal self-attention mechanism is to ensure that the generation process of the current text token only depends on the previous text token to ensure the causality of the input sequence. Based on this mechanism, the causal embedding sequence of the text token sequence can be obtained:

[0069]

[0070] Where Q, K, and V represent the learnable query mapping, key mapping, and value mapping, respectively. k is the vector dimension, M is the mask matrix, F TS Represents a text token sequence;

[0071] Step 1.3: Use the cross-attention layer to integrate the image embedding F p and causal embedding sequence That is, integrating the multimodal information of images and texts to embed the causal sequence To update:

[0072]

[0073] in, denote the query mapping, key mapping, and value mapping that can be learned in the cross-attention layer, respectively;

[0074] Step 1.4: Use the cross-attention layer to integrate the multimodal information of images and texts and update the causal embedding sequence. Input is a feed-forward layer composed of a multilayer perceptron (MLP), outputs the word with the highest probability in the VoCap vocabulary, and obtains the scene language perception text corresponding to the current text token

[0075] Step 1.5, repeat the above process until a terminator is generated or all text tokens are traversed, and the complete scene language-aware text T corresponding to the image is immediately obtained.

[0076] Finally, the language text T is input into the text encoder, and the text encoder is used to map T to the language embedding space to obtain the scene language-aware embedding For the present invention, the input to the scene language perception module is the visible light image and the infrared image, and the output of the scene language perception module is the scene language perception embedding of the two modalities. and

[0077] Step 2: To capture the global dependency of each element in the feature map M, the visible light feature sequence M V and infrared characteristic sequences Input into the Transformer encoder for feature enhancement to obtain a feature enhancement sequence A feature enhancement sequence consisting of two modalities and Feature Enhancement Sequence The elements in are

[0078] Enhance the sequence of features of the two modalities and Input the classifier and get the classification graph Among them, k represents the number of categories, and the classification map encodes the classification probability of each element in the feature map belonging to each category; the classification map C includes two modal classification maps C V and C I ;

[0079] Step 3: Considering that elements with higher classification probabilities may contain key target information, the classification graphs C of the two modalities are constructed based on the classification probabilities. V and C I Sort the elements in the , select the topN elements with the highest classification probability scores, and construct the initial query sequence E = {e1,...,e n ,..,e N}, the query sequence of the two modalities is recorded as E V and E I .

[0080] Step 4, and use the attention mechanism to analyze the query sequence E(E V and E I ) to update and obtain the updated query sequence ( and ), update the query sequence of the two modes input into the conditional parameter generation module and and scene language-aware embedding and The fusion parameters are input into the conditional parameter generation module to customize the personalized fusion parameter W for each element in the query sequence. Specifically:

[0081] First, the self-attention mechanism is used to model the dependencies between the elements in the query sequence E, so that each element in the query sequence E can contain the context information of other elements. Then, the cross-attention mechanism is used to allow the query sequence E to "query" relevant target features from the image, thereby improving the feature expression ability of the query sequence. Finally, the feedforward layer is used to perform a linear transformation on the query sequence E, and further enhance the features of each element in the query sequence E to obtain the updated

[0082] Next, considering Each element in encodes the target feature information. Therefore, the present invention constructs a perceptual fusion layer to obtain the conditional fusion vector C by integrating the scene language perception embedding and the target feature information in the query sequence. F .

[0083] Finally, the conditional fusion vector is input into the conditional parameter generation network to generate dynamic parameters based on target features and spatial context information:

[0084]

[0085] Among them, CPGN(·) is a conditional parameter generation network, which is composed of a multi-layer perceptron. W is the dynamic fusion parameter corresponding to the query sequence, including the dynamic fusion parameter W of the visible light modality. V = {Q V ,K V ,V V} and the dynamic fusion parameter W of the infrared modality I = {Q I ,K I ,V I}.

[0086] Step 5: Based on the characteristic parameter W, the query sequence and Fusion is performed to obtain the fusion query sequence E F , specifically:

[0087] In order to capture the complementary task-related information in the visible light modality and the infrared modality, the present invention designs a scene-aware guided dynamic symmetric fusion mechanism in the multimodal dynamic fusion decoder to integrate the query sequences of the two modalities to obtain a fused query sequence. F It can be expressed as:

[0088]

[0089] Among them, λ1 and λ2 are modal trade-off coefficients.

[0090]

[0091] Where MTM(·) represents the modal trade-off module.

[0092] Step 6: merge the query sequence E F Input the prediction head to obtain the location prediction and category prediction of the target.

[0093] The query sequence E F Input prediction head, which includes target recognition head and target location head, to obtain the category prediction corresponding to each element in the query sequence and location prediction

[0094]

[0095] Among them, ORH(·) and ORL(·) are the object recognition head and object localization head, respectively, which are composed of simple linear layers.

[0096] Next, the prediction loss is calculated based on the predicted results and the true labels. To ensure a one-to-one correspondence between the true labels and the predicted results, Hungarian matching is used to find the one-to-one match with the lowest cost between the true labels and the predicted results by minimizing the cost function. is the optimal matching function to be solved:

[0097]

[0098] Among them, L match (·) represents the matching loss, y i represents the true label, represents the predicted label, S N represents the set of all possible matches, and J represents the number of true labels.

[0099] During the training process, the model uses an optimization problem to find the best one-to-one match between the prediction and the true annotation, thereby optimizing the association between the model output and the actual annotation. The total loss is calculated based on the matching results to optimize the model. The total loss can be expressed as:

[0100] L total =L cls +β1L L1 +β2L GIOU

[0101] Among them, L clsis the cross entropy loss between the true label and the corresponding category prediction, GIOU(·) takes into account the overlapping area of ​​bounding boxes and the relationship between the spatial positions of bounding boxes. β1 and β2 are weight factors used to balance the contribution of each part in the loss function, y pos,i Indicates the real location. represents the predicted position and J represents the number of true labels.

[0102] In summary, the vision-language dynamic detection Transformer (VLD-DETR) proposed in the present invention embeds the vision-language pre-trained model (VLPM) into the multimodal target detection framework for the first time, proposes a scene language perception module, and uses the vision-language pre-trained model VLPM to provide the model with scene intelligent perception and understanding capabilities, thereby realizing adaptive scene perception without the need for additional supervisory information; it breaks through the limitations of existing methods based on fixed fusion parameters.

[0103] Secondly, a dynamic fusion mechanism was developed, which dynamically generated fusion parameters according to the current scene and target features through a conditional parameter generation module. The current scene in the present invention refers to visible light images and infrared images, which contain different features and information, and the target features refer to features extracted from visible light images and infrared images. By customizing personalized fusion modes for different targets, the model's adaptability to scene changes is improved.

[0104] Finally, a multimodal dynamic fusion decoder was designed, and a dynamic symmetric fusion mechanism was established. Multimodal feature fusion was achieved by dynamically mining the complex complementary correlations between multimodal data, and the importance of different modalities was adjusted in real time according to the current scenario. The decoding mechanism of DETR was extended from single modality to multimodality, which improved the detection performance of the model in dense and occluded scenes, and provided a new paradigm for multimodal UAV target detection without the need for a priori frames.

Claims

1. A multimodal target detection method, characterized in that: The steps include: Step 1: Input the infrared and visible light images into the backbone network to obtain a multimodal feature map, inject position coding into each element of the multimodal feature map, and obtain a visible light feature sequence. and infrared characteristic sequences At the same time, the infrared and visible light images are input into the scene language perception module respectively to obtain the scene language perception embedding of the two modalities. and Step 2: The visible light feature sequence and infrared characteristic sequences Input into the Transformer encoder to capture the spatial dependencies between elements in the feature sequence and obtain the feature enhancement sequence of the two modes and Enhance the sequence of features of the two modalities and Input the classifier and get the classification graph C of the two modes V and C I ; Step 3: Classification diagram C based on two modes V and C I , select the topN elements of the classification probability score and construct the query sequence E of the two modalities V and E I ; Step 4: Query sequence E of two modalities V and E I Perform feature enhancement to obtain the updated bimodal query sequence and The query sequence of the two modes and and scene language-aware embedding and The two are input together into the conditional parameter generation module to customize the personalized fusion parameter W for each element in the query sequence; Step 5: Based on the fusion parameter W, the query sequence and Fusion is performed to obtain the fusion query sequence E F ; Step 6: merge the query sequence E F Input the prediction head to obtain the location prediction and category prediction of the target.

2. A multimodal target detection method according to claim 1, characterized in that: In step 1, the infrared and visible light images are input into the scene language perception module respectively to obtain the scene language perception embedding of the two modalities. and The details are as follows: Step 1.1, dividing the image into a plurality of image blocks, wherein the image refers to a visible image and an infrared image; Step 1.2: Input the image block into the image encoder based on the VIT architecture to obtain the image embedding F p Step 1.3, initialize a set of text token sequences F TS , in the token sequence F TS Add a decoding tag to the beginning of , and use the causal self-attention mechanism to update the text token sequence F TS , and then based on the causal self-attention mechanism, obtain the updated text token sequence F TS The causal embedding sequence Causal Embedding Sequence It is expressed as: Where Q, K, and V represent the learnable query mapping, key mapping, and value mapping, respectively. k is the vector dimension, M is the mask matrix; Step 1.4: Use the cross-attention layer to integrate the multimodal information of images and texts to embed the causal sequence Update to get the updated causal embedding sequence in, denote the query mapping, key mapping, and value mapping that can be learned in the cross-attention layer, respectively; Embed the updated causal sequence Input the feed-forward layer composed of MLP, output the word with the highest probability in the VoCap vocabulary, and obtain the scene language-aware text T corresponding to the current text token; Among them, FFL(·) is the feed-forward layer; Step 1.5: Input the next image block into the image encoder based on the VIT architecture, and go to step 1.

2. Repeat the above process until a terminator is generated or all image blocks or all text tokens are traversed, and the complete scene language-aware text T corresponding to the image is obtained. Finally, the language text is input into the text encoder, and the text encoder is used to map T to the language embedding space to obtain the scene language-aware embedding The scene language-aware embedding Context-aware embeddings including two modalities and 3. A multimodal target detection method according to claim 1, characterized in that: The query sequence of the two modalities in step 4 and and scene language-aware embedding and The fusion parameters W are input into the conditional parameter generation module to customize the personalized fusion parameter W for each element in the query sequence. The specific steps are as follows: Step 4.1, construct the perception fusion layer to integrate scene language perception and target feature information as follows: The query sequence and and scene language-aware embedding and Input to the perceptual fusion layer to obtain the conditional fusion vector C F , including the visible light condition fusion vector C F V and infrared condition fusion vector C F I ; Step 4.2: Conditional fusion vector C F Input to the conditional parameter generation network for dynamic parameter generation, expressed as: W=CPGN(C F ) Among them, CPGN(·) is a conditional parameter generation network, which is composed of multi-layer perceptrons; W is the dynamic fusion parameter corresponding to the query sequence, including the dynamic fusion parameter W of the visible light modality V = {Q V ,K V ,V V } and the dynamic fusion parameter W of the infrared modality I = {Q I ,K I ,V I }.

4. The multimodal target detection method according to claim 1, characterized in that: In step 5, based on the fusion parameter W, the query sequence and Fusion is performed to obtain the fusion query sequence E F , as follows: Integrate the query sequence of two modalities and To obtain the fusion query sequence E F , fused query sequence E F It is expressed as: Among them, λ1 and λ2 are modal trade-off coefficients.

5. A multimodal target detection method according to claim 4, characterized in that: The modal trade-off coefficients λ1 and λ2 are obtained as follows: Embedding scene language awareness and Input into the modal trade-off module to obtain the fusion weight of multimodal features, namely the modal trade-off coefficients λ1 and λ2; Where MTM(·) represents the modal trade-off module.

6. A multimodal target detection method according to claim 1, characterized in that: Step 6 is specifically as follows: Fusion query sequence E F Input the target recognition head and the target location head to obtain the category prediction corresponding to each element in the query sequence and location prediction Among them, ORH(·) and ORL(·) are the object recognition head and object localization head, respectively, which are composed of simple linear layers.

7. The multimodal target detection method according to claim 1, characterized in that: The prediction head in step 6 is the prediction head of Hungarian matching; Find the one-to-one match with the lowest cost between the true label and the predicted result, is the optimal matching function to be solved: Among them, L match (·) represents the matching loss, y i represents the true label, represents the predicted label, S N represents the set of all possible matches, and J represents the number of true labels.

8. The multimodal target detection method according to claim 1, characterized in that: The total loss is expressed as: L total =L cls +β1L L1 +β2L GIOU Among them, L cls is the cross entropy loss between the true label and the corresponding category prediction, GIOU(·) takes into account the overlapping area of ​​bounding boxes and the relationship between the spatial positions of bounding boxes. β1 and β2 are weight factors. pos,i Indicates the real location. represents the predicted position and J represents the number of true labels.

9. The multimodal target detection method according to claim 1, characterized in that: The query sequence E for the two modalities in step 4 V and E I Perform feature enhancement to obtain the updated bimodal query sequence and Specifically: First, the query sequence E is modeled using the self-attention mechanism V and E I The dependencies between the elements in the Then, the cross attention mechanism is used to make the query sequence E V and E I "Query" relevant target features from the image; Finally, a feed-forward layer is used to train the query sequence E V and E I Perform a linear transformation to obtain the updated and

Images