A target detection method and device based on multimodal image fusion

Abstract

The present invention provides a target detection method and device based on multimodal image fusion. The method comprises: acquiring video images and infrared images in real time, and inputting them into a target detection model composed of a Transformer respectively; performing global feature extraction on the video images and infrared images respectively; fusing the extracted video image features and infrared image features; inputting the fused features of the video image and infrared image into a prediction module composed of a fully connected layer of the Transformer, and outputting the target category and target position. The present invention utilizes pure Transformer to construct a target detection model, which can give full play to the model advantages brought by the overall structure of the Transformer; the present invention performs target detection based on the feature fusion of video images and infrared images, and can realize target detection under any lighting conditions, solving the problem that the existing detection system has poor detection effect in dark environments such as at night.

Description

Technical Field

[0001] The present invention belongs to the technical field of target detection, and in particular relates to a target detection method and device based on multimodal image fusion. Background Art

[0002] Helping visually impaired individuals achieve better mobility has long been a critical social issue. Accurate and timely perception of the surrounding environment is essential for improving the safety and quality of life of individuals. Leveraging the rapid advancements in computer vision technology in recent years, various deep learning models based on convolutional neural networks (CNNs) have demonstrated remarkable capabilities in real-time recognition of natural scene images, even surpassing human accuracy and stability. These models have been successfully deployed in products, such as the recently successful autonomous driving technology.

[0003] Emerging wearable electronic devices for visually impaired people, including visually assisted perception devices, have also benefited from this. Using onboard micro-cameras or sensors, they capture real-time image or video data from scenes, and their onboard models perform corresponding calculations, providing the wearer with scene target detection results. However, most target detection models are based on brightly lit visible light color image data. This significantly reduces the model's performance when receiving visible light image inputs in environments with poor lighting conditions (such as those found in everyday life at night or in dark spaces), preventing sufficient recognition capabilities. Consequently, the corresponding visually impaired assistive devices are unable to provide timely danger alerts to the wearer. Summary of the Invention

[0004] In order to solve the above problems existing in the prior art, the present invention provides a target detection method and device based on multimodal image fusion.

[0005] In order to achieve the above objectives, the present invention adopts the following technical solutions.

[0006] In a first aspect, the present invention provides a target detection method based on multimodal image fusion, comprising the following steps:

[0007] Acquire video images and infrared images captured by a video camera and an infrared camera respectively in real time, and input them into the target detection model composed of Transformer.

[0008] Performing global feature extraction on the video image and the infrared image respectively using a feature encoding module composed of a Transformer encoder;

[0009] The extracted video image features and infrared image features are fused using a feature fusion module composed of a Transformer decoder;

[0010] The fused features of the video image and the infrared image are input into the prediction module composed of the Transformer fully connected layer, and the target category and target location are output.

[0011] Furthermore, the method further includes performing the following operations on the input video image and infrared image respectively before performing global feature extraction:

[0012] Cut the image into N slices;

[0013] Expand each slice in the channel dimension and input it into a linear fully connected layer to obtain a d-dimensional vector;

[0014] Calculate the sine and cosine position encoding in the row and column directions of the slice and add it to the output of the linear fully connected layer to obtain an N×d encoding matrix.

[0015] Furthermore, the feature encoding module is composed of a stack of Transformer encoders, each Transformer encoder includes a multi-head self-attention module layer and a feedforward network layer, as well as a normalization layer and a residual unit connected to each layer; the N×d encoding matrix of the video image or infrared image input to the multi-head self-attention module undergoes three different linear transformations to obtain a query vector, a key vector and a value vector of size N×d′, the similarity between the query vector and the key vector is calculated by the vector dot product with a scaling factor, and the attention weight matrix is ​​obtained after normalization by the softmax function, and the weight matrix is ​​multiplied by the value vector to obtain a one-way attention result; the multiple attention results are spliced ​​and then mapped back to the original dimension d′ to obtain the feature encoding of the video image or infrared image.

[0016] Furthermore, the feature fusion module is composed of a stack of Transformer decoders, each of which includes a multi-head self-attention module layer, a multi-head mutual attention module layer, and a feedforward network layer, as well as a normalization layer and a residual unit connected to each layer; the query vector Q of the multi-head mutual attention module layer of the i-th Transformer decoder is i Output from the multi-head self-attention module layer, key vector K i Sum value vector V i The video image features A and infrared image features B are output from the feature encoding module respectively; the query vector Q of the multi-head mutual attention module layer of the i+1th Transformer decoder i+1 Output from the multi-head self-attention module layer, key vector K i+1 Sum value vector V i+1 from B and A respectively; key vector K i Sum value vector V iBoth are N×d′ matrices, and the query vector Q i is an N′×d′ matrix, N′ <N;i=1,2,…。

[0017] Furthermore, the method further includes: judging dangerous targets and their positions according to target categories and target positions, and issuing danger warning information.

[0018] In a second aspect, the present invention provides an object detection device based on multimodal image fusion, comprising:

[0019] The image acquisition module is used to acquire video images and infrared images taken by the video camera and infrared camera respectively in real time, and input them into the target detection model composed of Transformer respectively;

[0020] A feature extraction module, configured to extract global features from the video image and the infrared image respectively using a feature encoding module composed of a Transformer encoder;

[0021] A feature fusion module is used to fuse the extracted video image features and infrared image features using a feature fusion module composed of a Transformer decoder;

[0022] The target prediction module is used to input the fusion features of the video image and the infrared image into the prediction module composed of the Transformer fully connected layer, and output the target category and target position.

[0023] Furthermore, the device further includes a vector embedding module, configured to:

[0024] Cut the image into N slices;

[0025] Expand each slice in the channel dimension and input it into a linear fully connected layer to obtain a d-dimensional vector;

[0026] Calculate the sine and cosine position encoding in the row and column directions of the slice and add it to the output of the linear fully connected layer to obtain an N×d encoding matrix.

[0027] Furthermore, the feature encoding module is composed of a stack of Transformer encoders, each Transformer encoder includes a multi-head self-attention module layer and a feedforward network layer, as well as a normalization layer and a residual unit connected to each layer; the N×d encoding matrix of the video image or infrared image input to the multi-head self-attention module undergoes three different linear transformations to obtain a query vector, a key vector and a value vector of size N×d′, the similarity between the query vector and the key vector is calculated by vector dot product with a scaling factor, and the attention weight matrix is ​​obtained after normalization by the softmax function, and the weight matrix is ​​multiplied by the value vector to obtain a one-way attention result; the multiple attention results are spliced ​​and then mapped back to the original dimension d′ to obtain the feature encoding of the video image or infrared image.

[0028] Furthermore, the feature fusion module is composed of a stack of Transformer decoders, each of which includes a multi-head self-attention module layer, a multi-head mutual attention module layer, and a feedforward network layer, as well as a normalization layer and a residual unit connected to each layer; the query vector Q of the multi-head mutual attention module layer of the i-th Transformer decoder is i Output from the multi-head self-attention module layer, key vector K i Sum value vector V i The video image features A and infrared image features B are output from the feature encoding module respectively; the query vector Q of the multi-head mutual attention module layer of the i+1th Transformer decoder i+1 Output from the multi-head self-attention module layer, key vector K i+1 Sum value vector V i+1 from B and A respectively; key vector K i Sum value vector V i Both are N×d′ matrices, and the query vector Q i is an N′×d′ matrix, N′ <N;i=1,2,…。

[0029] Furthermore, the device also includes a danger warning module, which is used to judge dangerous targets and their directions according to target categories and target positions, and issue danger warning information.

[0030] Compared with the prior art, the present invention has the following beneficial effects.

[0031] The present invention acquires video and infrared images in real time, extracts global features from each of them using a pure Transformer target detection model. The extracted video and infrared image features are then fused, and target category prediction is performed based on the fused features, thereby achieving target detection based on multimodal image fusion. By using pure Transformers to construct a target detection model, the present invention can fully leverage the model advantages brought by the Transformer's overall structure. The present invention performs target detection based on feature fusion of video and infrared images, enabling target detection under any lighting conditions, addressing the issue of poor detection performance of existing detection systems in dark environments such as at night. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is a flowchart of a target detection method based on multimodal image fusion according to an embodiment of the present invention.

[0033] Figure 2 Schematic diagram of the overall structure of the target detection model according to an embodiment of the present invention.

[0034] Figure 3 Schematic diagram of the self-attention mechanism principle.

[0035] Figure 4 Schematic diagram of the connection between two Transformer decoders.

[0036] Figure 5 This is a block diagram of an object detection device based on multimodal image fusion according to an embodiment of the present invention. DETAILED DESCRIPTION

[0037] To make the objectives, technical solutions, and advantages of the present invention more clear and understandable, the present invention is further described below with reference to the accompanying drawings and specific embodiments. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0038] Figure 1 This is a flowchart of a target detection method for multimodal image fusion according to an embodiment of the present invention, comprising the following steps:

[0039] Step 101: Acquire video images and infrared images captured by a video camera and an infrared camera, respectively, in real time, and input them into an object detection model composed of a Transformer.

[0040] Step 102: extract global features from the video image and the infrared image respectively using a feature encoding module composed of a Transformer encoder;

[0041] Step 103: using a feature fusion module composed of a Transformer decoder to fuse the extracted video image features and infrared image features;

[0042] In step 104, the fusion features of the video image and the infrared image are input into a prediction module composed of a fully connected layer of a Transformer, and the target category and target position are output.

[0043] In this embodiment, step 101 is mainly used to acquire video images and infrared images in real time. Most of the existing target detection models for assisting visually impaired people are modeled based on visible light color image data with sufficient brightness. This greatly reduces the performance of the model when receiving visible light image inputs with poor ambient lighting conditions (such as scenes in life at night, dark spaces, etc.), and cannot achieve the required recognition capabilities. For this reason, this embodiment acquires infrared images while acquiring video images. Since the imaging principle of infrared cameras is not affected by lighting conditions, the collected infrared images can provide a powerful supplement to scene target information in dark environments. Therefore, the target detection model based on the fusion of video and infrared images can have a high level of generalization ability in both lighted and dark scenes. The target detection model of this embodiment adopts a pure Transformer structure, giving full play to the model advantages brought by the overall structure of the Transformer, and can achieve better results and generalization capabilities than the convolutional neural network CNN model in image recognition tasks. The overall structure of the target detection model is as follows Figure 2 shown.

[0044] In this embodiment, step 102 is primarily used to extract image features. This embodiment utilizes a feature encoding module comprised of a Transformer encoder to extract features from the video image and infrared image, respectively. The Transformer encoder employs an attention mechanism, primarily composed of a multi-head self-attention module, capable of extracting global features from the input image. This significantly improves target detection accuracy compared to CNNs, which can only extract local features.

[0045] In this embodiment, step 103 is primarily used for multimodal feature fusion. This embodiment utilizes a feature fusion module composed of a Transformer decoder to fuse the extracted video image features and infrared image features. Existing CNN-based network structure models employ three main fusion schemes for multimodal image fusion tasks: early, middle, and late fusion. Early fusion involves directly concatenating multimodal images in the channel dimension at the model input as the input for the entire network. Middle fusion uses independent feature extractors for different modalities, utilizing various defined fusion calculation methods to fuse the feature maps of each modality at a certain level. Late fusion combines the final results of each modality's independent feature extractors for prediction. Regardless of the fusion method, each simply seeks fusion input without sufficient theoretical support or task specificity. Furthermore, it assumes a one-to-one correspondence between the features of the modalities in spatial location, and convolution only performs local fusion calculations. However, images and even feature maps of different modalities may have certain positional deviations. Performing only local calculations may cause the corresponding features to be misaligned, resulting in low fusion efficiency and poor detection results. This embodiment uses the Transformer to provide a multimodal fusion method based on attention (the Transformer decoder includes a multi-head self-attention module and a multi-head mutual attention module) to replace CNN. This allows information from different modalities to pay attention to each other globally, thus avoiding the limitations caused by positional deviations, making the fusion more efficient and more theoretically supportive.

[0046] In this embodiment, step 104 is mainly used to predict the target category. In this embodiment, the prediction of the target category is achieved by inputting the fusion features of the video image and the infrared image into the prediction module composed of the Transformer fully connected layer. The target of this embodiment refers to a dangerous target that may pose a threat to the action. The target category is divided according to the level of danger. For example, the tunnel or telephone pole in front is high-risk, and the bicycle parked on the side is medium-risk. The prediction module generally outputs the target position while outputting the target category. The prediction module is composed of two fully connected layer branches, one of which is composed of N1 layers of fully connected layers to complete the prediction of the target category, and the other branch is composed of N2 layers of fully connected layers to complete the regression prediction of the target position (the coordinates of the upper left corner and lower right corner of the detection box), thereby achieving the target detection task. The inputs of the two branches are the same, and both are the fusion features finally output by the feature fusion module.

[0047] As an optional embodiment, the method further includes performing the following operations on the input video image and infrared image respectively before performing global feature extraction:

[0048] Cut the image into N slices;

[0049] Expand each slice in the channel dimension and input it into a linear fully connected layer to obtain a d-dimensional vector;

[0050] Calculate the sine and cosine position encoding in the row and column directions of the slice and add it to the output of the linear fully connected layer to obtain an N×d encoding matrix.

[0051] This embodiment provides a technical solution for vector embedding of input video images and infrared images. For the input video images and infrared images, embedding coding is first performed to convert them into sequential inputs acceptable to the Transformer. Specifically, for an image input of size C×H×W, it is sliced ​​(patched). Assuming that the spatial size of each patch is h×w, N = (H / h) × (W / w) slices of size C×h×w can be obtained. Each slice is flattened along the dimension of channel C to obtain a vector of C×h×w dimensions. The N×(C×h×w) matrix is ​​input into a linear fully connected layer to calculate the dimension change to d dimensions. In addition, in order to make the patch code contain two-dimensional position information instead of showing permutation invariance, a fixed d-dimensional sine or cosine position code is calculated for the row and column directions respectively and added to the output of the linear layer. Finally, an N×d matrix is ​​obtained, which is the linear embedded coding representation of the input image, where the d-dimensional vector of each row is the representative vector of a patch, and the number of rows N of the matrix can be called the number of representative vectors. It should be noted that N varies with the set patch size and can be flexibly set according to the actual needs of the specific task.

[0052] As an optional embodiment, the feature encoding module is composed of a stack of Transformer encoders, each Transformer encoder includes a multi-head self-attention module layer and a feedforward network layer, as well as a normalization layer and a residual unit connected to each layer; the N×d encoding matrix of the video image or infrared image input to the multi-head self-attention module undergoes three different linear transformations to obtain a query vector, a key vector and a value vector of size N×d′, the similarity between the query vector and the key vector is calculated by vector dot product with a scaling factor, and the attention weight matrix is ​​obtained after normalization by the softmax function, and the weight matrix is ​​multiplied by the value vector to obtain a one-way attention result; the multiple attention results are spliced ​​and then mapped back to the original dimension d′ to obtain the feature encoding of the video image or infrared image.

[0053] This embodiment provides a specific technical solution for feature extraction. Feature extraction is implemented by a feature encoding module, which is obtained by stacking Transformer encoders. The specific number of stacking layers can be determined according to the debugging of the specific task, and the encoders of the two branches corresponding to the two images are independent of each other, and the number of stacking layers can be the same or different. The specific structure of each Transformer encoder (in order) consists of a layer of multi-head self-attention module, a layer of forward propagation module, and residual connections and normalization applied to each layer. The calculation process of the self-attention mechanism can be referred to Figure 3 , the input N×d encoding matrix is ​​respectively subjected to the linear mapping function W q 、W k 、W v The transformation obtains a query vector (Query), a key vector (Key), and a value vector (Value) of size N×d′. The similarity between the query vector and the key vector is calculated based on the vector dot product with a scaling factor. After normalization by the softmax function, the attention weight matrix is ​​obtained, which is expressed as follows:

[0054]

[0055] Where α is the weight matrix, Q is the query vector, K T is the transpose of the key vector. The weight matrix is ​​used to multiply the value vector (that is, it is equivalent to performing a weighted summation of the value vector by column according to the weight to obtain the numerical value of a certain point on the result matrix). Multi-head self-attention is to repeat this process independently multiple times, concatenate the results of multiple times, and then map them back to the original feature dimension d′. The forward relay module layer is a multi-layer perceptron (MLP) structure with one hidden layer. After the Transformer encoder, the input image can model its own feature encoding on a global scale, that is, each representative vector will calculate the similarity with all other representative vectors including itself, and has a globality that CNN does not have when extracting image features.

[0056] As an optional embodiment, the feature fusion module is composed of a stack of Transformer decoders, each of which includes a multi-head self-attention module layer, a multi-head mutual attention module layer, and a feedforward network layer, as well as a normalization layer and a residual unit connected to each layer; the query vector Q of the multi-head mutual attention module layer of the i-th Transformer decoder is i Output from the multi-head self-attention module layer, key vector K i Sum value vector V iThe video image features A and infrared image features B are output from the feature encoding module respectively; the query vector Q of the multi-head mutual attention module layer of the i+1th Transformer decoder i+1 Output from the multi-head self-attention module layer, key vector K i+1 Sum value vector V i+1 from B and A respectively; key vector K i Sum value vector V i Both are N×d′ matrices, and the query vector Q i is an N′×d′ matrix, N′ <N;i=1,2,…。

[0057] This embodiment provides a specific technical solution for feature fusion. The fusion of the two modal image features is achieved by a feature fusion module, which is obtained by stacking two Transformer decoders. The structural diagram of the stacked two Transformer decoders is shown in the figure below. Figure 4As shown. Similarly, the specific number of stacked layers of the decoder can be determined by debugging for a specific task. The detailed structure of each Transformer decoder (in order) consists of a multi-head self-attention module layer, a multi-head cross-attention module layer, a forward propagation module layer, and residual connections and normalizations applied to each layer. The multi-head self-attention module layer and the forward propagation module layer are the same as those in the Transformer encoder. The calculation mechanism of the multi-head cross-attention module layer is the same as that of self-attention. The only difference is that the query vector it receives comes from the output of the previous multi-head self-attention module layer, and the key vector and value vector come from the video image feature A and the infrared image feature B output by the feature encoding module respectively. It should be noted that the order of the image features A and B connected by the key vector and value vector of adjacent decoders is exactly opposite. For example, if the key vector and value vector of the current decoder are connected to A and B respectively, the key vector and value vector of the previous decoder and the next decoder are connected to B and A respectively, so as to enable the query vector to alternately perform attention calculation and fusion on the features of the two modalities. Such a design can effectively balance some information biases that may exist between the two modalities, including position biases, extract effective content with similar distributions, and model key mutual relationships that may exist globally. However, it should be noted that a specially defined query vector needs to be initialized separately for the first layer of the Transformer decoder as the input. This query vector is a set of learnable parameters that can implicitly learn how to extract the position encoding of the region where the target exists in the multi-modal image and play a mediating role in the fusion. It has good task orientation and prior knowledge and is a key component for completing the object detection task and the multi-modal fusion task. The dimension of this query vector is the same as that of the modal image encoding, but the size N' (or the number, that is, the number of rows of the encoding matrix) should be much smaller than the number N of the modal image encoding, that is, N' << N, and slightly larger than the maximum value of the number of targets to be detected in the data image, so as to reduce missed detections, interact only necessary features during the attention calculation process, reduce information redundancy, and greatly reduce the computational cost.

[0058] As an optional embodiment, the method further includes: judging the dangerous target and its orientation according to the output target category and target position, and sending out a danger warning message.

[0059] This embodiment gives a technical solution for danger warning. Danger warning belongs to a post-processing step. This embodiment judges the dangerous target based on the target category and target position output by the prediction module, calculates the orientation (and may also include the distance) of the target relative to the user, and finally sends an alarm message to the user through the voice module to remind the user to pay attention or take evasive actions.

[0060] Figure 5This is a schematic diagram of the composition of a multimodal image fusion target detection device according to an embodiment of the present invention, the device comprising:

[0061] The image acquisition module 11 is used to acquire the video image and infrared image taken by the video camera and infrared camera respectively in real time, and input them into the target detection model composed of Transformer respectively;

[0062] A feature extraction module 12 is configured to extract global features from the video image and the infrared image using a feature encoding module composed of a Transformer encoder;

[0063] A feature fusion module 13 is used to fuse the extracted video image features and infrared image features using a feature fusion module composed of a Transformer decoder;

[0064] The target prediction module 14 is used to input the fusion features of the video image and the infrared image into the prediction module composed of the Transformer fully connected layer, and output the target category and target position.

[0065] The device of this embodiment can be used to perform Figure 1 The technical solution of the method embodiment shown has similar implementation principles and technical effects, which will not be described in detail here. The same is true for the following embodiments, which will not be described in detail.

[0066] As an optional embodiment, the apparatus further includes a vector embedding module, configured to:

[0067] Cut the image into N slices;

[0068] Expand each slice in the channel dimension and input it into a linear fully connected layer to obtain a d-dimensional vector;

[0069] Calculate the sine and cosine position encoding in the row and column directions of the slice and add it to the output of the linear fully connected layer to obtain an N×d encoding matrix.

[0070] As an optional embodiment, the feature encoding module is composed of a stack of Transformer encoders, each Transformer encoder includes a multi-head self-attention module layer and a feedforward network layer, as well as a normalization layer and a residual unit connected to each layer; the N×d encoding matrix of the video image or infrared image input to the multi-head self-attention module undergoes three different linear transformations to obtain a query vector, a key vector and a value vector of size N×d′, the similarity between the query vector and the key vector is calculated by vector dot product with a scaling factor, and the attention weight matrix is ​​obtained after normalization by the softmax function, and the weight matrix is ​​multiplied by the value vector to obtain a one-way attention result; the multiple attention results are spliced ​​and then mapped back to the original dimension d′ to obtain the feature encoding of the video image or infrared image.

[0071] As an optional embodiment, the feature fusion module is composed of a stack of Transformer decoders, each of which includes a multi-head self-attention module layer, a multi-head mutual attention module layer, and a feedforward network layer, as well as a normalization layer and a residual unit connected to each layer; the query vector Q of the multi-head mutual attention module layer of the i-th Transformer decoder is i Output from the multi-head self-attention module layer, key vector K i Sum value vector V i The video image features A and infrared image features B are output from the feature encoding module respectively; the query vector Q of the multi-head mutual attention module layer of the i+1th Transformer decoder i+1 Output from the multi-head self-attention module layer, key vector K i+1 Sum value vector V i+1 from B and A respectively; key vector K i Sum value vector V i Both are N×d′ matrices, and the query vector Q i is an N′×d′ matrix, N′ <N;i=1,2,…。

[0072] As an optional embodiment, the device further includes a danger warning module, which is used to determine dangerous targets and their directions according to target categories and target positions, and to issue danger warning information.

[0073] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A target detection method based on multimodal image fusion, characterized in that: The following steps are involved: Acquire video images and infrared images captured by a video camera and an infrared camera respectively in real time, and input them into the target detection model composed of Transformer respectively; Performing global feature extraction on the video image and the infrared image respectively using a feature encoding module composed of a Transformer encoder; The extracted video image features and infrared image features are fused using a feature fusion module composed of a Transformer decoder; The fused features of the video image and the infrared image are input into the prediction module composed of the fully connected layers of the Transformer, which outputs the target category and target location. The feature fusion module is composed of a stack of Transformer decoders, each of which includes a multi-head self-attention module layer, a multi-head mutual attention module layer, a feedforward network layer, and a normalization layer and residual unit connected to each layer; the query vector defined for the first layer of Transformer decoder is initialized separately as input, and the query vector is a set of learnable parameters that can implicitly learn to extract the position encoding of the area where the target exists in the multimodal image. The query vector Q of the multi-head mutual attention module layer of the i-th Transformer decoder is i Output from the multi-head self-attention module layer, key vector K i Sum value vector V i The video image feature A and infrared image feature B are output from the feature encoding module respectively; The query vector Q of the multi-head mutual attention module layer of the i+1th Transformer decoder i+1 Output from the multi-head self-attention module layer, key vector K i+1 Sum value vector V i+1 from B and A respectively; key vector K i Sum value vector V i Both are N×d' matrices, and the query vector Q i is an N'×d' matrix, N' <N;i=1,2,…N'-1。 2. The target detection method based on multimodal image fusion according to claim 1, characterized in that: The method further includes performing the following operations on the input video image and infrared image respectively before performing global feature extraction: Cut the image into N slices; Expand each slice in the channel dimension and input it into a linear fully connected layer to obtain a d-dimensional vector; Calculate the sine and cosine position encoding in the row and column directions of the slice and add it to the output of the linear fully connected layer to obtain an N×d encoding matrix.

3. The target detection method based on multimodal image fusion according to claim 2, characterized in that: The feature encoding module is composed of a stack of Transformer encoders, each of which includes a multi-head self-attention module layer and a feedforward network layer, as well as a normalization layer and a residual unit connected to each layer; the N×d encoding matrix of the video image or infrared image input to the multi-head self-attention module undergoes three different linear transformations to obtain a query vector, a key vector and a value vector of size N×d', the similarity between the query vector and the key vector is calculated by vector dot product with a scaling factor, and the attention weight matrix is ​​obtained after normalization by the softmax function, and the weight matrix is ​​multiplied by the value vector to obtain a one-way attention result; the multiple attention results are spliced ​​and then mapped back to the original dimension d' to obtain the feature encoding of the video image or infrared image.

4. The target detection method based on multimodal image fusion according to claim 1, characterized in that: The method further comprises: judging the dangerous target and its position according to the target category and the target position, and issuing a danger warning message.

5. A target detection device based on multimodal image fusion, characterized in that: include: The image acquisition module is used to acquire video images and infrared images taken by the video camera and infrared camera respectively in real time, and input them into the target detection model composed of Transformer respectively; A feature extraction module, configured to extract global features from the video image and the infrared image respectively using a feature encoding module composed of a Transformer encoder; A feature fusion module is used to fuse the extracted video image features and infrared image features using a feature fusion module composed of a Transformer decoder; The target prediction module is used to input the fusion features of the video image and the infrared image into the prediction module composed of the fully connected layer of the Transformer, and output the target category and target location; The feature fusion module is composed of a stack of Transformer decoders, each of which includes a multi-head self-attention module layer, a multi-head mutual attention module layer, a feedforward network layer, and a normalization layer and residual unit connected to each layer; the query vector defined for the first layer of Transformer decoder is initialized separately as input, and the query vector is a set of learnable parameters that can implicitly learn to extract the position encoding of the area where the target exists in the multimodal image. The query vector Q of the multi-head mutual attention module layer of the i-th Transformer decoder is i Output from the multi-head self-attention module layer, key vector K i Sum value vector V i The video image feature A and infrared image feature B are output from the feature encoding module respectively; The query vector Q of the multi-head mutual attention module layer of the i+1th Transformer decoder i+1 Output from the multi-head self-attention module layer, key vector K i+1 Sum value vector V i+1 from B and A respectively; key vector K i Sum value vector V i Both are N×d' matrices, and the query vector Q i is an N'×d' matrix, N' <N;i=1,2,…N'-1。 6. The target detection device based on multimodal image fusion according to claim 5, characterized in that: The apparatus further comprises a vector embedding module, configured to: Cut the image into N slices; Expand each slice in the channel dimension and input it into a linear fully connected layer to obtain a d-dimensional vector; Calculate the sine and cosine position encoding in the row and column directions of the slice and add it to the output of the linear fully connected layer to obtain an N×d encoding matrix.

7. The target detection device based on multimodal image fusion according to claim 6, characterized in that: The feature encoding module is composed of a stack of Transformer encoders, each of which includes a multi-head self-attention module layer and a feedforward network layer, as well as a normalization layer and a residual unit connected to each layer; the N×d encoding matrix of the video image or infrared image input to the multi-head self-attention module undergoes three different linear transformations to obtain a query vector, a key vector and a value vector of size N×d', the similarity between the query vector and the key vector is calculated by vector dot product with a scaling factor, and the attention weight matrix is ​​obtained after normalization by the softmax function, and the weight matrix is ​​multiplied by the value vector to obtain a one-way attention result; the multiple attention results are spliced ​​and then mapped back to the original dimension d' to obtain the feature encoding of the video image or infrared image.

8. The target detection device based on multimodal image fusion according to claim 5, characterized in that: The device also includes a danger warning module, which is used to judge dangerous targets and their directions according to target categories and target positions, and to issue danger warning information.

Images