A multimodal feature fusion target detection method

Abstract

The present invention relates to a feature fusion target detection method, and in particular to a multimodal feature fusion target detection method, which is used to solve the problem that a single-modal visible light image or infrared image in existing target detection cannot effectively obtain target information in the image, and thus cannot provide accurate prediction results. The multimodal feature fusion target detection method extracts features from low-level, middle-level and high-level layers respectively, and through multiple upsampling and splicing fusion, it can more comprehensively describe the target, improve the detection effect, and make the model more stable in complex environments.

Description

Technical Field

[0001] The present invention relates to a feature fusion target detection method, and in particular to a multimodal feature fusion target detection method based on yolov8s. Background Art

[0002] Remote sensing technology can obtain many types of images, including visible light images, infrared images, radar images, and hyperspectral images. However, due to the influence of multiple factors such as illumination, atmosphere, terrain, and target size, a single type of image often cannot provide accurate prediction results, limiting its application effect. In order to overcome these limitations, researchers began to explore the use of images of different spectral bands for information fusion to achieve complementary advantages, thereby effectively performing target detection in various complex environments (such as night, fog, occlusion, etc.).

[0003] Infrared images and visible light images are often used in image fusion research because they can present different features of objects and are easy to obtain. Infrared detectors can detect the thermal radiation information of objects, are suitable for discovering hidden targets, and have the ability to work all day and all night, but their image resolution is low, relatively blurry, and lacks color and texture information. Visible light detectors can provide rich color and texture details, but are easily disturbed by environmental factors such as lighting and weather. Therefore, how to effectively fuse infrared images and visible light images to obtain more information is an important current research direction. Summary of the invention

[0004] The purpose of the present invention is to solve the shortcomings of existing target detection that single-modality visible light images or infrared images cannot effectively obtain target information in the image and thus cannot provide accurate prediction results, and to provide a multi-modal feature fusion target detection method.

[0005] In order to solve the deficiencies of the above-mentioned prior art, the present invention provides the following technical solutions:

[0006] A multimodal feature fusion target detection model, which is special in that it includes an extraction branch network backbone1, an extraction branch network backbone2, three multimodal feature fusion modules, and a fusion feature extraction module and a target prediction module whose output and input are connected in sequence;

[0007] The extraction branch network backbone1 is used to extract low-level, medium-level, and high-level visible light image features from the visible light image; the extraction branch network backbone2 is used to extract low-level, medium-level, and high-level infrared image features from the infrared image;

[0008] Each of the multimodal feature fusion modules is used to divide the visible light image features and infrared image features of the same level into initial similar features and initial specific features, and perform redundancy removal processing on the initial similar features to obtain similar features, perform redundancy removal and feature enhancement processing on the initial specific features to obtain specific features, and then perform element-by-element summation of the similar features and the specific features to output the multimodal fusion features of the corresponding level;

[0009] The fusion feature extraction module is used to upsample the high-level multimodal fusion features and splice them with the middle-level multimodal fusion features, extract and fuse the spliced ​​features to output feature A, upsample feature A and splice it with the low-level multimodal fusion features, further extract and fuse the spliced ​​features to output the first feature to the target prediction module; and is used to perform convolution, batch normalization and activation operations on the first feature, splice it with feature A, and then output the second feature to the target prediction module after feature extraction and fusion; and is used to perform convolution, batch normalization and activation operations on the second feature, splice it with the high-level multimodal fusion features, and then output the third feature to the target prediction module after feature extraction and fusion;

[0010] The target prediction module is used to obtain a prediction result based on the first feature, the second feature and the third feature, and the prediction result includes a bounding box, a category and a confidence level of the target.

[0011] Furthermore, the extraction branch network backbone1 and the extraction branch network backbone2 have the same structure, both including CBS (Convolution+Batch Normalization+ActivationFunction), CBS, C2f (CSP (Cross Stage Partial) Bottleneckwith 2 Fusion, CSP (cross-stage partial network) bottleneck structure with 2 fusions), CBS, C2f, CBS, C2f, CBS, C2f, SPPF (Spatial Pyramid Pooling-Fast), whose output and input are connected in sequence.

[0012] Furthermore, the multimodal feature fusion module includes a feature division part, a first attention module, a second attention module and a fusion part P4; the output end of the feature division part P1 is respectively connected to the input end of the first attention module and the second attention module, and the output end of the first attention module and the second attention module are both connected to the input end of the fusion part;

[0013] The working principle of the feature division part is as follows:

[0014] Step A1: Calculate the k-th layer visible light image features through feature division and infrared image features The cosine similarity between the mth channels m (θ), as follows:

[0015]

[0016] Where m represents the mth channel in the kth layer feature, j represents the jth pixel in the mth channel, and its value ranges from 1 to n. Represents the visible light image features obtained at the kth layer The value of the jth pixel in the mth channel of Represents the infrared image features obtained at the kth layer The value of the th pixel in the mth channel; cosine similarity cos m (θ) takes the value [-1, 1];

[0017] Step A2, generating a similarity matrix C by calculating the cosine similarity of each channel of the kth layer;

[0018] Step A3: When extracting the initial similarity features, all values ​​greater than 0 in the similarity matrix C are set to 1, and all values ​​less than or equal to 0 are set to 0, to obtain a new similarity matrix C. com , the similarity matrix C com Respectively with the original input visible light image features and infrared image features Initial similarity features obtained by multiplication Similar features to the initial The formula is as follows:

[0019]

[0020]

[0021] When extracting the initial specific features, all values ​​greater than 0 in the similarity matrix C are set to 0, and all values ​​less than 0 are set to 1, and a new similarity matrix C is obtained. dif , the similarity matrix C dif Respectively with the original input visible light image features and infrared image features Multiply to get the initial specific features and initial specific features The formula is as follows:

[0022]

[0023]

[0024] The first attention module is used to Similar features to the initial Perform redundancy removal and output similar features to the fusion part;

[0025] The second attention module is used to remove redundancy and strengthen feature processing of the initial specific features, and output the specific features to the fusion part;

[0026] The fusion part is used to sum similar features and specific features element by element, and output multimodal fusion features of the corresponding level.

[0027] Further, the fusion feature extraction module includes a first upsampling module, a second upsampling module, a first splicing module, a second splicing module, a third splicing module, a fourth splicing module, a first C2f module, a second C2f module, a third C2f module, a fourth C2f module, a first CBS module, and a second CBS module;

[0028] The output end of the fusion part of one of the multimodal feature fusion modules is respectively connected to the input end of the first upsampling module and the first input end of the fourth splicing module, so as to output high-level multimodal fusion features; the output ends of the fusion parts of the other two multimodal feature fusion modules are respectively connected to the first input end of the first splicing module and the first input end of the second splicing module, so as to output mid-level and low-level multimodal fusion features respectively; the output end of the first upsampling module is connected to the second input end of the first splicing module, the output end of the first splicing module is connected to the input end of the first C2f module, the first output end and the second output end of the first C2f module are respectively connected to the input end of the second upsampling module and the first input end of the third splicing module, and are both used to output feature A;

[0029] The output end of the second up-sampling module is connected to the second input end of the second splicing module, the output end of the second splicing module is connected to the input end of the second C2f module, and the output end of the second C2f module is respectively connected to the first input end of the target prediction module and the input end of the first CBS module, for outputting the first feature;

[0030] The output end of the first CBS module is connected to the second input end of the third splicing module, the output end of the third splicing module is connected to the input end of the third C2f module, the third C2f module is used to output the second feature, its first output end is connected to the second input end of the target prediction module, its second output end is connected to the second input end of the fourth splicing module through the second CBS module, the output end of the fourth splicing module is connected to the third input end of the target prediction module through the fourth C2f module, and the fourth C2f module is used to output the third feature.

[0031] A multimodal feature fusion target detection method, which is special in that it includes the following steps:

[0032] Step 1: Obtain visible light images and corresponding infrared images, and construct training sets, validation sets, and test sets;

[0033] Step 2: construct the above multimodal feature fusion target detection model;

[0034] Step 3: Use the training set and validation set obtained in step 1 to train the multimodal feature fusion target detection model in step 2 to obtain a trained multimodal feature fusion target detection model;

[0035] Step 4: Use the test set obtained in step 1 to test the generalization performance of the multimodal feature fusion target detection model trained in step 3; if the preset standard is met, execute step 5, otherwise return to step 3; the preset standard is multiple thresholds of accuracy, precision, recall, F1-score and average precision mean;

[0036] Step 5: Input the visible light image and infrared image to be detected into the multimodal feature fusion target detection model trained in step 3 to obtain the prediction result, thereby completing the multimodal image feature fusion target detection.

[0037] Furthermore, the step 3 is specifically as follows:

[0038] Step 3.1, set the parameters of the model, including initial learning rate (lr0), weight decay, optimizer, momentum, batch size, image size, and total epochs;

[0039] Step 3.2: Load the training set and validation set and perform preprocessing and data enhancement;

[0040] Step 3.3, input the training set obtained in step 1 into the multimodal feature fusion target detection model of step 2 for forward propagation to obtain the output features, calculate the loss value according to the output features and the true label; calculate the gradient of the model parameters according to the loss value; update the model parameters according to the gradient and the optimizer; after each training cycle, use the validation set to verify the accuracy of the model; until the termination condition is reached, the trained multimodal feature fusion target detection model is obtained;

[0041] The termination condition is reaching the total training cycle or triggering the early stopping mechanism;

[0042] The early stopping mechanism is: if the accuracy does not increase within a number of training cycles, the early stopping mechanism is triggered and the training is stopped; the accuracy includes accuracy, precision, recall, F1-score and average precision. Compared with the prior art, the beneficial effects of the present invention are:

[0043] (1) A multimodal feature fusion target detection model of the present invention comprises an extraction branch network backbone 1, an extraction branch network backbone 2, three multimodal feature fusion modules, a fusion feature extraction module and a target prediction module; the present invention extracts low-level, middle-level and high-level visible light image features and infrared image features respectively, and processes the visible light image features and infrared image features of the same level through the multimodal feature fusion module, and then processes them through the fusion feature extraction module, and finally completes the target detection through the target prediction module; the present invention extracts features from low-level, middle-level and high-level, and through multiple upsampling and splicing fusion, it can more comprehensively describe the target, improve the detection effect, and make the model more stable in complex environments.

[0044] (2) A multimodal feature fusion target detection method of the present invention captures more information by fusing visible light and infrared image features, thereby improving detection accuracy; Map50 (Mean Average Precision at IoU=0.50, the mean of the average precision when the intersection-over-union ratio threshold is 0.50) and Map50-95 (Mean Average Precisionat IoU=0.50 to 0.95, the mean of the average precision when the intersection-over-union ratio threshold is from 0.50 to 0.95 (step size is 0.05)) of the method of the present invention reached 0.792 and 0.487, respectively, both of which are better than other comparison methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A schematic diagram of the structure of an embodiment of a multimodal feature fusion target detection model of the present invention;

[0046] Figure 2 Schematic diagram of the structure of a multimodal feature fusion module in an embodiment of the present invention;

[0047] Figure 3 A flowchart of an embodiment of a multimodal feature fusion target detection method of the present invention;

[0048] Figure 4 A prediction result diagram obtained by an embodiment of the present invention;

[0049] Figure 5 This is another prediction result diagram obtained according to an embodiment of the present invention.

[0050] The following are the descriptions of the reference numerals:

[0051] 2-Multimodal feature fusion module, P1-feature division part, P2-first attention module, P3-second attention module, P4-fusion part;

[0052] 3- fusion feature extraction module, 301- first upsampling module, 302- second upsampling module, 311- first splicing module, 312- second splicing module, 313- third splicing module, 314- fourth splicing module, 321- first C2f module, 322- second C2f module, 323- third C2f module, 324- fourth C2f module, 331- first CBS module, 332- second CBS module;

[0053] 4-Target prediction module. DETAILED DESCRIPTION

[0054] The present invention will be further described below in conjunction with the accompanying drawings and exemplary embodiments.

[0055] Reference Figure 1 A multimodal feature fusion target detection model includes an extraction branch network backbone1, an extraction branch network backbone2, three multimodal feature fusion modules 2, and a fusion feature extraction module 3 and a target prediction module 4 whose output and input are connected in sequence.

[0056] Extract branch network backbone1 for visible light image F rgb Extract low-level, mid-level, and high-level visible light image features The superscript k represents the kth layer, and the values ​​of k are 4, 6, and 9, corresponding to the low level, middle level, and high level, respectively.

[0057] Extract branch network backbone2 for infrared image F ir Extract low-level, medium-level and high-level infrared image features

[0058] The extraction branch network backbone1 and the extraction branch network backbone2 have the same structure, both including CBS, CBS, C2f, CBS, C2f, CBS, C2f, CBS, C2f, SPPF, whose outputs and inputs are connected in sequence.

[0059] Each multimodal feature fusion module 2 is used to combine the visible light image features at the same level and infrared image features Divide into initial similar features and initial specific features, and remove redundancy from the initial similar features to obtain similar features Remove redundancy from the initial specific features and strengthen feature processing to obtain specific features Then, similar features and specific features Perform element-by-element summation to output the multimodal fusion features of the corresponding level

[0060] Reference Figure 2 Each multimodal feature fusion module 2 includes a feature division part P1, a first attention module P2, a second attention module P3 and a fusion part P4; the output end of the feature division part P1 is respectively connected to the input end of the first attention module P2 and the second attention module P3, and the output end of the first attention module P2 and the second attention module P3 are both connected to the input end of the fusion part P4.

[0061] The working principle of the feature partitioning part P1 is as follows:

[0062] Step A1: Calculate the k-th layer visible light image features through the feature division part P1 and infrared image features The cosine similarity between the mth channels m (θ), as follows:

[0063]

[0064] Where m represents the mth channel in the kth layer feature, j represents the jth pixel in the mth channel, and its value ranges from 1 to n. Represents the visible light image features obtained at the kth layer The value of the jth pixel in the mth channel of Represents the infrared image features obtained at the kth layer The value of the th pixel in the mth channel; cosine similarity cos m (θ) takes the value [-1, 1];

[0065] Step A2, generating a similarity matrix C by calculating the cosine similarity of each channel of the kth layer;

[0066] Step A3: When extracting the initial similarity features, all values ​​greater than 0 in the similarity matrix C are set to 1, and all values ​​less than or equal to 0 are set to 0, to obtain a new similarity matrix C. com , the similarity matrix C com Respectively with the original input visible light image features and infrared image features Initial similarity features obtained by multiplication Similar features to the initial The formula is as follows:

[0067]

[0068]

[0069] When extracting the initial specific features, all values ​​greater than 0 in the similarity matrix C are set to 0, and all values ​​less than 0 are set to 1, and a new similarity matrix C is obtained. dif , the similarity matrix C dif Respectively with the original input visible light image features and infrared image features Multiply to get the initial specific features and initial specific features The formula is as follows:

[0070]

[0071]

[0072] The first attention module P2 is used to focus on the initial similar features Similar features to the initial Perform redundancy processing, remove repeated features, and output similar features The formula is as follows:

[0073]

[0074] The second attention module P3 is used to focus on the initial specific features and initial specific features Perform feature screening and feature enhancement, retain the advantages of different modal features while removing unimportant features and outputting specific features The formula is as follows:

[0075]

[0076] The fusion part P4 is used to combine similar features and specific features Perform element-by-element summation and output the multimodal fusion features of the corresponding level. The formula is as follows:

[0077]

[0078] The superscript k indicates the kth layer, and the values ​​of k are 4, 6, and 9, corresponding to the low level, middle level, and high level, respectively;

[0079] Fusion feature extraction module 3 is used to upsample high-level multimodal fusion features And combined with the multimodal features at the middle level Splice, extract and fuse the spliced ​​features to output feature A, upsample feature A and fuse it with the low-level multimodal features Splicing, further extracting and fusing the spliced ​​features to output the first feature to the target prediction module 4; and performing convolution, batch normalization and activation operations on the first feature, splicing it with feature A, and then outputting the second feature to the target prediction module 4 after feature extraction and fusion; and performing convolution, batch normalization and activation operations on the second feature, splicing it with a high-level multimodal fusion feature, and then outputting the third feature to the target prediction module 4 after feature extraction and fusion.

[0080] The fusion feature extraction module 3 includes a first upsampling module 301, a second upsampling module 302, a first splicing module 311, a second splicing module 312, a third splicing module 313, a fourth splicing module 314, a first C2ff module 321, a second C2f module 322, a third C2ff module 323, a fourth C2ff module 324, a first CBS module 331, and a second CBS module 332.

[0081] The output end of the fusion part P4 of one multimodal feature fusion module 2 is respectively connected to the input end of the first upsampling module 301 and the first input end of the fourth splicing module 314, for outputting high-level multimodal fusion features, and the output ends of the fusion parts P4 of the other two multimodal feature fusion modules 2 are respectively connected to the first input end of the first splicing module 311 and the first input end of the second splicing module 312, for outputting mid-level and low-level multimodal fusion features respectively; the output end of the first upsampling module 301 is connected to the second input end of the first splicing module 311, the output end of the first splicing module 311 is connected to the input end of the first C2f module 321, the first output end and the second output end of the first C2ff module 321 are respectively connected to the input end of the second upsampling module 302 and the first input end of the third splicing module 313, for outputting feature A; The output end of the sampling module 302 is connected to the second input end of the second splicing module 312, and the output end of the second splicing module 312 is respectively connected to the first input end of the target prediction module 4 and the input end of the first CBS module 331 through the second C2ff module 322, for outputting the first feature; the output end of the first CBS module 331 is connected to the second input end of the third splicing module 313, and the output end of the third splicing module 313 is connected to the input end of the third C2ff module 323, and the third C2f module 323 is used to output the second feature, and its first output end is connected to the second input end of the target prediction module 4, and the second output end is connected to the second input end of the fourth splicing module 314 through the second CBS module 332 in turn, and the output end of the fourth splicing module 314 is connected to the third input end of the target prediction module 4 through the fourth C2ff module 324, for outputting the third feature.

[0082] The principle of fusion feature extraction module 3 is as follows:

[0083] The high-level multimodal fusion features are fused by the first upsampling module 301 After upsampling, the first concatenation module 311 and the middle-level multimodal fusion features are input. The first C2f module 321 is then used to extract and fuse the features and output feature A. The second upsampling module 302 upsamples feature A and then inputs it into the second splicing module 312 and the low-level multimodal fusion feature splicing, extracting and fusing the output of the second splicing module 312 through the second C2ff module 322, and then outputting the first feature to the target prediction module 4 and the first CBS module 331 respectively; performing a convolution operation on the first feature through the first CBS module 331, and performing batch normalization and activation operations after the convolution, and then splicing the output of the first CBS module 331 with the feature A through the third splicing module 313, and then extracting and fusing the output of the third splicing module 313 through the third C2ff module 323, and then outputting the second feature to the target prediction module 4 and the second CBS module 332 respectively; performing a convolution operation on the second feature through the second CBS module 332, and performing batch normalization and activation operations after the convolution, and then combining the output of the second CBS module 332 with the high-level multimodal fusion feature through the fourth splicing module 314 The fourth C2f module 324 performs feature extraction and fusion on the output of the fourth splicing module 314 and then outputs the third feature to the target prediction module 4.

[0084] The target prediction module 4 is used to obtain a prediction result according to the first feature, the second feature and the third feature, and the prediction result includes a bounding box, a category and a confidence level of the target.

[0085] Reference Figure 3 , a multimodal feature fusion target detection method, comprising the following steps:

[0086] Step 1: Obtain visible light images and infrared images, and construct training sets, validation sets, and test sets;

[0087] Step 2: construct the above multimodal feature fusion target detection model;

[0088] Step 3: Use the training set and validation set obtained in step 1 to train the multimodal feature fusion target detection model in step 2 to obtain a trained multimodal feature fusion target detection model, as follows:

[0089] Step 3.1, set the parameters of the model as follows: initial learning rate = 0.01, weight decay = 0.0005, optimizer is SGD, momentum = 0.937, batch size = 32, image size = 640, total training cycle = 1000;

[0090] Step 3.2, load the training set and perform preprocessing and data enhancement. The data enhancement methods include Mosaic, random angle transformation, MixUp, copy and paste, and brightness adjustment to increase the diversity of input data;

[0091] Step 3.3, input the training set obtained in step 1 into the multimodal feature fusion target detection model of step 2 for forward propagation to obtain the output features, calculate the loss value according to the output features and the true label; calculate the gradient of the model parameters according to the loss value; update the model parameters according to the gradient and the optimizer; after each training cycle, verify the accuracy of the model on the verification set; until the termination condition is reached, the trained multimodal feature fusion target detection model is obtained;

[0092] The termination condition is reaching the total training cycle or triggering the early stopping mechanism;

[0093] The early stopping mechanism is: if the accuracy does not increase within a number of training cycles, the early stopping mechanism is triggered and training is stopped; accuracy includes accuracy, precision, recall, F1-score and average precision;

[0094] Step 4, using the test set obtained in step 1 to test the generalization performance of the multimodal feature fusion target detection model trained in step 3; if the preset standard is met, execute step 5, otherwise return to step 3; the preset standard is that the accuracy, precision, recall, F1-score and average precision mean all meet the threshold; specifically, the accuracy threshold is 95%, the precision threshold is 90%, the recall threshold is 90%, the F1-score threshold is 0.90, and the average precision mean threshold is 85%;

[0095] Step 5: Input the visible light image and infrared image to be detected into the multimodal feature fusion target detection model trained in step 3 to obtain the prediction result, thereby completing the multimodal image feature fusion target detection.

[0096] The prediction results include the bounding box, category and confidence of the target;

[0097] The non-maximum suppression method is used to remove the filtered target box to obtain the filtered prediction result, and the result is drawn on the input image to visualize the prediction result.

[0098] The experimental platform of this embodiment is based on the Windows operating system, a single NVIDIA GeForce RTX 3090 GPU, and the code running framework pytorch, torch 2.1.1, and python 3.10.

[0099] In order to verify the effectiveness of this implementation method, the prediction results of this implementation method and yolov8s are compared, and the results are shown in Table 1:

[0100] Table 1

[0101]

[0102]

[0103] In Table 1, Map50 and Map50-95 are used as evaluation indicators;

[0104] Yolov8-visible light refers to the detection performance of the Yolov8 model on visible light images, and Yolov8-infrared refers to the detection performance of the Yolov8 model on infrared images, which is generally lower than visible light; the CMAFF method performs well on the Map50 indicator, but is similar to Yolov8-visible light on Map50-95; the AFEFM method performs well on both indicators, especially the Map50-95 indicator; the GFU-v2 method is similar to the AFEFM method, and has good performance on both indicators; the MCFF method is better than Yolov8-infrared on both indicators, but not as good as the detection effect of visible light images; the ICAFusion method performs well on the Map50 indicator, but is slightly lower than AFEFM and GFU-v2 on the Map50-95 indicator; the multimodal image feature fusion target detection method performs best on both indicators, indicating that the present invention effectively combines the features of visible light and infrared images and improves the accuracy and robustness of target detection.

[0105] Figure 4 , Figure 5 This is a prediction result diagram of an embodiment of the present invention. Different colors represent different target types. The bounding box accurately surrounds the target. The letter in the upper right corner of the bounding box indicates the category to which the target belongs, and the number indicates the confidence level. There is no missed detection in both images, and the vehicle target contained in the image is effectively detected.

Claims

1. A multimodal feature fusion target detection method, characterized in that: A multimodal feature fusion target detection model is adopted, which includes an extraction branch network backbone 1, an extraction branch network backbone 2, three multimodal feature fusion modules (2), and a fusion feature extraction module (3) and a target prediction module (4) whose output and input are connected in sequence; The extraction branch network backbone1 is used to extract low-level, medium-level, and high-level visible light image features from the visible light image; the extraction branch network backbone2 is used to extract low-level, medium-level, and high-level infrared image features from the infrared image; Each of the multimodal feature fusion modules (2) is used to divide visible light image features and infrared image features at the same level into initial similar features and initial specific features, and perform redundancy removal processing on the initial similar features to obtain similar features, perform redundancy removal and feature enhancement processing on the initial specific features to obtain specific features, and then perform element-by-element summation on the similar features and the specific features to output multimodal fusion features at the corresponding level; the multimodal feature fusion module (2) comprises a feature division part (P1), a first attention module (P2), a second attention module (P3) and a fusion part (P4); the output end of the feature division part (P1) is respectively connected to the input end of the first attention module (P2) and the second attention module (P3), and the output ends of the first attention module (P2) and the second attention module (P3) are both connected to the input end of the fusion part (P4); The working principle of the feature partitioning part (P1) is as follows: Step A1: Calculate the first Layer visible light image features and infrared image features No. Cosine similarity between channels ,as follows: in, Indicates The first layer of features channels, Indicates The first pixels, with a value of 1~ , Indicates Visible light image features obtained by layer No. In the channel The value of pixels, Indicates Infrared image features obtained by layer No. In the channel The value of pixels; cosine similarity The value is [-1,1]; Step A2: By calculating The cosine similarity of each channel of the layer generates a similarity matrix C; Step A3: When extracting the initial similarity features, all values ​​greater than 0 in the similarity matrix C are set to 1, and all values ​​less than or equal to 0 are set to 0, to obtain a new similarity matrix , the similarity matrix Respectively compared with the original input visible light image features and infrared image features Initial similarity features obtained by multiplication Similar features to the initial , the formula is as follows: ； When extracting the initial specific features, all values ​​greater than 0 in the similarity matrix C are set to 0, and all values ​​less than 0 are set to 1, and a new similarity matrix is ​​obtained. , the similarity matrix Respectively with the original input visible light image features and infrared image features Multiply to get the initial specific features and initial specific features , the formula is as follows: ； The first attention module (P2) is used to focus on the initial similar features Similar features to the initial Perform redundancy removal and output similar features to the fusion part (P4); The second attention module (P3) is used to remove redundancy and strengthen feature processing of the initial specific features, and output the specific features to the fusion part (P4); The fusion part (P4) is used to sum similar features and specific features element by element, and output multimodal fusion features of the corresponding level; The fusion feature extraction module (3) is used to upsample the high-level multimodal fusion features and splice them with the middle-level multimodal fusion features, extract and fuse the spliced ​​features to output feature A, upsample feature A and splice it with the low-level multimodal fusion features, further extract and fuse the spliced ​​features to output the first feature to the target prediction module (4); and to perform convolution, batch normalization and activation operations on the first feature, splice it with feature A, and then output the second feature to the target prediction module (4) after feature extraction and fusion; and to perform convolution, batch normalization and activation operations on the second feature, splice it with the high-level multimodal fusion features, and then output the third feature to the target prediction module (4) after feature extraction and fusion; The target prediction module (4) is used to obtain a prediction result based on the first feature, the second feature and the third feature, and the prediction result includes a bounding box, a category and a confidence level of the target; The multimodal feature fusion target detection method includes the following steps: Step 1: Obtain visible light images and corresponding infrared images, and construct training sets, validation sets, and test sets; Step 2: construct the above multimodal feature fusion target detection model; Step 3: Use the training set and validation set obtained in step 1 to train the multimodal feature fusion target detection model in step 2 to obtain a trained multimodal feature fusion target detection model; Step 4: Use the test set obtained in step 1 to test the generalization performance of the multimodal feature fusion target detection model trained in step 3; if the preset standard is met, execute step 5, otherwise return to step 3; the preset standard is multiple thresholds of accuracy, precision, recall, F1-score and average precision mean; Step 5: Input the visible light image and infrared image to be detected into the multimodal feature fusion target detection model trained in step 3 to obtain the prediction result, thereby completing the multimodal image feature fusion target detection.

2. The multimodal feature fusion target detection method according to claim 1, characterized in that: The extraction branch network backbone1 and the extraction branch network backbone2 have the same structure, and both include CBS, CBS, C2f, CBS, C2f, CBS, C2f, CBS, C2f, and SPPF, whose outputs and inputs are connected in sequence.

3. The multimodal feature fusion target detection method according to claim 2, characterized in that: The fusion feature extraction module (3) comprises a first up-sampling module (301), a second up-sampling module (302), a first splicing module (311), a second splicing module (312), a third splicing module (313), a fourth splicing module (314), a first C2f module (321), a second C2f module (322), a third C2f module (323), a fourth C2f module (324), a first CBS module (331), and a second CBS module (332); The output end of the fusion part (P4) of one of the multimodal feature fusion modules (2) is respectively connected to the input end of the first upsampling module (301) and the first input end of the fourth splicing module (314), and is used to output high-level multimodal fusion features; the output ends of the fusion parts (P4) of the other two multimodal feature fusion modules (2) are respectively connected to the first input end of the first splicing module (311) and the first input end of the second splicing module (312), and are respectively used to output mid-level and low-level multimodal fusion features; the output end of the first upsampling module (301) is connected to the second input end of the first splicing module (311), the output end of the first splicing module (311) is connected to the input end of the first C2f module (321), the first output end and the second output end of the first C2f module (321) are respectively connected to the input end of the second upsampling module (302) and the first input end of the third splicing module (313), and are both used to output feature A; The output end of the second up-sampling module (302) is connected to the second input end of the second splicing module (312), the output end of the second splicing module (312) is connected to the input end of the second C2f module (322), and the output end of the second C2f module (322) is respectively connected to the first input end of the target prediction module (4) and the input end of the first CBS module (331), for outputting the first feature; The output end of the first CBS module (331) is connected to the second input end of the third splicing module (313), the output end of the third splicing module (313) is connected to the input end of the third C2f module (323), the third C2f module (323) is used to output the second feature, the first output end thereof is connected to the second input end of the target prediction module (4), the second output end thereof is connected to the second input end of the fourth splicing module (314) through the second CBS module (332), the output end of the fourth splicing module (314) is connected to the third input end of the target prediction module (4) through the fourth C2f module (324), and the fourth C2f module (324) is used to output the third feature.

4. The multimodal feature fusion target detection method according to claim 1, characterized in that: The step 3 is specifically as follows: Step 3.

1. Set the model parameters, including initial learning rate, weight decay, optimizer, momentum, batch size, image size, and total training cycles. Step 3.2: Load the training set and validation set and perform preprocessing and data enhancement; Step 3.3, input the training set obtained in step 1 into the multimodal feature fusion target detection model of step 2 for forward propagation to obtain the output features, calculate the loss value according to the output features and the true label; calculate the gradient of the model parameters according to the loss value; update the model parameters according to the gradient and the optimizer; after each training cycle, use the validation set to verify the accuracy of the model; until the termination condition is reached, the trained multimodal feature fusion target detection model is obtained; The termination condition is reaching the total training cycle or triggering the early stopping mechanism; The early stopping mechanism is: if the accuracy no longer increases within several training cycles, the early stopping mechanism is triggered and the training is stopped; the accuracy includes accuracy, precision, recall, F1-score and average precision mean.

Images