Tri-modal image feature fusion target detection method and device

Abstract

The invention provides a three-modal image feature fusion target detection method and device, and the method comprises the steps: S100: building a visible light-infrared-low light three-modal data collection platform, obtaining three-modal data through the data collection platform, and constructing a three-modal data set based on the obtained three-modal data; and S200, constructing a three-modal image feature fusion target detection model based on a YOLOv5 model, training the three-modal image feature fusion target detection model by using the three-modal data set, and carrying out target detection by using the trained three-modal image feature fusion target detection model. Through three-mode fusion, the method can effectively solve the problems of limited imaging conditions, low imaging definition, single image features, difficult target recognition and the like in an all-day application scene, and has the characteristics of accurate positioning of small targets and shielded targets, high detection precision, all-weather performance and real-time performance.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision, pattern recognition and artificial intelligence technology, and in particular to a tri-modal image feature fusion target detection method and device. Background Art

[0002] The purpose of target detection is to search for objects of interest in images or videos and identify their location and size. It has been widely used in fields such as autonomous driving and civil security, and its technical routes are divided into single-stage and two-stage. The representative algorithms of two-stage target detection are mainly R-CNN (region based convolutional neural network) series. The more mature single-stage detection algorithms are SSD (single shot multi-box detector) series algorithms and YOLO (you only look once) series algorithms. The accuracy of single-stage target detection algorithm is slightly lower than that of two-stage target detection algorithm, but the detection speed is fast and can be detected in real time. YOLOv5 is a high-performance real-time target detection model with five different sizes of n / s / m / l / x. It can be flexibly selected according to different application scenarios and has been widely used in visible light image detection.

[0003] With the requirement of all-day application, a single visual sensor is difficult to meet the requirements. Multispectral target detection technology has emerged in due time, mainly based on the joint detection of visible light and infrared sensors. Visible light images have rich texture detail information, but are greatly affected by light; infrared images are suitable for all-weather work at night and under bad antennas, with long detection distance, but low contrast and blurred edges. This involves the fusion technology of visible light and infrared, including front fusion, feature fusion and decision-level fusion. Wang Yuting et al. adopted the strategy of fusion first and then detection, and proposed a visible light image and infrared image fusion model DAPR-Net. The model is mainly composed of an encoder and a decoder. At the same time, a cross-layer residual connection is designed to improve the effect of visible light and infrared image fusion and enhance the performance of target detection under weak light conditions of the network. He Nan et al. proposed a feature fusion network architecture, introduced spatial and channel attention mechanisms, and integrated the extracted infrared features into the visible light features of the corresponding dimensions in an adaptive weighted manner to fully compensate for the limitations of the target under the single-modal model, and applied it in actual power grid equipment detection. Fang Qingyun et al. proposed a Transformer-based feature fusion network that can fully obtain the potential feature relationship between visible light and infrared. The detection performance after feature fusion is better than that of single-modal detection. However, the introduced feature fusion module will significantly increase the number of parameters and cannot be deployed in a lightweight manner. Ning Dahai et al. adopted a strategy of first detection and then fusion and proposed a decision-level fusion algorithm based on model reliability, which improved the performance of the visible light and infrared image decision-level fusion target detection algorithm. Although feature fusion is more complex than the other two fusions, it can extract features more effectively, so that the fusion algorithm meets the requirements of high accuracy, fast detection, and real-time performance.

[0004] In summary, the shortcomings of the existing technology are: visible light images have poor performance in target detection under conditions such as low light, darkness, and occlusion; infrared images are not conducive to target detection due to low contrast, blurred edges, and few local texture details; low-light imaging is to capture the reflected light in the scene for imaging, which has a high ability to represent texture details under low light conditions, but the detection distance is limited. The target detection models obtained by the pre-fusion and post-fusion methods have poor detection performance. Summary of the invention

[0005] The present invention aims to provide a three-modal image feature fusion target detection method and device to solve the problems of limited imaging conditions, low imaging clarity, single image features, and difficult target recognition of a single visual sensor in all-day application scenarios.

[0006] The present invention provides a three-modal image feature fusion target detection method, comprising:

[0007] S100: Building a visible light-infrared-low light tri-modal data acquisition platform, using the data acquisition platform to acquire tri-modal data, and constructing a tri-modal data set based on the acquired tri-modal data;

[0008] S200: Construct a trimodal image feature fusion target detection model based on the YOLOv5 model, train the trimodal image feature fusion target detection model using the trimodal data set, and perform target detection using the trained trimodal image feature fusion target detection model.

[0009] In some embodiments, step S100 includes the following sub-steps:

[0010] S101: Build a visible light, infrared, and low-light tri-modal data acquisition platform. When building, you need to consider the focal length, field of view, and relative positions of the three cameras for subsequent image registration.

[0011] S102: Collect tri-modal data using the visible light-infrared-low light tri-modal data collection platform;

[0012] S103: Perform image registration on the collected data and label them using annotation software to obtain a tri-modal data set.

[0013] In some embodiments, in step S200, the constructing of a tri-modal image feature fusion target detection model based on a YOLOv5 model includes:

[0014] (1) Replace the Backbone of the YOLOv5 model with the MobileNetV3-large feature extraction layer;

[0015] (2) A cross-modal fusion module is introduced into the YOLOv5 model for tri-modal feature fusion;

[0016] (3) Replace the coupling head of the Head in the YOLOv5 model with the decoupling head.

[0017] In some embodiments, the MobileNetV3-large feature extraction layer includes multiple standard convolutional layers and an inverse residual module.

[0018] In some embodiments, the MobileNetV3-large feature extraction layer includes 17 layers, wherein the 1st and 17th layers are standard convolutional layers, and the 2nd to 16th layers are inverse residual modules.

[0019] In some embodiments, the MobileNetV3-large feature extraction layer is divided into three parts, where:

[0020] Layers 1 to 7 are the first part;

[0021] The 8th to 13th floors are the second part;

[0022] The 14th to 17th floors are the third part;

[0023] The cross-modal fusion module is connected between the first part and the second part.

[0024] In some embodiments, the processing process in the tri-modal image feature fusion target detection model is as follows:

[0025] S201, the three modal image data of visible light, infrared and low light are simultaneously and parallelly input into the 1st to 7th layers of the MobileNetV3-large feature extraction layer in the Backbone of the YOLOv5 model for feature extraction;

[0026] S202: The shallow features extracted from the 1st to 7th layers of the MobileNetV3-large feature extraction layer are input into the cross-modal fusion module;

[0027] S203: After the features processed by the cross-modal fusion module are added to the shallow features of the original modal branch, one path is input into the 8th to 13th layers of the MobileNetV3-large feature extraction layer for feature extraction, and the other path is input into Neck;

[0028] S204: The feature maps outputted by the 8th to 13th layers of the three-way parallel MobileNetV3-large feature extraction layer are respectively inputted into the 14th to 17th layers of the MobileNetV3-large feature extraction layer for feature extraction; on the other hand, they are added and inputted into Neck;

[0029] S205: The high-level feature maps obtained after feature extraction of the 14th to 17th layers of the three-way parallel MobileNetV3-large feature extraction layer are added and input into Neck;

[0030] S206: The features extracted from Backbone are input into Neck and Head for subsequent processing.

[0031] In some embodiments, in step S202, the processing of the cross-modal fusion module includes:

[0032] S2021: Given a visible light feature map, an infrared convolution feature map, and a low-light feature map, obtain the corresponding modal sequences by flattening each feature map and arranging the order of the matrix;

[0033] S2022: concatenate each modality sequence and add a learnable position embedding to obtain the input sequence of the cross-modality fusion module;

[0034] S2023: perform position encoding;

[0035] S2024: Project the input sequence onto three weight matrices to compute a set of queries, keys, and values;

[0036] S2025: Input the output Z′ of the multi-head attention mechanism based on the query, key and value. It contains multiple self-attention layers. Pass the input to h different self-attention layers respectively to calculate the output matrix Z i , i=1,2,...,h, the output matrix Z i After splicing, projection is performed to obtain the final output matrix Z′;

[0037] S2026: For the output matrix Z′, Transform uses a two-layer fully connected feed-forward neural network with a GELU(·) activation in the middle to compute the output sequence O, which has the same shape as the input sequence I;

[0038] S2027: Using the inverse transformation of step S2021, the output sequence O is converted into a recalibrated result and added to the original modal branch as complementary information.

[0039] In some embodiments, in step S206, the decoupling head processing in the Head includes:

[0040] S2061: Get feature maps of three sizes from Backbone, and get three new feature maps through convolution, upsampling and concatenation;

[0041] S2062: The three new feature maps are respectively input into the decoupling head. In the decoupling head, channel 6 is compressed by 1×1Conv, and features are extracted by 3×3Conv. Finally, they are decoupled into three-way prediction results. Finally, the three prediction results are stacked to obtain the final result.

[0042] The present invention also provides a three-modal image feature fusion target detection device, comprising:

[0043] Visible light-infrared-low light tri-modal data acquisition platform, used to obtain tri-modal data;

[0044] A first processing unit, configured to construct a tri-modal data set based on the acquired tri-modal data;

[0045] The second processing unit is used to build a trimodal image feature fusion target detection model based on the YOLOv5 model, train the trimodal image feature fusion target detection model using the trimodal data set, and perform target detection using the trained trimodal image feature fusion target detection model.

[0046] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0047] 1. Compared with a single visual sensor, the present invention can effectively solve the problems of limited imaging conditions, low imaging clarity, single image features, and difficult target recognition in all-day application scenarios through three-modal fusion. It has the characteristics of accurate positioning of small targets and occluded targets, high detection accuracy, all-weather and real-time performance; it is not only suitable for image detection, but also for dynamic target detection in visible light, infrared and low-light video streams.

[0048] 2. The tri-modal image feature fusion target detection model trained by the present invention can be deployed on mobile terminals such as drones and unmanned vehicles. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is a flow chart of a three-modal image feature fusion target detection method in an embodiment of the present invention.

[0050] Figure 2 It is an architecture diagram of a tri-modal image feature fusion target detection model based on the YOLOv5 model in an embodiment of the present invention.

[0051] Figure 3 4 is a structural diagram of an inverted residual module (Inverted Residual) in an embodiment of the present invention.

[0052] Figure 4 It is a structural diagram of the cross-modal fusion module in an embodiment of the present invention.

[0053] Figure 5 4 is a structural diagram of a decoupled head in an embodiment of the present invention.

[0054] Figure 6 Schematic diagram of the structure of a three-modal image feature fusion target detection device in an embodiment of the present invention. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0056] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] In some embodiments, Figure 1 As shown, a three-modal image feature fusion target detection method is disclosed, comprising the following steps:

[0058] S100: Building a visible light-infrared-low light tri-modal data acquisition platform, using the data acquisition platform to acquire tri-modal data, and constructing a tri-modal data set based on the acquired tri-modal data;

[0059] S200: Construct a trimodal image feature fusion target detection model based on the YOLOv5 model, train the trimodal image feature fusion target detection model using the trimodal data set, and perform target detection using the trained trimodal image feature fusion target detection model.

[0060] In some embodiments, step S100 includes the following sub-steps:

[0061] S101: Build a visible light, infrared, and low-light tri-modal data acquisition platform. When building, you need to consider the focal length, field of view and other parameters of the three cameras, as well as the relative positions of the three cameras for subsequent image registration;

[0062] Among them, the visible light-infrared-low-light tri-modal data acquisition platform can detect targets simultaneously photographed by visible light, infrared and low-light cameras in real time, and is less affected by natural conditions such as fog, lighting, and occlusion.

[0063] S102: Collect tri-modal data using the visible light-infrared-low light tri-modal data collection platform;

[0064] The visible light-infrared-low light tri-modal data acquisition platform can be mounted on a vehicle or a drone, and the collected tri-modal data can be temporarily stored in a terminal of the data acquisition platform.

[0065] It should be noted that the richness of the dataset needs to be fully considered during collection. The richness of the dataset is reflected in three aspects: First, the richness of the scene; for example, city, field. Second, the richness of the target; for example, pedestrians, motor vehicles, non-motor vehicles, poultry, etc. Third, the richness of the lighting; for example, daytime, dim, night. Fourth, the richness of the target size; for example, large (1m~10m), medium (11m~20m), small (21m~50m).

[0066] S103: Perform image registration on the collected data and label them using annotation software to obtain a trimodal data set. For subsequent model training, the trimodal data set needs to be divided into a training set, a validation set, and a test set in sequence according to a set ratio (e.g., 6:2:2).

[0067] In some embodiments, in step S200, the constructing of a tri-modal image feature fusion target detection model based on a YOLOv5 model includes:

[0068] (1) Considering the excellent performance of MobileNetV3 in lightweight, the Backbone of the YOLOv5 model is replaced with the MobileNetV3-large feature extraction layer;

[0069] (2) Considering the advantages of Transformer in feature extraction, the Cross Modality Fusion Transformer (CMFT) module is introduced into the YOLOv5 model for tri-modal feature fusion;

[0070] (3) Considering the characteristics of faster convergence and higher accuracy of the decoupled head, the coupled head of the Head in the YOLOv5 model is replaced by the decoupled head.

[0071] The specific operations of constructing the tri-modal image feature fusion target detection model based on the YOLOv5 model are as follows: construct a virtual environment for running the YOLOv5 model on a computer, replace Backbone with the MobileNetV3-large feature extraction layer at the code level based on the original project file of the YOLOv5 model, add a cross-modal fusion module, replace the coupling head of the Head with the decoupling head, and download the pre-trained weights.

[0072] like Figure 2 , Figure 3As shown, the MobileNetV3-large feature extraction layer includes multiple standard convolutional layers (ConvBNActivation) and inverted residual modules (InvertedResidual). The specific structure is as follows: the MobileNetV3-large feature extraction layer includes 17 layers, of which the 1st and 17th layers are standard convolutional layers, and the 2nd to 16th layers are inverted residual modules; since the width and height requirements of the feature map passed into Neck are 80×80, 40×40 and 20×20, the MobileNetV3-large feature extraction layer can be divided into three parts: MobileNetV3-large Feature ExtractionLayer 1-7 (1st to 7th layers), MobileNetV3-large Feature Extraction Layer 8-13 (8th to 13th layers) and MobileNetV3-large Feature Extraction Layer 14-17 (14th to 17th layers), the output feature map sizes are 80×80×40, 40×40×112 and 20×20×960 respectively; then, a cross-modal fusion module is added after MobileNetV3-largeFeature Extraction Layer 1-7, the Neck remains unchanged, and the coupling head of the Head is replaced by a decoupling head, thereby completing the construction of a tri-modal image feature fusion target detection model based on the YOLOv5 model, and the tri-modal image feature fusion target detection model is trained using the tri-modal dataset.

[0073] In some embodiments, Figure 2 As shown, the processing process in the tri-modal image feature fusion target detection model is as follows:

[0074] S201, the three modal image data of visible light, infrared and low light are simultaneously and parallelly input into the MobileNetV3-large Feature Extraction Layer 1-7 (1st to 7th layers) in the Backbone of the YOLOv5 model for feature extraction;

[0075] S202: The shallow features extracted by MobileNetV3-large Feature Extraction Layer 1-7 (1st to 7th layers) are input into the cross-modal fusion module;

[0076] S203: After the features processed by the cross-modal fusion module are added to the shallow features of the original modal branch (feature map size 80×80×40), one path continues to be input into MobileNetV3-large Feature ExtractionLayer 8-13 (8th to 13th layers) for feature extraction, and the other path is input into the subsequent Neck;

[0077] S204: The feature maps (feature map size 40×40×112) output by three parallel MobileNetV3-large Feature Extraction Layers 8-13 (layers 8 to 13) are input into MobileNetV3-largeFeature Extraction Layers 14-17 (layers 14 to 17) for feature extraction. On the other hand, they are added and input into the subsequent Neck.

[0078] S205: The high-level feature maps (feature map size 20×20×960) obtained after feature extraction by three parallel MobileNetV3-large Feature Extraction Layers 14-17 (14th to 17th layers) are added and input into the subsequent Neck;

[0079] For example, the feature extraction layer of MobileNetV3-large inputs the feature map of (1, 3, 640, 640), and the outputs of each layer are shown in Table 1.

[0080] Table 1, output of each layer of the feature extraction part of MobileNetV3-large:

[0081]

[0082] S206: The features extracted from Backbone are input into Neck and Head for subsequent processing.

[0083] Through the above process, it can be known that the present invention improves the Backbone part of the YOLOv5 model, replaces it with the lightweight model MobileNetV3-large feature extraction layer, and adds a cross-modal fusion module. The Neck part is retained, and the coupling head of the Head is replaced with a decoupling head. Considering that trimodal fusion will cause a substantial increase in training costs, and the cross-modal fusion module will take up a lot of computing resources, the cross-modal fusion module is only introduced after MobileNetV3-large FeatureExtraction Layer 1-7 (1st to 7th layers). In addition, it should be noted that since the features extracted from the Backbone part must also be input into the Neck, the replacement of the Backbone must take into account the matching problem of the feature map size.

[0084] In some embodiments, the structure of the cross-modal fusion module is as follows: Figure 4 As shown, the processing of the cross-modal fusion module in the above step S202 includes the following steps:

[0085] S2021: Given a visible light feature map F R ∈R H×W×C , infrared convolution feature map F T ∈R H×W×C and the low-light feature map F L ∈R H×W×C By flattening each feature map and arranging the order of the matrix, the corresponding modal sequence I is obtained. R ∈R HW×C , I T ∈R HW×C and I L ∈R HW×C According to step S201, F R 、F T and F L The size of each is 80×80×40, and the obtained modal sequence I R ,I T and I L The size is 6400×40.

[0086] S2022: Concatenate each modality sequence and add a learnable position embedding with a dimension of 3HW×C to obtain the input sequence I∈R of the cross-modal fusion module 3HW×C , with a size of 19200×40.

[0087] S2023: Perform positional embedding (PE).

[0088] S2024: Input sequence I∈R 3HW×CProjected onto three weight matrices to compute a set of queries (Q), keys (K), and values ​​(V):

[0089] Q=IW Q

[0090] K=IW K

[0091] V=IW V

[0092] in, and is the weight matrix, where C is the number of channels, and D Q =D K =D V =C are the number of matrix columns. According to step S2022, W Q ,W K and W V The size is 40×40.

[0093] S2025: Input the output Z′ of the multi-head attention mechanism according to the query, key and value. It contains multiple self-attention layers. The input is passed to h different self-attention layers respectively, and the output matrix Z is calculated. i ,i=1,2,...,h,Z i After splicing, projection is performed to obtain the final output matrix Z′, Z i The size is 19200×40.

[0094]

[0095] Z′=MultiHead(Q,K,V)=Concat(Z 1 ,...,Z h )W O

[0096] Where, the subscript h represents the number of heads, W O ∈R hC×C Concat(Z 1 ,...,Z h )’s projection matrix; It is a scaling factor used to prevent the Softmax function from falling into a region with extremely small gradient when the value of the dot product becomes larger.

[0097] S2026: For the output matrix Z′, Transform uses a two-layer fully connected feedforward neural network (Multi-Layer Perceptron, MLP) with a GELU (·) activation in the middle to calculate the output sequence O, which has the same shape as the input sequence I. The output sequence O is expressed as:

[0098] O=MLP(Z″)+Z″=FC 2 (GELU(FC 1 (Z″)))+Z″

[0099] Among them, Z″=Z′+I.

[0100] S2027: Using the inverse transformation of step S2021, the output sequence O is converted into the recalibration result F R ′、F T ′ and F L ′, and added to the original modal branch as complementary information, F R ′、F T ′ and F L The size of the ' is still 80×80×40.

[0101] From the above steps, it can be seen that in the cross-modal fusion module of the present invention, with the help of Transformer, the self-attention mechanism is used to learn the binary relationship between the three modalities of visible light, infrared and low light, that is, the correlation matrix is ​​used to weight each position of the input feature map.

[0102]

[0103] Among them, α i,j Represents the correlation between the i-th position and the j-th position on the feature map.

[0104] In some embodiments, the structure of the decoupling head is as follows Figure 5 As shown, the above step S206 includes the following steps:

[0105] S2061: Get three feature maps of different sizes from Backbone: 80×80×40, 40×40×112 and 20×20×960. After a series of convolution, upsampling and concatenation operations inside Neck, get three new feature maps: 80×80×256, 40×40×512 and 20×20×1024.

[0106] S2062: Replace the coupling head of the Head part with the decoupling head, input the three new feature maps obtained in step S2061 into the decoupling head respectively, 1×1Conv is used to compress the channel to 256, 3×3Conv is used for feature extraction, and finally decoupled into three-way prediction results:

[0107] Cls.(H×W×C) is used to determine the type of object contained in each feature point;

[0108] Reg.(H×W×4) is used to determine the regression parameters of each feature point. After the regression parameters are adjusted, the prediction box can be obtained;

[0109] Obj(H×W×1), also known as IoU, is used to determine whether each feature point contains the target.

[0110] Finally, the three prediction results are stacked and the output is H×W×(C+4+1).

[0111] From the above steps, we can see that although the decoupling head increases the computational complexity to a certain extent, it can greatly improve the convergence speed of the network.

[0112] Based on the same inventive concept, in some embodiments, Figure 6 As shown, a three-modal image feature fusion target detection device is also provided, comprising:

[0113] Visible light-infrared-low light tri-modal data acquisition platform, used to obtain tri-modal data;

[0114] A first processing unit, configured to construct a tri-modal data set based on the acquired tri-modal data;

[0115] The second processing unit is used to build a trimodal image feature fusion target detection model based on the YOLOv5 model, train the trimodal image feature fusion target detection model using the trimodal data set, and perform target detection using the trained trimodal image feature fusion target detection model.

[0116] The specific implementation working principles of the above-mentioned visible light-infrared-low light tri-modal data acquisition platform, the first processing unit and the second processing unit can refer to the aforementioned tri-modal image feature fusion target detection method, which will not be repeated here.

[0117] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A tri-modal image feature fusion target detection method, characterized in that: include: S100: Building a visible light-infrared-low light tri-modal data acquisition platform, using the data acquisition platform to acquire tri-modal data, and constructing a tri-modal data set based on the acquired tri-modal data; S200: Construct a trimodal image feature fusion target detection model based on the YOLOv5 model, train the trimodal image feature fusion target detection model using the trimodal data set, and perform target detection using the trained trimodal image feature fusion target detection model.

2. The tri-modal image feature fusion target detection method according to claim 1, characterized in that: Step S100 includes the following sub-steps: S101: Build a visible light, infrared, and low-light tri-modal data acquisition platform. When building, you need to consider the focal length, field of view, and relative positions of the three cameras for subsequent image registration. S102: Collect tri-modal data using the visible light-infrared-low light tri-modal data collection platform; S103: Perform image registration on the collected data and label them using annotation software to obtain a tri-modal data set.

3. The tri-modal image feature fusion target detection method according to claim 1, characterized in that: In step S200, the construction of a tri-modal image feature fusion target detection model based on the YOLOv5 model includes: (1) Replace the Backbone of the YOLOv5 model with the MobileNetV3-large feature extraction layer; (2) A cross-modal fusion module is introduced into the YOLOv5 model for tri-modal feature fusion; (3) Replace the coupling head of the Head in the YOLOv5 model with the decoupling head.

4. The tri-modal image feature fusion target detection method according to claim 3, characterized in that: The MobileNetV3-large feature extraction layer includes multiple standard convolutional layers and an inverse residual module.

5. The tri-modal image feature fusion target detection method according to claim 4, characterized in that: The MobileNetV3-large feature extraction layer includes 17 layers, among which the 1st and 17th layers are standard convolutional layers, and the 2nd to 16th layers are inverse residual modules.

6. The tri-modal image feature fusion target detection method according to claim 5, characterized in that: The feature extraction layer of MobileNetV3-large is divided into three parts: Layers 1 to 7 are the first part; The 8th to 13th floors are the second part; The 14th to 17th floors are the third part; The cross-modal fusion module is connected between the first part and the second part.

7. The tri-modal image feature fusion target detection method according to claim 6, characterized in that: The processing process in the tri-modal image feature fusion target detection model is as follows: S201, the three modal image data of visible light, infrared and low light are simultaneously and parallelly input into the 1st to 7th layers of the MobileNetV3-large feature extraction layer in the Backbone of the YOLOv5 model for feature extraction; S202: The shallow features extracted from the 1st to 7th layers of the MobileNetV3-large feature extraction layer are input into the cross-modal fusion module; S203: After the features processed by the cross-modal fusion module are added to the shallow features of the original modal branch, one path is input into the 8th to 13th layers of the MobileNetV3-large feature extraction layer for feature extraction, and the other path is input into Neck; S204: The feature maps outputted by the 8th to 13th layers of the three-way parallel MobileNetV3-large feature extraction layer are respectively inputted into the 14th to 17th layers of the MobileNetV3-large feature extraction layer for feature extraction; on the other hand, they are added and inputted into Neck; S205: The high-level feature maps obtained after feature extraction of the 14th to 17th layers of the three-way parallel MobileNetV3-large feature extraction layer are added and input into Neck; S206: The features extracted from Backbone are input into Neck and Head for subsequent processing.

8. The tri-modal image feature fusion target detection method according to claim 7, characterized in that: In step S202, the processing of the cross-modal fusion module includes: S2021: Given a visible light feature map, an infrared convolution feature map, and a low-light feature map, obtain the corresponding modal sequences by flattening each feature map and arranging the order of the matrix; S2022: concatenate each modality sequence and add a learnable position embedding to obtain the input sequence of the cross-modality fusion module; S2023: perform position encoding; S2024: Project the input sequence onto three weight matrices to compute a set of queries, keys, and values; S2025: Input the output Z′ of the multi-head attention mechanism based on the query, key and value. It contains multiple self-attention layers. Pass the input to h different self-attention layers respectively to calculate the output matrix Z i , i=1,2,…,h, the output matrix Z i After splicing, projection is performed to obtain the final output matrix Z′; S2026: For the output matrix Z′, Transform uses a two-layer fully connected feed-forward neural network with a GELU(·) activation in the middle to compute the output sequence O, which has the same shape as the input sequence I; S2027: Using the inverse transformation of step S2021, the output sequence O is converted into a recalibrated result and added to the original modal branch as complementary information.

9. The tri-modal image feature fusion target detection method according to claim 7, characterized in that: In step S206, the decoupling head processing in the Head includes: S2061: Get feature maps of three sizes from Backbone, and get three new feature maps through convolution, upsampling and concatenation; S2062: The three new feature maps are respectively input into the decoupling head. In the decoupling head, channel 6 is compressed by 1×1Conv, and features are extracted by 3×3Conv. Finally, they are decoupled into three-way prediction results. Finally, the three prediction results are stacked to obtain the final result.

10. A three-modal image feature fusion target detection device, characterized in that: include: Visible light-infrared-low light tri-modal data acquisition platform, used to obtain tri-modal data; A first processing unit, configured to construct a tri-modal data set based on the acquired tri-modal data; The second processing unit is used to build a trimodal image feature fusion target detection model based on the YOLOv5 model, train the trimodal image feature fusion target detection model using the trimodal data set, and perform target detection using the trained trimodal image feature fusion target detection model.

Images