A target detection method, system, device and medium based on cross-modal fusion and guided attention mechanism

Abstract

A target detection method, system, device and medium based on cross-modal fusion and guided attention mechanism, the target detection method comprising: acquiring a visible-infrared image paired dataset, processing and dividing the dataset to obtain a training set, a validation set and a test set; constructing a multi-modal target detection network; setting network training parameters; training and optimizing the multi-modal target detection network using the training set, outputting a training weight file after training is completed, and verifying the weight file using the validation set to select the weight file with the highest precision as the optimal weight file; loading the test set and the optimal weight file into the multi-modal target detection network to detect targets in the test set and obtain a target detection result; the system, device and medium are used to carry and implement the method; the application has lower false detection and error detection, and improves the robustness and accuracy of target detection.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of target detection, and particularly relates to a target detection method, system, device and medium based on cross-modal fusion and guided attention mechanism. BACKGROUND

[0002] In recent years, target detection algorithms have developed rapidly and are widely used in various practical applications such as automatic driving, unmanned aerial vehicles and robots. Through processing the scene pictures captured by visual sensors, they identify and locate the target and region of interest. Visual target detection based on a single visible modality (RGB) has achieved great success, and most existing intelligent agents (such as unmanned aerial vehicles, robots, etc.) can complete tasks with detectors based on single-mode sensors. Unfortunately, in real-world scenarios, there are many challenges such as lighting changes, adverse weather conditions and geographical changes, and these detectors are very limited in harsh environments (rain, fog, weak light). Due to the influence of challenging environmental conditions such as adverse weather conditions and changing lighting, a single sensor is limited in capturing environmental information.

[0003] Compared with single modality, multi-modality provides richer target visual information, and they have complementary interactions. Under good lighting conditions, visible light sensors perform well in capturing target information such as color and texture details, but it is difficult to distinguish target details through visible light modality in poor lighting conditions, while infrared sensors can still provide information about enemies, vehicle movements and other factors due to their unique imaging mechanism, so visible light images and infrared images have uniqueness and complementarity. Inspired by this, in order to achieve more reliable and robust target detection in changing lighting and adverse weather conditions, multispectral imaging and fusion technology has been introduced, which has recently attracted increasing attention.

[0004] Target detection has always been a very key technology in computer vision tasks, and deep learning-based target detection algorithms can be divided into two categories: candidate-based and regression-based. Candidate-based target detection algorithms include FastR-CNN, Mask R-CNN and Cascade R-CNN, which are all based on R-CNN as the basic framework. Regression-based target detection algorithms such as YOLO series and SSD series, researchers have also developed infrared-based object detection algorithms, which are developed from visible object detection algorithms. Existing multi-modal target detection algorithms can be divided into traditional methods and deep learning methods, such as weight fusion, aggregated channel features, and using feature pyramid to fuse features at each layer.

[0005] An agent finds the location of the target of interest from the pictures or videos taken by carrying sensors, using target detection algorithms, optimizing end-to-end models, and achieving continuous and real-time tracking and detection of the target of the task. The challenges of existing target detection algorithms include: 1) In actual scenes, there are problems of insufficient color information, weak energy and low resolution in infrared images. In some special environments, such as night, overexposure, visible light images are difficult to provide the outline and detail information of the target. Using only one modality for object detection is difficult to adapt to harsh environments and obtain satisfactory detection results. 2) Most fusion methods usually fuse and enhance feature representations by using intra-modality information or weight fusion strategies to achieve multi-modal fusion, ignoring the relationship between different modalities. 3) Existing methods are mostly based on CNN networks, which are limited by limited receptive fields and can only capture local information, so they cannot take advantage of the long-range dependencies within and between modalities. Visible-infrared sensor fusion can effectively capture more comprehensive target information and obtain more stable detection results by combining data from different spectra, improving the all-weather perception ability of the agent.

[0006] Several multi-modal target detection methods have been proposed.

[0007] For example, the patent application file with publication number CN118154844A discloses a multi-modal target detection method based on contrast learning. The method first acquires training data, including a training sample set and a test sample set; (2) a multi-modal target detection model based on contrast learning is constructed, and a loss function of the model is designed to iteratively train the processed data. (3) The image to be detected is input into the trained target detection model, and the multi-modal target detection network infers the input to obtain the final target detection result, outputting the class, position and confidence of the target. The invention improves the target detection accuracy through the multi-modal target detection network and the contrast learning network. However, this method ignores the interaction of modal features.

[0008] The patent application file with publication number CN116452937A discloses a multi-modal target detection method based on dynamic convolution and attention mechanism. The method uses convolution operation to extract features of visible-infrared modalities; fuses channel and spatial attention to weight features of visible and infrared images; and applies a residual network to refine the weighted and fused features; it adds a multi-head attention mechanism to a new convolution structure to establish a new dynamic convolution module; pre-processes the visible and infrared light data, inputs them into the new network for training, optimizes the network parameters, obtains the optimal weights, and performs target detection on the input images. This method uses traditional attention mechanisms to form a modal fusion module for cross-modal fusion and learning. However, this method is limited by local feature interaction and produces a large amount of noise.

[0009] The patent application file with publication number CN117542020A discloses a multi-modal target detection method based on infrared images and visual images. The method uses a classic convolutional neural network to obtain multi-scale feature maps of visual images and infrared images and aligns them. A transformer model based on a local attention mechanism is used to sample local features from the feature maps and aggregate units are used to locally interact and aggregate the feature maps of the two modalities to obtain two aggregated features of visible light images and infrared images. Then, the two aggregated features are fused to obtain a fused feature, which is fed into a detection head for detection, and the final target information is obtained. This method uses different expressions and descriptions of targets by infrared images and visible light images to reduce the problem of missed detection and false detection of a single modality. However, the attention module is unidirectional, cannot simultaneously use its own modality and cross-modal features, and ignores the uniqueness of the modalities.

[0010] In summary, the above prior art has the above shortcomings and defects:

[0011] 1. Single modality environment is not adaptive. In the dark night and foggy weather conditions, the visible light image has less target information. In the good visible environment, the infrared image has less texture information and blurred edges, which cannot effectively distinguish foreground and background information. Using a single modality can easily cause missed detection and false detection problems.

[0012] 2. Most existing multi-modal target detection techniques are based on CNN networks, which are limited by limited receptive fields and can only capture local information, so they cannot use long-range dependencies within and between modalities.

[0013] 3. Existing multi-modal target detection techniques mostly use traditional attention mechanisms to improve target detection performance, which ignores the feature information enhancement between modalities.

[0014] 4. Existing technologies ignore bidirectional feature interaction between modalities in the fusion process. Considering bidirectional feature interaction, the inherent relationship of the self features is also ignored, resulting in unsatisfactory detection results. SUMMARY

[0015] To overcome the shortcomings of the above prior art, the purpose of the present application is to provide a target detection method, system, device and medium based on cross-modal fusion and guided attention mechanism. By cross-fusing infrared images and visible light images at the channel and spatial levels, and redesigning the guided attention mechanism in a bidirectional guiding combined with self-guiding manner, the infrared modality and visible light modality can realize feature interaction at the spatial and channel levels and simultaneous use of cross-modal features and self-modal features. The present application can reduce noise generated during the fusion of infrared images and visible light images, and has the technical effect of capturing long-distance cross-modal features and self-modal features.

[0016] To achieve the above object, the technical scheme adopted by the present application is:

[0017] A target detection method based on cross-modal fusion and guided attention mechanism, comprising the following steps:

[0018] Step one: obtain a visible-infrared image paired dataset, perform data processing on the visible-infrared image paired dataset and divide it to obtain a training set, a validation set and a test set;

[0019] Step two: build a multi-modal target detection network, including an input network Input, a backbone network Backbone, a neck network Neck and a detection head Head, wherein the backbone network Backbone includes a dual-branch CSPDarket53 network, four cross-modal feature enhancement modules (CFM) and four double-modal cross-guided attention modules (DMCT), and the neck network Neck includes a feature enhancement module (FEM);

[0020] Step three: set network training parameters;

[0021] Step four: according to the network training parameters set in step three, train and optimize the multi-modal target detection network built in step two using the training set obtained in step one, output the training weight file after training, and verify the weight file after training using the validation set obtained in step one, and select the weight file with the highest precision as the optimal weight file;

[0022] Step five: load the test set obtained in step one and the optimal weight file selected in step four into the multi-modal target detection network built in step two, perform target detection on the test set, and obtain the target detection result.

[0023] The implementation method of step one comprises:

[0024] Step 101: obtain a visible-infrared image paired dataset; use an infrared and visible light modal camera carried by a UAV to shoot target images in harsh environments, align the shot images and put them into an image set D;

[0025] The harsh environment includes but is not limited to night, poor lighting conditions, and foggy environment;

[0026] Step 102: label the images in the image set D obtained in step 101 with target class and position information to obtain an original image dataset O and an image label dataset L;

[0027] Step 103: divide the original image dataset O and the image label dataset L obtained in step 102 together to generate a training set, a validation set and a test set.

[0028] The implementation method of the second step comprises:

[0029] Step 201: input the training set generated in step 103 into the multi-modal target detection network;

[0030] Step 202: build a multi-modal target detection network, including an input network Input, a backbone network Backbone, a neck network Neck, and a detection head Head;

[0031] The input network Input uses the Mosaic algorithm to perform data enhancement processing on the training set input in step 201 to obtain an enhanced training set; and uses a clustering algorithm to optimize the anchor box of the enhanced training set to obtain an optimal anchor box size.

[0032] The backbone network Backbone is used for extraction, interaction, and fusion of infrared modality and visible light modality features. The backbone network Backbone includes a double-branch CSPDarket53 network, four cross-modal feature enhancement modules (CFM), and four double-modal cross-guided attention modules (DMCT). The double-branch CSPDarket53 network obtains infrared modality and visible light modality semantic feature maps with channel numbers of 64, 128, 256, 512, and 1024. The infrared modality and visible light modality features corresponding to the channel numbers of 64, 128, 256, and 512 are input into the four cross-modal feature enhancement modules (CFM) respectively, and the infrared modality and visible light modality features are cross-enhanced in space and channel. The cross-enhanced infrared modality and visible light modality features are input into the four double-modal cross-guided attention modules (DMCT) to realize long-distance feature interaction of the infrared modality and visible light modality features. The features processed by the double-modal cross-guided attention modules (DMCT) are added to the features extracted by the double-branch CSPDarket53 network to obtain fusion features with different channel numbers.

[0033] The neck network Neck fuses the fusion features with different channel numbers obtained by the backbone network Backbone through upsampling, a feature enhancement module (FEM), and a fully connected layer (Concat) to obtain a fused high-level semantic feature map.

[0034] The detection head Head is used for multi-scale target detection on the high-level semantic feature map fused by the neck network Neck, including an anchor box, a convolution layer, a prediction layer, and a non-maximum suppression, and outputs a prediction result.

[0035] The construction of the backbone network Backbone in step 202 specifically comprises:

[0036] Step 2021: constructing a double-branch CSPDarket53 network;

[0037] The double-branch CSPDarket53 network is used to extract multi-scale feature information of the infrared and visible light modalities, wherein the double-branch CSPDarket53 network includes four CBS modules and four C3 modules, and each of the CBS modules and the C3 modules is composed of a convolution layer Conv, an activation function, and a batch normalization operation, and has a sampling channel number of 64, 128, 256, 512, and 1024; a series of convolution operations are performed in the backbone network Backbone to extract infrared and visible light modality semantic feature maps with channel numbers of 64, 128, 256, 512, and 1024;

[0038] Step 2022: constructing a cross-modal feature enhancement module (CFM);

[0039] A plurality of 1x1 convolutions, a plurality of different activation functions (Sigmoid, Softmax), layer normalization (LayerNorm), and a plurality of global pooling (Global Pooling) are connected in series and in parallel, and the cross-modal features are enhanced through cross modulation in the spatial and channel levels; the infrared and visible light modality semantic feature maps with channel numbers of 64, 128, 256, and 512 extracted by step 2021 are input into the cross-modal feature enhancement module (CFM), the cross-modal feature enhancement module (CFM) uses an auxiliary modality to enhance the feature representation of another modality, and the expression of the cross-modal feature enhancement module (CFM) is:

[0040]

[0041]

[0042] wherein F i R , F i R is the input feature of the i-th (i = 1, 2, 3, 4) stage, represents a 1x1 convolution, σ1, σ2, σ3 represent a change tensor operation, F SG , F SM , F GP represent an activation function Sigmod, an activation function SoftMax, and a global pooling (Global Pooling) operation, respectively, and represents a tensor element-wise point multiplication;

[0043] The constructed cross-modal feature enhancement module (CFM) is added behind each C3 module of the double-branch CSPDarket53 network in step 2021;

[0044] Step 2023: constructing a dual-modal cross-guided attention module (DMCT);

[0045] The dual-modal cross-guided attention module (DMCT) is composed of cross-guided attention mechanisms and self-attention mechanisms. The feature map processed by the cross-modal feature enhancement module (CFM) constructed in step 2022 is input into the dual-modal cross-guided attention module (DMCT). The shared modal is composed by multiplying the infrared modal and the visible light modal features. The shared modal is used as the basic modal for guidance to guide the infrared modal and the visible light modal, realizing the deep information interaction of the two modalities. It is expressed as follows:

[0046]

[0047] The self-attention mechanism (Self-attention Layer Mechanisms) is expressed as:

[0048] Q = FW q K = FW k V = FW v

[0049]

[0050] The cross-guided attention mechanism (Guided-attention Layer Mechanisms) is expressed as:

[0051]

[0052] Wherein, Q, K, and V represent query vector, key vector, and value vector respectively, F represents infrared image feature map and visible light image feature map, W q W k W v represent trainable weight matrix, and B represents bias matrix.

[0053] The constructed dual-modal cross-guided attention module (DMCT) is connected behind the cross-modal feature enhancement module (CFM) constructed in step 2022. The features processed by the dual-modal cross-guided attention module (DMCT) are added to the features extracted by the dual-branch CSPDarket53 network, obtaining fusion features with different channel numbers.

[0054] The feature enhancement module (FEM) in the step 202 specifically comprises: channel attention, spatial attention, four convolution branches; the features processed by the double-modal cross-guided attention module (DMCT) are added to the features extracted by the double-branch CSPDarket53 network to obtain fusion features with different channel numbers, the infrared modal and visible light modal fusion features with channel numbers of 256, 512 and 1024 are input into the feature enhancement module (FEM) of the neck network Neck, the fusion features are enhanced in the channel and space through the channel attention (Channel attention) and the spatial attention (Spatial Attention), the enhanced features are input into the four convolution branches, and the output feature maps of the convolution are subjected to feature addition operation to obtain enhanced fusion features, the enhanced fusion features output by each stage are fused through the up-sampling, the full connection layer (Concat) and the regular convolution in the neck network Neck to obtain high-level semantic feature maps of the fusion features.

[0055] The four convolution branches comprise: two regular convolution series branches, three regular convolution and one hole convolution series branches, three regular convolution and one hole convolution branches, and one regular convolution branch.

[0056] The network training parameter setting in the step three is: the training round epoch is greater than or equal to 100, the batch size is greater than or equal to 16, the learning rate is less than or equal to 10 -5 , the loss threshold is less than or equal to 0.001, and the momentum is greater than or equal to 0.9 and less than or equal to 1.

[0057] The training parameter setting in the step four is: the batch size is greater than or equal to 16, the correlation coefficient conf-thres is less than or equal to 0.5, and the intersection over union coefficient iou-thres is less than or equal to 0.5.

[0058] The test parameter setting in the step five is: the batch size is greater than or equal to 24, the correlation coefficient conf-thres is less than or equal to 0.5, and the intersection over union coefficient iou-thres is less than or equal to 0.5.

[0059] The application further provides a target detection system based on a cross-modal fusion and guided attention mechanism, comprising:

[0060] A data set processing module is used for acquiring a visible light-infrared image paired data set, performing data processing on the visible light-infrared image paired data set, and dividing the visible light-infrared image paired data set to obtain a training set, a verification set and a test set.

[0061] The network construction module is used for constructing a multi-modal target detection network, including an input network Input, a backbone network Backbone, a neck network Neck and a detection head Head, wherein the backbone network Backbone includes a double-branch CSPDarket53 network, four cross-modal feature enhancement modules (CFM) and four double-modal cross-guided attention modules (DMCT), and the neck network Neck includes a feature enhancement module (FEM);

[0062] The network training module is used for training and optimizing the multi-modal target detection network according to the set network training parameters, outputting a training weight file after the training is completed, and verifying the training completed weight file by using a verification set, and selecting the weight file with the highest precision as the optimal weight file.

[0063] The target detection module is used for loading the test set and the optimal weight file into the multi-modal target detection network, performing target detection on the test set, and obtaining a target detection result.

[0064] The application further provides a target detection device based on the cross-modal fusion and guided attention mechanism.

[0065] The memory stores the computer program of the target detection method based on the cross-modal fusion and guided attention mechanism, and is a computer-readable device.

[0066] The processor is used for executing the computer program to realize the target detection method based on the cross-modal fusion and guided attention mechanism.

[0067] The application further provides a computer-readable storage medium, which stores a computer program, and the computer program can realize the target detection method based on the cross-modal fusion and guided attention mechanism when executed by a processor.

[0068] Compared with the prior art, the application has the following beneficial effects:

[0069] 1、The application expands the double-branch CSPDarket53 network, can fully extract visible light modal and infrared modal features, fully utilizes the complementary information provided by the visible light modal and the infrared modal, avoids the limitation of using a single sensor, reduces the problems of missed detection and false detection in target detection in dark night, poor lighting conditions, fog and other harsh environments, and improves the target detection capability.

[0070] 2、The application increases a cross-modal feature enhancement module (CFM) and a dual-modal cross-guided attention module (DMCT) on the basis of the dual-branch CSPDarket53 network. The mechanism of the two modules is bidirectional, that is, while the unique features of the visible light mode and the infrared mode are utilized, the feature interaction of the visible light mode and the infrared mode is also fully utilized. The application enhances the feature representation of the self mode through the features of the other mode while the respective mode features are utilized. This design solves the problem that the self mode and the cross-modal features cannot be utilized simultaneously in other methods, thereby improving the efficiency and accuracy of feature fusion.

[0071] 3、The application constructs the dual-modal cross-guided attention module (DMCT) by introducing the cross-attention mechanism and the self-attention mechanism and improving the cross-guided attention, avoids the local feature extraction limitation of using the conventional convolution, increases the long-range dependency, and reduces the noise generated due to the local feature extraction; meanwhile, the infrared mode and the visible light mode features are multiplied to form a shared mode, the shared mode is used as a basic guide mode, the long-range interaction of the infrared mode and the visible light mode information is enhanced, the information of the two modes is fully utilized, and the problem that a large amount of noise is generated due to the local feature interaction limitation in most methods is solved.

[0072] 4、The application fuses and enhances the information of the infrared mode and the visible light mode by adding the feature enhancement module (FEM) composed of the conventional convolution and the hollow convolution before the original detection head Head, reduces the noise generated in the mode fusion process, and improves the target recognition accuracy.

[0073] In summary, the application first builds a dual-branch CSPDarket53 network to realize feature extraction of infrared images and visible light images, then adds a cross-modal feature enhancement module (CFM) in the network, improves the semantic feature representation of the two modes through cross modulation of the modes, in order to avoid the limitation of local feature extraction of the conventional convolution in the fusion process, a dual-modal cross-guided attention module (DMCT) based on transform is proposed, long-distance feature interaction of the two modes is realized, feature interaction ability of the two modes is improved, and the internal connection of the module is enhanced through the addition of the self-attention mechanism, in order to avoid the noise generated due to the introduction of the above fusion module, the application proposes a feature enhancement module (FEM), improves the target recognition accuracy, and has the advantages of high target recognition accuracy and high algorithm robustness. BRIEF DESCRIPTION OF DRAWINGS

[0074] Figure 1 is a principle flow schematic diagram of an embodiment of the application.

[0075] Figure 2 is a structure diagram of a multi-modal target detection network of the application.

[0076] Figure 3 is a cross-modal feature enhancement module (CFM) structural diagram of the present application.

[0077] Figure 4 is a dual-modal cross-guided attention module (DMCT) structural diagram of the present application; wherein, Figure 4(a) is a structural connection diagram of the dual-modal cross-guided attention module (DMCT), Figure 4(b) is a network structure diagram of the cross-guided attention mechanism (Guided-attention Layer Mechanisms), and Figure 4(c) is a network structure diagram of two self-attention mechanisms (Self-attention Layer Mechanisms).

[0078] Figure 5 is a feature enhancement module (FEM) structural diagram of the present application.

[0079] Figure 6 is a simulation diagram of the present application; wherein, Figure 6 (a) is an original infrared image and a visible light image containing a target to be detected, Figure 6 (b) is an image after target recognition by the present application, Figure 6 (a) is an image after target recognition by the present application, Figure 6 (c) is an original infrared image and a visible light image containing a target to be detected, Figure 6 (d) is an image after target recognition by the present application, Figure 6 (c) is an image after target recognition by the present application. DETAILED DESCRIPTION

[0080] The technical solutions of the present application will be further described in detail below in combination with the drawings.

[0081] The present application first expands the branch structure of YOLO v5, extracts the features of infrared images and optical images respectively by using a dual-branch CSPDarket53 network; then introduces a cross-modal feature enhancement module (CFM), which can use the information between modalities to cross-feature enhance the feature maps at the spatial and channel levels; then in order to improve the information interaction ability between the two modalities, a dual-modal cross-guided attention module (DMCT) is proposed, which can solve the limitation of local feature extraction of traditional convolution, capture the long-range dependency relationship of the two modalities, and in order to enhance the internal connection of the two modal interaction and fusion, a self-attention mechanism is added in the module. Finally, in order to reduce the noise brought by the fusion process, a feature enhancement module (FEM) is introduced, and the enhanced result is sent to the detection head Head to improve the robustness and accuracy of target detection. The simulation results in the test environment show that the present application overcomes the shortcomings of the prior art and achieves excellent detection results.

[0082] As Figure 1As shown, a target detection method based on cross-modal fusion and guided attention mechanism includes the following steps:

[0083] Step one: obtain visible-infrared image paired dataset;

[0084] The infrared image and visible light image dataset are preprocessed to obtain the aligned visible-infrared image paired dataset, and are divided into training set, validation set and test set;

[0085] Step 101: dataset acquisition and alignment; using a visual sensor, obtain the infrared image and visible light image dataset, find the characteristic points of the infrared image and visible light image with uniqueness and stability, find and calculate the corresponding relationship of the characteristic points in the two images, eliminate the images that do not match the infrared image and visible image, obtain the final aligned image, and form the visible-infrared image paired dataset D from the aligned image; the visible-infrared image paired dataset D includes the aligned infrared image and visible light image dataset O={O1, O2, ……O n} and infrared image and visible light image label dataset L={L1, L2, ……L n}, n>5000, D={O, L}, wherein n represents the nth infrared image and visible light image, the size of each infrared image and visible light image in O is 512*512;

[0086] Step 102: dataset division; the visible-infrared image paired dataset D obtained in step 101 is divided into training set D tr ={O tr ,L tr}, validation set D v ={O v ,L v} and test set D te ={O te ,L te} in the proportion of 6:2:2;

[0087] Step two: build a multi-modal target detection network;

[0088] Step 201: use the training set D tr ={O tr ,L tr} generated in step 102 as the input of the multi-modal target detection network;

[0089] Step 202: build a multi-modal target detection network, including input network Input, backbone network Backbone, neck network Neck and detection head Head;

[0090] The input network Input performs a data enhancement operation on the training set D input in step 201 using the Mosaic algorithm to obtain an enhanced training set tr = {O tr , L tr} The clustering algorithm is used to optimize the anchor frame in the enhanced training set to obtain the optimal anchor frame size; data enhancement and anchor frame optimization improve the robustness of the network;

[0091] The backbone network Backbone is used for extraction, interaction and fusion of infrared and visible light modal features, and the backbone network Backbone includes a double-branch CSPDarket53 network, four cross-modal feature enhancement modules (CFM) and four double-modal cross-guided attention modules (DMCT). The double-branch CSPDarket53 network obtains infrared and visible light modal semantic feature maps with channel numbers of 64, 128, 256, 512 and 1024, and the infrared and visible light modal features with channel numbers of 64, 128, 256 and 512 are input into the four cross-modal feature enhancement modules (CFM). The cross-modal feature enhancement module (CFM) combines the attention mechanism characteristics to cross-enhance the infrared and visible light modal features in space and channel to avoid information loss. The cross-enhanced infrared and visible light modal features are input into the four double-modal cross-guided attention modules (DMCT) to realize long-distance feature interaction of the infrared and visible light modal features. The features processed by the double-modal cross-guided attention module (DMCT) are added to the features extracted by the double-branch CSPDarket53 network to obtain fusion features with different channel numbers.

[0092] Step 2021: constructing a double-branch CSPDarket53 network;

[0093] The double-branch CSPDarket53 network is used to extract multi-scale feature information of infrared and visible light modalities, wherein the double-branch CSPDarket53 network includes four CBS modules and four C3 modules, and each of the CBS module and the C3 module is composed of a convolution layer Conv, an activation function and a batch normalization operation, and the sampling channel number is 64, 128, 256, 512 and 1024. A series of convolution operations are performed in the backbone network Backbone to extract infrared and visible light modal semantic feature maps with channel numbers of 64, 128, 256, 512 and 1024.

[0094] Step 2022: constructing a cross-modal feature enhancement module (CFM); as Figure 3 ​As shown, several 1x1 convolutions, several different activation functions (Sigmoid, Softmax), layer normalization (LayerNorm), several global pooling (GlobalPooling) and element-wise point multiplication operation and multiplication operation are used to form the channel branch and the spatial branch, and the channel branch and the spatial branch are connected in series to form the cross-modal feature enhancement module (CFM). For the channel branch, the present application uses 1x1 convolution to convert the input infrared image features and visible light image features into query vectors and numerical vectors, then performs element-wise point multiplication on the query vectors of the infrared image features and the visible light image features, adds an activation function Softmax in the back and respectively performs matrix multiplication with the numerical vectors of the infrared image features and the visible light image features, then connects 1x1 convolution, layer normalization (LayerNorm), activation function Sigmoid in the back and performs element-wise point multiplication with the infrared image features and the visible light image features, then cross adds with the infrared image features and the visible light image features, and outputs the channel branch result; for the spatial branch, the present application uses 1x1 convolution to convert the result of the channel branch into query vectors and numerical vectors, performs global pooling (GlobalPooling) operation on the query vectors, then performs element-wise point multiplication on the query vectors of the results of the infrared image features and the visible light image features, connects 1x1 convolution, layer normalization (LayerNorm), activation function (Sigmoid) in the back and performs matrix multiplication with the numerical vectors of the channel branch result, then connects 1x1 convolution and layer normalization (LayerNorm) in the back, and performs element-wise point multiplication with the result of the channel branch, then cross adds with the result of the channel branch, and outputs the cross-modal feature enhancement module (CFM) result; the expression of the cross-modal feature enhancement module (CFM) is as follows:

[0095]

[0096]

[0097] Wherein F i R , F i R is the input feature of the i-th (i=1, 2, 3, 4) stage, represents 1x1 convolution, σ1, σ2, σ3 represents the change tensor operation, F SG , F SM , F GP respectively represent the activation function Sigmod, the activation function SoftMax, the global pooling (Global Pooling) operation, and · represents the element-wise point multiplication of the tensor.

[0098] Step 2023: constructing a dual-modal cross-guided attention module (DMCT); as shown in FIG. 4(a), the dual-modal cross-guided attention module (DMCT) is composed of a cross-guided attention mechanism and a self-attention mechanism, and the shared modality is composed by multiplying the infrared modality and the visible light modality features, the shared modality is used as the basic modality of guidance, and the infrared modality and the visible light modality are guided to realize the deep information interaction of the two modalities; represented as follows:

[0099]

[0100] As shown in FIG. 4(c), the self-attention mechanism (Self-attention Layer Mechanisms) is represented as:

[0101] Q = FW q K = FW k V = FW v

[0102]

[0103] As shown in FIG. 4(b), the cross-guided attention mechanism (Guided-attention Layer Mechanisms) is represented as:

[0104]

[0105]

[0106] Wherein, Q, K, and V represent query vectors, key vectors, and value vectors, respectively, F represents infrared image feature maps and visible light image feature maps, W q W k W v represent trainable weight matrices, and B represents a bias matrix.

[0107] The neck network Neck fuses the fusion features of different channel numbers obtained by the backbone network Backbone through upsampling, a feature enhancement module (FEM), and a full connection layer (Concat), extracts effective structural information of the network, and obtains a fused high-level semantic feature map; wherein the feature enhancement module (FEM) includes a plurality of traditional convolutions and hole convolutions with different convolution kernels.

[0108] As Figure 5As shown, the structure of the feature enhancement module (FEM) includes channel attention, spatial attention, four convolution branches, including two regular convolution series branches, three regular convolution and one hole convolution series branches, three regular convolution and one hole convolution branches, and one regular convolution branch; the features processed by the double-modal cross-guided attention module (DMCT) are added to the features extracted by the double-branch CSPDarket53 network to obtain fusion features with different channel numbers; the infrared modal and visible light modal fusion features with channel numbers of 256, 512 and 1024 are input into the feature enhancement module (FEM) of the neck network Neck, the fusion features are enhanced in the channel and space through channel attention (Channel attention) and spatial attention (Spatial Attention), the enhanced features are input into the four convolution branches, and the output feature maps of the convolution are subjected to feature addition operation to obtain enhanced fusion features; the enhanced fusion features output by each stage are fused through upsampling, full connection layer (Concat) and regular convolution in the neck network Neck to obtain high-level semantic feature maps of the fusion features. The generation of noise in the fusion process of the traditional method is reduced.

[0109] The detection head Head includes an anchor frame, a convolution layer, a prediction layer and a non-maximum suppression, and is responsible for multi-scale target detection on the high-level semantic feature maps fused by the neck network Neck, and outputs a prediction result.

[0110] The detection head is a small target detection head with three different scales, which is used for detecting targets with different scales, and improves the model generalization ability and target detection and recognition accuracy.

[0111] Step three: set the network training parameters;

[0112] Set the training round epoch to be greater than or equal to 100, the batch size batch_size to be greater than or equal to 16, the learning rate to be less than or equal to 10 -5 , the loss threshold to be less than or equal to 0.001, the correlation coefficient conf-thres to be less than or equal to 0.5, the intersection over union coefficient iou-thres to be less than or equal to 0.5, and the momentum to be 0.937.

[0113] Step four: according to the network training parameters set in step three, using the enhanced training set obtained in step 202 and the optimal anchor frame size, the multi-modal target detection network constructed in step two is trained and optimized, and 100 rounds of training are designed, and after every five rounds of training, a training weight file pt i is obtained, wherein i represents the number of training rounds; and the verification set Dv v v i

[0114] Step five: input the test set D te te te and the optimal weight file pt obtained in step four into the multi-modal target detection network constructed in step two, and perform target recognition on the test set D te te te , to obtain target recognition results, and output the category of the target, the detection frame with the target position, and the recall accuracy.

[0115] Set the network test parameters: set the batch size batch_size≥16, the correlation coefficient conf-thres≤0.5, the intersection-over-union coefficient iou-thres≤0.5, and the momentum=0.937.

[0116] As shown in Figure 2 , the multi-modal target detection network of the application is composed of four parts. The first part is an input network Input, which includes Mosaic data enhancement and clustering algorithm optimization anchor box, and obtains a picture with a size of 512×512. The second part is a backbone network Backbone, which uses a double-branch CSPDarknet53 network. Each branch passes through a CBS layer and a C3 layer to extract features of infrared images and visible light images, so as to obtain semantic features of the infrared images and the visible light images. A cross-modal feature enhancement module (CFM) and a double-modal cross-guided attention module (DMCT) are used to fuse the features of the visible light images and the infrared images, to realize cross-modulation and cross-feature enhancement of the two modalities at the channel and spatial levels, to realize information interaction of the two modalities at a deeper level, and to improve the detection accuracy of the target. The third part is a neck Neck, which uses a multi-scale detection method. A feature enhancement module (FEM) is added to the shallow feature part of the network to fuse and enhance three pairs of infrared image and visible light image feature maps in the feature extraction network, to reduce the loss caused by noise generated in the fusion process, and to improve the detection accuracy of the target. The fourth part is a detection head Head, which mainly includes an anchor box, a convolution layer, a prediction layer, and a non-maximum suppression, and is responsible for multi-scale target detection of the fused and enhanced features of the neck Neck, to improve the model generalization ability and the target detection and recognition accuracy.

[0117] The effect of the application will be further described in combination with a simulation experiment:

[0118] 1. Simulation experiment conditions​​​​​​​​

[0119] The hardware platform of the simulation experiment of the present invention is: the processor is Intel i7-8750H, and the GPU is NUVIDA GeForceRTX 4060.

[0120] The software platform for the simulation experiment of the present invention is: Ubuntu20.04 operating system and PyCharm 2022, PyTorch1.12.0, and CUDA11.2.

[0121] 2. Simulation steps

[0122] The training set D of the visible light-infrared image pairing dataset tr ={O tr ,L tr} and validation set D v ={O v ,L v} is input into the multimodal target detection network model for optimization training. The training process is: Mosaic algorithm is used to train the training set D tr ={O tr ,L tr} Perform data enhancement and use clustering algorithm to enhance the training set Clustering is performed to obtain the optimal size of the anchor frame; the input image is subjected to feature extraction through a dual-branch CSPDarknet53 network to obtain feature maps of infrared images and visible light images at different scales, and then these feature maps are cross-modulated and fused at the spatial and channel levels, interacting with deep information and enhancing the fused features to obtain the most appropriate fused feature map, and then these feature maps are classified and regressed, and the regression results are subjected to feature reconstruction operations to obtain more refined feature maps, on this basis, classification and regression operations are performed again, and the loss is calculated to complete the multimodal target detection based on the present invention.

[0123] 3. Simulation content and results analysis

[0124] The simulation experiment of the present invention is to perform target recognition processing on four pairs of infrared images and visible light images, and the results are as follows: Figure 6 shown.

[0125] The following combination Figure 6 Table 1 further describes the simulation results of the present invention.

[0126] Figure 6 (a) is the original infrared image and visible light image containing the target to be detected, Figure 6 (b) For the use of the present invention Figure 6 (a) Image after target recognition, Figure 6(c) is the original infrared image and visible light image containing the target to be detected, Figure 6 (d) is the detection efficiency of the existing method using the present application Figure 6 (c) the image after target recognition.

[0127] As Figure 6 (a) and Figure 6 (c) show, due to factors such as overexposure, low light, and element overlap, it is difficult to capture color and texture information of the target to be detected, while the thermal infrared modality can still provide foreground information about pedestrians, vehicle movement, and vehicle position. As Figure 6 (b) and Figure 6 (d) show that the present application can clearly and accurately mark the target position and size range in the infrared image and the visible light image in the form of a rectangular box, proving the effectiveness of the multi-modal target detection network proposed by the present application. In summary, the experimental results show that the target detection method based on cross-modal fusion and guided attention mechanism proposed by the present application can accurately identify the target to be detected in the infrared image and the visible light image.

[0128] Tables 1, 2, and 3 are comparison tables of target detection efficiency of the present application and existing methods, and the test data set includes three public data sets VEDAI, FLIR, and LLVIP.

[0129] Table 1 is a comparison table of detection efficiency of the present application and existing methods for VEDAI public data set.

[0130]

[0131] Table 2 is a comparison table of detection efficiency of the present application and existing methods for FLIR public data set.

[0132]

[0133] Table 3 is a comparison table of detection efficiency of the present application and existing methods for LLVIP public data set.

[0134]

[0135]

[0136] As shown in Table 1, Table 2, Table 3, it can be seen that, compared with Faster RCNN, EfficientDet and YOLOv5, the application makes full use of the target information of the visible light mode and the infrared mode, and realizes excellent accuracy. During the test process, the application uses mAP(%)、mAP50(%) and mAP75(%) as evaluation indexes to evaluate the model. mAP(%) reflects the average detection accuracy of the model under different IOU(intersection over union) threshold values, mAP50(%) represents the average accuracy when the IOU threshold value is 0.5, and mAP75(%) represents the average accuracy when the IOU threshold value is 0.5; during the test process using the VEDAI dataset, compared with the existing method using a single mode, the method of the application improves the mAP(%) by 14.7% and 14.7% respectively compared with the existing method YOLOV5 using only visible light images (RGB) and only infrared images (Thermal). During the test process using the FLIR dataset, the method of the application improves the mAP(%) by 11.6% and 4.1% respectively compared with the existing method YOLOV5 using only visible light images (RGB) and only infrared images (Thermal). During the test process using the LLVIP dataset, the method of the application improves the mAP(%) by 15.4% and 3.5% respectively compared with the existing method YOLOV5 using only visible light images (RGB) and only infrared images (Thermal). At the same time, compared with Faster R-CNN using only visible light images (RGB) and only infrared images (Thermal), the evaluation indexes mAP(%)、mAP50(%) and mAP75(%) of the method of the application are all improved. Therefore, by introducing the cross-modal feature enhancement module (CFM), the dual-modal cross-guided attention module (DMCT) and the feature enhancement module (FEM), the application makes full use of the complementary information provided by the visible light mode and the infrared mode, and improves the target detection accuracy under lighting changes and adverse weather conditions.

[0137] The simulation results show that the target detection method based on cross-modal fusion and guided attention mechanism can realize target detection requirements under lighting changes and adverse weather conditions.

[0138] The application also provides a target detection system based on cross-modal fusion and guided attention mechanism, comprising:

[0139] A dataset processing module is used to realize the acquisition of the visible light-infrared image paired dataset in step one, and the visible light-infrared image paired dataset is processed and divided to obtain a training set, a verification set and a test set.

[0140] a network construction module for implementing the construction of the multi-modal target detection network in step two, including an input network Input, a backbone network Backbone, a neck network Neck and a detection head Head, wherein the backbone network Backbone includes a double-branch CSPDarket53 network, four cross-modal feature enhancement modules (CFM) and four double-modal cross-guided attention modules (DMCT), and the neck network Neck includes a feature enhancement module (FEM);

[0141] a network training module for implementing the training and optimization of the multi-modal target detection network constructed in step two according to the network training parameters set in step three by using the training set obtained in step one, outputting the training weight file after the training is completed, and verifying the weight file after the training is completed by using the verification set obtained in step one, and selecting the weight file with the highest precision as the optimal weight file;

[0142] a target detection module for implementing the target detection of the test set by loading the test set obtained in step one and the optimal weight file selected in step four into the multi-modal target detection network constructed in step two, and obtaining the target detection result.

[0143] The application further provides a target detection device based on the cross-modal fusion and guided attention mechanism, which comprises:

[0144] a memory for storing the computer program of the target detection method based on the cross-modal fusion and guided attention mechanism, and being readable by a computer;

[0145] a processor for implementing the target detection method based on the cross-modal fusion and guided attention mechanism when the computer program is executed.

[0146] The application further provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program can implement the target detection method based on the cross-modal fusion and guided attention mechanism when the computer program is executed by a processor.

Claims

1. A target detection method based on cross-modal fusion and guided attention mechanism, characterized in that, The method comprises the following steps: Step 1: obtaining a visible-infrared image paired dataset, performing data processing on the visible-infrared image paired dataset, and dividing the visible-infrared image paired dataset to obtain a training set, a verification set and a test set; Step 2: constructing a multi-modal target detection network, including an input network Input, a backbone network Backbone, a neck network Neck and a detection head Head, wherein the backbone network Backbone comprises a double-branch CSPDarket53 network, four cross-modal feature enhancement modules (CFM) and four double-modal cross-guided attention modules (DMCT), and the neck network Neck comprises a feature enhancement module (FEM); The implementation method of step 2 comprises: Step 201: taking the training set generated in step 103 as the input of the multi-modal target detection network; Step 202: constructing a multi-modal target detection network, including an input network Input, a backbone network Backbone, a neck network Neck and a detection head Head; The input network Input uses a Mosaic algorithm to perform data enhancement processing on the training set input in step 201 to obtain an enhanced training set; and uses a clustering algorithm to optimize the anchor box of the enhanced training set to obtain an optimal anchor box size; The backbone network Backbone is used for extraction, interaction and fusion of infrared modal and visible light modal features, and comprises a double-branch CSPDarket53 network, four cross-modal feature enhancement modules (CFM) and four double-modal cross-guided attention modules (DMCT). The double-branch CSPDarket53 network obtains infrared modal and visible light modal semantic feature maps with channel numbers of 64, 128, 256, 512 and 1024, and the infrared modal and visible light modal features corresponding to the channel numbers of 64, 128, 256 and 512 are input into the four cross-modal feature enhancement modules (CFM) respectively, the infrared modal and visible light modal features are cross-enhanced in space and channel, and the cross-enhanced infrared modal and visible light modal features are input into the four double-modal cross-guided attention modules (DMCT) to realize long-distance feature interaction of the infrared modal and visible light modal features. The features processed by the double-modal cross-guided attention module (DMCT) are added to the features extracted by the double-branch CSPDarket53 network to obtain fusion features with different channel numbers; The neck network Neck fuses the fusion features with different channel numbers obtained by the backbone network Backbone through upsampling, a feature enhancement module (FEM) and a fully connected layer (Concat) to obtain a fused high-level semantic feature map; The detection head Head is used for multi-scale target detection on the high-level semantic feature map fused by the neck network Neck, and comprises an anchor box, a convolution layer, a prediction layer and a non-maximum suppression, and outputs a prediction result; The construction of the backbone network Backbone in step 202 comprises: Step 2021: constructing a double-branch CSPDarket53 network; The dual-branch CSPDarket53 network is used to extract multiscale feature information of the infrared modality and the visible light modality, wherein the dual-branch CSPDarket53 network comprises four CBS modules and four C3 modules, the CBS module and the C3 module are both composed of a convolution layer Conv, an activation function and a batch normalization operation, and the sampling channel numbers are 64, 128, 256, 512 and 1024; a series of convolution operations are performed in the backbone network to extract infrared modality and visible light modality semantic feature maps with channel numbers of 64, 128, 256, 512 and 1024; Step 2022: constructing a cross-modal feature enhancement module (CFM); A plurality of 1x1 convolutions, a plurality of different activation functions (Sigmoid, Softmax), layer normalization (LayerNorm) and a plurality of global pooling (Global Pooling) are connected in series and parallel, and the dual-modality features are enhanced through cross modulation in the spatial and channel levels; the infrared modality and the visible light modality semantic feature maps with channel numbers of 64, 128, 256 and 512 extracted by step 2021 are input into the cross-modal feature enhancement module (CFM), the cross-modal feature enhancement module (CFM) enhances the feature representation of the other modality by using the auxiliary modality, and the expression of the cross-modal feature enhancement module (CFM) is: wherein F i R , F i R is the input feature of the i-th (i = 1, 2, 3, 4) stage, W q- , W v- represents a 1x1 convolution, σ1, σ2, σ3 represent a change tensor operation, F SG , F SM , F GP respectively represent an activation function Sigmod, an activation function SoftMax, a global pooling (Global Pooling) operation, and represent a tensor element-wise point multiplication; The constructed cross-modal feature enhancement module (CFM) is added behind each C3 module of the dual-branch CSPDarket53 network in step 2021; Step 2023: constructing a dual-modality cross-guided attention module (DMCT); The dual-modality cross-guided attention module (DMCT) is composed of cross-guided attention mechanisms and self-attention mechanisms (Self-attention Layer Mechanisms), the feature map processed by the cross-modal feature enhancement module (CFM) constructed by step 2022 is input into the dual-modality cross-guided attention module (DMCT), the shared modality is composed by multiplying the infrared modality and the visible light modality, the shared modality is used as the basic modality of guidance, and the infrared modality and the visible light modality are guided to realize the deep information interaction of the two modalities; represented as follows: Wherein, the self-attention mechanism (Self-attention Layer Mechanisms) is represented as: Q = FW q K = FW k V = FW v The cross-guided attention mechanism (Guided-attention Layer Mechanisms) is represented as: Wherein, Q, K, V represent query vector, key vector and value vector respectively, F represents infrared image feature map and visible light image feature map, W q ,W k ,W v represents a trainable weight matrix, and B represents a bias matrix. The constructed dual-modality cross-guided attention module (DMCT) is connected behind the cross-modal feature enhancement module (CFM) constructed by step 2022, the features processed by the dual-modality cross-guided attention module (DMCT) are added to the features extracted by the dual-branch CSPDarket53 network to obtain fusion features with different channel numbers; The feature enhancement module (FEM) in the step 202 specifically comprises: channel attention, spatial attention, four convolution branches; the features processed by the dual-modal cross-guided attention module (DMCT) are added to the features extracted by the dual-branch CSPDarket53 network to obtain fusion features with different channel numbers, the infrared modal and visible light modal fusion features with channel numbers of 256, 512 and 1024 are input into the feature enhancement module (FEM) of the neck network Neck, the fusion features are enhanced in the channel and space by the channel attention (Channel attention) and the spatial attention (Spatial Attention), the enhanced features are input into the four convolution branches, and the output feature maps of the convolution are subjected to feature addition operation to obtain enhanced fusion features, the enhanced fusion features output by each stage are fused by upsampling, full connection layer (Concat) and regular convolution in the neck network Neck to obtain high-level semantic feature maps of the fusion features; The four convolution branches comprise: two regular convolution series branches, three regular convolution and one hole convolution series branches, three regular convolution and one hole convolution branches, and one regular convolution branch; Step three: setting network training parameters; Step four: according to the network training parameters set in step three, the training set obtained in step one is used to train and optimize the multi-modal target detection network constructed in step two, the training weight file is output after the training is completed, and the verification set obtained in step one is used to verify the weight file after the training is completed, and the weight file with the highest precision is selected as the optimal weight file; Step five: the test set obtained in step one and the optimal weight file selected in step four are loaded into the multi-modal target detection network constructed in step two, the test set is subjected to target detection, and the target detection result is obtained.

2. The target detection method based on cross-modal fusion and guided attention mechanism according to claim 1, characterized in that, The implementation method of the step one comprises: Step 101: obtaining a visible light-infrared image pairing data set; using an infrared and visible light modal camera carried by a drone to shoot target images in a harsh environment, aligning the shot images and putting them into an image set D; Step 102: labeling the images in the image set D obtained in step 101 with target categories and position information to obtain an original image data set O and an image label data set L; Step 103: dividing the original image data set O and the image label data set L obtained in step 102 together to generate a training set, a verification set and a test set. 3.The target detection method based on cross-modal fusion and guided attention mechanism according to claim 1, characterized in that, The network training parameter setting in the third step: the practice round epoch is greater than or equal to 100, the batch size is greater than or equal to 16, the learning rate is less than or equal to 10 -5 , the loss threshold is less than or equal to 0.001, and the momentum is greater than or equal to 0.9 and less than or equal to 1.

4. The target detection method based on cross-modal fusion and guided attention mechanism according to claim 1, characterized in that, The training parameter setting in the step four: batch size batch_size≥16, correlation coefficient conf-thres≤0.5, and intersection over union coefficient iou-thres≤0.5; The test parameter setting in the step five: setting batch size batch_size≥24, correlation coefficient conf-thres≤0.5, and intersection over union coefficient iou-thres≤0.

5.

5. A target detection system based on cross-modal fusion and attention-guided mechanism according to any one of claims 1 to 4, characterized in that, It comprises: The data set processing module is configured to obtain a visible-infrared image paired data set, perform data processing on the visible-infrared image paired data set, and divide the visible-infrared image paired data set to obtain a training set, a verification set, and a test set; The network construction module is configured to construct a multi-modal target detection network, including an input network Input, a backbone network Backbone, a neck network Neck, and a detection head Head, wherein the backbone network Backbone includes a dual-branch CSPDarket53 network, four cross-modal feature enhancement modules (CFM), and four double-modal cross-guided attention modules (DMCT), and the neck network Neck includes a feature enhancement module (FEM); The network training module is configured to train and optimize the multi-modal target detection network using the training set according to the set network training parameters, output a training weight file after the training is completed, and verify the training completed weight file using the verification set, and select the weight file with the highest precision as the optimal weight file; The target detection module is configured to load the test set and the optimal weight file into the multi-modal target detection network, perform target detection on the test set, and obtain a target detection result.

6. A target detection device based on cross-modal fusion and guided attention mechanism, characterized in that, The computer program for the target detection method based on cross-modal fusion and guided attention mechanism according to any one of claims 1-4 is stored in the memory, and the computer program is readable by the computer; The processor is configured to execute the computer program to implement the target detection method based on cross-modal fusion and guided attention mechanism according to any one of claims 1-4. The computer program is stored in the computer readable storage medium, and the computer program is executed by the processor to implement the target detection method based on cross-modal fusion and guided attention mechanism according to any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, ​

Images