Target detection method, system and device based on cross-modal fusion and attention guiding mechanism, and medium

Abstract

A target detection method, system and device based on cross-modal fusion and an attention guiding mechanism and a medium, the target detection method comprises the following steps: obtaining a visible light-infrared image pairing data set, and carrying out data processing and division on the visible light-infrared image pairing data set to obtain a training set, a verification set and a test set; constructing a multi-modal target detection network; setting network training parameters; training and optimizing the multi-modal target detection network by using the training set, outputting a training weight file after the training is finished, verifying the trained weight file by using the verification set, and selecting the weight file with the highest precision as an optimal weight file; loading the test set and the optimal weight file into a multi-modal target detection network, and performing target detection on the test set to obtain a target detection result; the system, the device and the medium are used for bearing and implementing the method. The method has lower false detection and false detection conditions, and improves the robustness and accuracy of target detection.

Description

Technical Field

[0001] The present invention belongs to the technical field of target detection, and in particular relates to a target detection method, system, device and medium based on cross-modal fusion and guided attention mechanism. Background Art

[0002] In recent years, object detection algorithms have developed rapidly and are widely used in various practical applications such as autonomous driving, drones, and robots. They process scene images captured by visual sensors to identify and locate mission targets and regions of interest. Visual object detection based on a single visible modality (RGB) has achieved great success. Most existing intelligent agents (such as drones, robots, etc.) carry detectors based on single-mode sensors to complete the task. Unfortunately, in real-world scenarios, there are many challenges such as lighting changes, adverse weather conditions, and geographical changes. These detectors are very limited in harsh environments (rain, fog, low light). Due to the impact of challenging environmental conditions such as harsh weather conditions and changing lighting, a single sensor is limited in capturing environmental information.

[0003] Compared with single modality, multi-modality provides richer target visual information, and they have complementary interactions. Under good lighting conditions, visible light sensors perform well in capturing target information such as color and texture details, but under poor lighting conditions, it is difficult to distinguish target details through visible light modality, while infrared sensors can still provide foreground information about enemies, vehicle motion and other factors due to their unique imaging mechanism. Therefore, visible light images and infrared images are unique and complementary. Inspired by this, in order to achieve more reliable and robust target detection under lighting changes and adverse weather conditions, multispectral imaging and fusion technologies have been introduced, which have attracted increasing attention recently.

[0004] Object detection has always been a very critical technology in computer vision tasks. Object detection algorithms based on deep learning can be divided into two categories: candidate region-based and regression-based. Candidate-based object detection algorithm regions include FastR-CNN, Mask R-CNN, and Cascade R-CNN, all of which use R-CNN as the basic framework. Regression-based object detection algorithms, such as the YOLO series and the SSD series, have also developed infrared-based object detection algorithms, which are detection algorithms developed from visible objects. Existing multimodal object detection algorithms can be divided into traditional methods and deep learning methods, such as weight fusion, aggregated channel features, and the use of feature pyramids to fuse features at each layer.

[0005] The intelligent agent carries sensors, uses target detection algorithms, optimizes end-to-end models, finds the location of the target of interest from the captured pictures or videos, and realizes continuous and real-time tracking and detection of the mission target. The challenges of existing target detection algorithms include: 1) In actual scenes, infrared images have the problems of insufficient color information, weak energy and low resolution. In some special environments, such as night and overexposure, it is difficult for visible light images to provide the outline and detail information of the target. It is difficult to adapt to harsh environments and obtain satisfactory detection results by using only one modality for object detection. 2) Most fusion methods usually fuse and enhance feature representation by using information within each modality or weight fusion strategies to achieve multimodal fusion, ignoring the relationship between different modalities. 3) Most existing methods are based on CNN networks, which are limited by limited receptive fields and can only capture local information, so they cannot utilize long-range dependencies within and between modalities. Visible light-infrared sensor fusion can effectively capture more comprehensive target information and obtain more stable detection results by combining data from different spectra, thereby improving the all-weather perception ability of intelligent agents.

[0006] Several multimodal object detection methods have been proposed.

[0007] For example, the patent application document with publication number CN118154844A discloses a multimodal target detection method based on contrastive learning. The method first obtains training data, including a training sample set and a test sample set; (2) constructs a multimodal target detection model based on contrastive learning, designs the loss function of the model, and iteratively trains the processed data. (3) The image to be detected is used as the input of the trained target detection model, and the multimodal target detection network infers the input to obtain the final target detection result, and outputs the target category, location and confidence. This invention improves the target detection accuracy through a multimodal target detection network and a contrastive learning network. However, this method ignores the interaction between modal features.

[0008] The patent application document with the publication number CN116452937A discloses a multimodal target detection method based on dynamic convolution and attention mechanism. The method uses convolution operation to extract the features of visible light-infrared mode; fuses channels and spatial attention to weight the features of visible light and infrared images; and applies residual network to refine the weighted fused features; it adds multi-head attention mechanism to the new convolution structure and establishes a new dynamic convolution module; pre-processes the visible light and infrared light data, inputs them into the new network for training, optimizes the network parameters, obtains the optimal weights, and performs target detection tasks on the input image. This method uses the traditional attention mechanism to form a modal fusion module for cross-modal fusion and learning. However, this method is limited by the interaction of local features and generates a lot of noise.

[0009] The patent application document with publication number CN117542020A discloses a multimodal target detection method based on infrared images and visual images. The method uses a classic convolutional neural network to obtain and align multi-scale feature maps of visual images and infrared images; uses a transform model based on a local attention mechanism to perform local feature sampling on the feature map and uses an aggregation unit to perform local interactive aggregation on the feature maps of the two modalities to obtain two aggregate features of visible light images and infrared images; then the two aggregate features are feature fused to obtain fused features, and the fused features are sent to the detection head for detection, and finally the target information is obtained. This method utilizes the different expressions and descriptions of the target by infrared images and visible light images, reducing the problems of missed detection and false detection in a single modality. However, its attention module is unidirectionally guided, and it is impossible to use its own modality and cross-modal features at the same time, ignoring the uniqueness of the modality.

[0010] In summary, the above existing technologies currently have the following shortcomings and defects:

[0011] 1. The single modality has poor environmental adaptability. Visible light images have little target information in dark nights and foggy weather conditions. Infrared images have little target texture information and blurred edges in good visual environments, making it impossible to effectively distinguish foreground and background information. Using a single modality is prone to missed detection and false detection problems.

[0012] 2. Most of the existing multimodal object detection technology methods are based on CNN networks. Due to the limitation of limited receptive field, they can only capture local information and therefore cannot utilize long-range dependencies within and between modalities.

[0013] 3. Existing multimodal target detection technologies mostly use traditional attention mechanisms to improve target detection performance, but they ignore the problem of enhancing feature information between modalities.

[0014] 4. The existing technology ignores the bidirectional feature interaction between modalities during the fusion process. While considering the bidirectional feature interaction, it also ignores the intrinsic connection of its own features, resulting in unsatisfactory detection results. Summary of the invention

[0015] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide a target detection method, system, device and medium based on cross-modal fusion and guided attention mechanism, which can realize the feature interaction of infrared modality and visible light modality at both spatial and channel levels and the synchronous utilization of cross-modal features and self-modal features by cross-fusing infrared images and visible light images at both channel and spatial levels, and redesigning the guided attention mechanism in a two-way guidance combined with self-guidance manner; the present invention can reduce the noise generated in the process of fusion of infrared images and visible light images, and has the technical effect of capturing cross-modal features and self-modal features over a long distance.

[0016] In order to achieve the above object, the technical solution adopted by the present invention is:

[0017] A target detection method based on cross-modal fusion and guided attention mechanism, comprising the following steps:

[0018] Step 1: Obtain a visible light-infrared image pairing dataset, process and divide the visible light-infrared image pairing dataset to obtain a training set, a validation set, and a test set;

[0019] Step 2: Construct a multimodal target detection network, including the input network Input, the backbone network Backbone, the neck network Neck and the detection head Head. The backbone network Backbone includes a dual-branch CSPDarket53 network, four cross-modal feature enhancement modules (CFMs), and four bimodal cross-guided attention modules (DMCTs). The neck network Neck includes a feature enhancement module (FEM).

[0020] Step 3: Set network training parameters;

[0021] Step 4: According to the network training parameters set in step 3, the multimodal target detection network constructed in step 2 is trained and optimized using the training set obtained in step 1. After the training, the training weight file is output, and the trained weight file is verified using the verification set obtained in step 1. The weight file with the highest accuracy is selected as the optimal weight file.

[0022] Step 5: Load the test set obtained in step 1 and the optimal weight file selected in step 4 into the multimodal target detection network constructed in step 2, perform target detection on the test set, and obtain the target detection result.

[0023] The implementation method of step 1 includes:

[0024] Step 101: Obtain a visible light-infrared image pairing dataset; use infrared and visible light modality cameras carried by a drone to capture target images in a harsh environment, align the captured images, and put them into an image set D;

[0025] The adverse environment includes but is not limited to nighttime, poor lighting conditions, and foggy environment;

[0026] Step 102: annotate the images in the image set D obtained in step 101 with target categories and location information to obtain an original image dataset O and an image label dataset L;

[0027] Step 103: The original image dataset O and the image label dataset L obtained in step 102 are partitioned together to generate a training set, a validation set, and a test set.

[0028] The implementation method of step 2 includes:

[0029] Step 201: using the training set generated in step 103 as input to a multimodal object detection network;

[0030] Step 202: construct a multimodal target detection network, including an input network Input, a backbone network Backbone, a neck network Neck and a detection head Head;

[0031] The input network Input uses the Mosaic algorithm to perform data enhancement processing on the training set input in step 201 to obtain an enhanced training set; uses a clustering algorithm to optimize the anchor frame of the enhanced training set to obtain an optimal anchor frame size;

[0032] The backbone network Backbone is used for the extraction, interaction and fusion of infrared modality and visible light modality features. The backbone network Backbone includes a dual-branch CSPDarket53 network, four cross-modal feature enhancement modules (CFMs) and four dual-modal cross-guided attention modules (DMCTs). The dual-branch CSPDarket53 network obtains infrared modality and visible light modality semantic feature maps with channel numbers of 64, 128, 256, 512 and 1024. The infrared modality corresponding to the channel numbers of 64, 128, 256, 512 is converted into visible light modality semantic feature maps. The infrared modality and visible light modality features are respectively input into four cross-modal feature enhancement modules (CFM), and the infrared modality and visible light modality features are cross-enhanced in space and channels. The cross-enhanced infrared modality and visible light modality features are input into four dual-modal cross-guided attention modules (DMCT) to achieve long-distance feature interaction of infrared modality and visible light modality features. Among them, the features processed by the dual-modal cross-guided attention module (DMCT) are added to the features extracted by the dual-branch CSPDarket53 network to obtain fusion features with different numbers of channels.

[0033] The neck network Neck fuses the fusion features of different channel numbers obtained by the backbone network Backbone through upsampling, feature enhancement module (FEM) and full connection layer (Concat) to obtain a fused high-level semantic feature map;

[0034] The detection head Head is used to perform multi-scale target detection on the high-level semantic feature map after fusion by the neck network Neck, including anchor boxes, convolution layers, prediction layers and non-maximum suppression, and output prediction results.

[0035] The construction of the backbone network Backbone in step 202 specifically includes:

[0036] Step 2021: Build a dual-branch CSPDarket53 network;

[0037] The dual-branch CSPDarket53 network is used to extract multi-scale feature information of infrared modality and visible light modality. The dual-branch CSPDarket53 network includes four CBS modules and four C3 modules. Both the CBS module and the C3 module are composed of convolutional layers Conv, activation functions and batch normalization operations, and the number of sampling channels is 64, 128, 256, 512 and 1024. A series of convolution operations are performed in the backbone network Backbone to extract semantic feature maps of infrared modality and visible light modality with channel numbers of 64, 128, 256, 512 and 1024.

[0038] Step 2022: construct a cross-modal feature enhancement module (CFM);

[0039] A number of 1×1 convolutions, a number of different activation functions (Sigmoid, Softmax), layer normalization (LayerNorm), and a number of global pooling (Global Pooling) are connected in series and parallel, and the dual-modal features are enhanced by cross-modulation at the spatial and channel levels; the infrared modality and visible light modality semantic feature maps with 64, 128, 256, and 512 channels extracted by step 2021 are input into a cross-modal feature enhancement module (CFM). The cross-modal feature enhancement module (CFM) uses the auxiliary modality to enhance the feature representation of another modality. The expression of the cross-modal feature enhancement module (CFM) is:

[0040]

[0041]

[0042] where F i R , F i R is the input feature of the i-th (i=1,2,3,4) stage, represents a 1×1 convolution, σ1, σ2, σ3 represent the change tensor operation, and F SG , F SM , F GP They represent the activation function Sigmod, the activation function SoftMax, and the global pooling operation, respectively, and represent the element-by-element dot product of the tensor;

[0043] Add the constructed cross-modal feature enhancement module (CFM) to the back of each C3 module of the dual-branch CSPDarket53 network in step 2021;

[0044] Step 2023: construct a bimodal cross-guided attention module (DMCT);

[0045] The dual-modal cross-guided attention module (DMCT) is composed of a cross-guided attention mechanism and a self-attention mechanism. The feature map processed by the cross-modal feature enhancement module (CFM) constructed in step 2022 is input into the dual-modal cross-guided attention module (DMCT). The infrared modality and the visible light modality features are multiplied to form a shared modality. The shared modality is used as the basic modality for guidance to guide the infrared modality and the visible light modality to achieve deep information interaction between the two modalities. It is expressed as the following formula:

[0046]

[0047] Among them, the self-attention layer mechanism is expressed as:

[0048] Q=FW q ,K=FW k ,V=FW v

[0049]

[0050] The cross-guided attention mechanism (Guided-attention Layer Mechanisms) is expressed as:

[0051]

[0052] Among them, Q, K, V represent the query vector, key vector and value vector respectively, F represents the infrared image feature map and the visible light image feature map, W q ,W k ,W v represents the trainable weight matrix, and B represents the bias matrix;

[0053] Connect the constructed bimodal cross-guided attention module (DMCT) to the cross-modal feature enhancement module (CFM) constructed in step 2022, add the features processed by the bimodal cross-guided attention module (DMCT) to the features extracted by the dual-branch CSPDarket53 network, and obtain fused features with different numbers of channels.

[0054] The feature enhancement module (FEM) in step 202 specifically includes: channel attention, spatial attention, and four convolution branches; the features processed by the dual-modal cross-guided attention module (DMCT) are added to the features extracted by the dual-branch CSPDarket53 network to obtain fused features with different channel numbers, and the infrared mode and visible light mode fusion features with channel numbers of 256, 512 and 1024 are input into the feature enhancement module (FEM) of the neck network Neck, and the fused features are enhanced in channels and space through channel attention and spatial attention, and the enhanced features are input into four convolution branches, and the output feature map of the convolution is subjected to feature addition operation to obtain the enhanced fused features, and the enhanced fused features outputted at each stage are fused through upsampling, a fully connected layer (Concat), and conventional convolution in the neck network Neck to obtain a high-level semantic feature map of the fused features;

[0055] The four convolution branches include: two conventional convolution series branches, three conventional convolutions and one dilated convolution series branch, three conventional convolutions and one dilated convolution branch, and one conventional convolution branch.

[0056] The network training parameter settings in step 3 are: training round epoch ≥ 100, batch size batch_size ≥ 16, learning rate ≤ 10 -5 , loss threshold ≤ 0.001, 0.9 ≤ momentum ≤ 1.

[0057] The training parameter settings in step 4 are: batch size batch_size ≥ 16, correlation coefficient conf-thres ≤ 0.5, intersection-over-union coefficient iou-thres ≤ 0.5.

[0058] The test parameter settings in step 5 are as follows: batch size batch_size ≥ 24, correlation coefficient conf-thres ≤ 0.5, and intersection-over-union coefficient iou-thres ≤ 0.5.

[0059] The present invention also provides a target detection system based on cross-modal fusion and guided attention mechanism, comprising:

[0060] A data set processing module is used to obtain a visible light-infrared image pairing data set, process and divide the visible light-infrared image pairing data set to obtain a training set, a validation set, and a test set;

[0061] A network construction module is used to construct a multimodal target detection network, including an input network Input, a backbone network Backbone, a neck network Neck and a detection head Head. The backbone network Backbone includes a dual-branch CSPDarket53 network, four cross-modal feature enhancement modules (CFMs), and four bimodal cross-guided attention modules (DMCTs). The neck network Neck includes a feature enhancement module (FEM).

[0062] The network training module is used to train and optimize the multimodal target detection network using the training set according to the set network training parameters. After the training, the training weight file is output, and the trained weight file is verified using the verification set, and the weight file with the highest accuracy is selected as the optimal weight file.

[0063] The target detection module is used to load the test set and the optimal weight file into the multimodal target detection network, perform target detection on the test set, and obtain the target detection result.

[0064] The present invention also provides a target detection device based on cross-modal fusion and guided attention mechanism, comprising:

[0065] Memory: a computer program storing the above-mentioned target detection method based on cross-modal fusion and guided attention mechanism, which is a computer-readable device;

[0066] Processor: used to implement the target detection method based on cross-modal fusion and guided attention mechanism when executing the computer program.

[0067] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it can implement the target detection method based on cross-modal fusion and guided attention mechanism.

[0068] Compared with the prior art, the present invention has the following beneficial effects:

[0069] 1. The present invention expands the dual-branch CSPDarket53 network, which can fully extract the features of visible light modality and infrared modality, make full use of the complementary information provided by visible light modality and infrared modality, and avoid the limitations of using a single sensor. It reduces the problems of missed detection and false detection when detecting targets in harsh environments such as night, poor lighting conditions, and fog, and improves the target detection capability.

[0070] 2. The present invention adds a cross-modal feature enhancement module (CFM) and a dual-modal cross-guided attention module (DMCT) on the basis of the dual-branch CSPDarket53 network. The action mechanism of these two modules is bidirectional, that is, while utilizing the unique features of the visible light modality and the infrared modality, it also makes full use of the feature interaction between the visible light modality and the infrared modality. While utilizing the features of each modality, the present invention also enhances the feature representation of its own modality through the features of another modality. This design solves the problem that other methods cannot simultaneously utilize the features of their own modality and cross-modality, thereby improving the efficiency and accuracy of feature fusion.

[0071] 3. The present invention introduces the cross-attention mechanism and the self-attention mechanism and improves the cross-guided attention to construct a dual-modal cross-guided attention module (DMCT) module, thereby avoiding the local feature extraction limitations of conventional convolution, increasing long-range dependencies, and reducing the noise generated by local feature extraction; at the same time, the infrared modality and visible light modality features are multiplied to form a shared modality, and the shared modality is used as the basic guiding modality to enhance the long-range interaction of information between the infrared modality and the visible light modality, making full use of the information of the two modalities, and solving the problem that most methods generate a large amount of noise due to the limitation of local feature interaction.

[0072] 4. The present invention adds a feature enhancement module (FEM) composed of conventional convolution and hole convolution in front of the original detection head to fuse and enhance the information of infrared mode and visible light mode, reduce the noise generated in the modal fusion process, and improve the recognition accuracy of the target.

[0073] In summary, the present invention first builds a dual-branch CSPDarket53 network to realize feature extraction of infrared images and visible light images; then a cross-modal feature enhancement module (CFM) is added to the network to improve the semantic feature representation of the two modalities through cross-modulation of the modalities. In order to avoid the limitations of traditional convolutional local feature extraction in the fusion process, a transform-based dual-modal cross-guided attention module (DMCT) is proposed to realize long-distance feature interaction of the two modalities, improve the feature interaction ability of the two modalities, and enhance the internal connection of the module by adding a self-attention mechanism. In order to avoid the noise generated by the introduction of the above-mentioned fusion module, the present invention proposes a feature enhancement module (FEM) to improve the recognition accuracy of the target, which has the advantages of high target recognition accuracy and high algorithm robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 It is a schematic diagram of the principle flow of an embodiment of the present invention.

[0075] Figure 2 It is a structural diagram of the multimodal target detection network of the present invention.

[0076] Figure 3 It is a structural diagram of the cross-modal feature enhancement module (CFM) of the present invention.

[0077] Figure 4 is a structural diagram of the dual-modal cross-guided attention module (DMCT) of the present invention; wherein, Figure 4(a) is a structural connection diagram of the dual-modal cross-guided attention module (DMCT), Figure 4(b) is a network structure diagram of the cross-guided attention mechanism (Guided-attention Layer Mechanisms), and Figure 4(c) is a network structure diagram of two self-attention mechanisms (Self-attention Layer Mechanisms).

[0078] Figure 5 It is a structural diagram of the feature enhancement module (FEM) of the present invention.

[0079] Figure 6 is a simulation diagram of the present invention; wherein, Figure 6 (a) is the original infrared image and visible light image containing the target to be detected. Figure 6 (b) In order to adopt the present invention Figure 6 (a) Image after target recognition, Figure 6 (c) is the original infrared image and visible light image containing the target to be detected, Figure 6 (d) In order to adopt the present invention Figure 6 (c) Image after target recognition. DETAILED DESCRIPTION

[0080] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings.

[0081] The present invention firstly expands the branch structure of YOLO v5, and uses the dual-branch CSPDarket53 network to extract the features of infrared images and optical images respectively; then introduces the cross-modal feature enhancement module (CFM), which can use the information between the modalities to perform cross-feature enhancement on the feature map at the spatial and channel levels; then, in order to improve the information interaction ability between the two modalities, it is proposed to use the dual-modal cross-guided attention module (DMCT), which can solve the local feature extraction limitations of traditional convolution and capture the long-range dependency relationship between the two modalities. In order to enhance the internal connection of the interactive fusion of the two modalities, a self-attention mechanism is added to the module. Finally, in order to reduce the noise caused by the fusion process, the feature enhancement module (FEM) is introduced, and the enhanced results are sent to the detection head to improve the robustness and accuracy of target detection. The simulation results in the test environment show that the present invention overcomes the shortcomings of the prior art and achieves excellent detection results.

[0082] like Figure 1As shown, a target detection method based on cross-modal fusion and guided attention mechanism includes the following steps:

[0083] Step 1: Obtain a visible light-infrared image pairing dataset;

[0084] Preprocess the infrared image and visible light image datasets to obtain an aligned visible light-infrared image pairing dataset, and divide it into a training set, a validation set, and a test set;

[0085] Step 101: Dataset acquisition and alignment: Use a visual sensor to acquire infrared image and visible light image datasets, find out the unique and stable feature points of the infrared image and the visible light image, find and calculate the correspondence between the feature points in the two images, remove the images that do not match the infrared image and the visible image, obtain the final aligned images, and form the aligned images into a visible light-infrared image paired dataset D; the visible light-infrared image paired dataset D includes the aligned infrared image and visible light image dataset O = {O1, O2, ... O n} and infrared image and visible light image label dataset L = {L1, L2, ... L n}, n>5000, D={O,L}, where n represents the nth infrared image and visible light image, and the size of each infrared image and visible light image in O is 512×512;

[0086] Step 102: Dataset division: The visible light-infrared image pairing dataset D obtained in step 101 is divided into training set D according to the ratio of 6:2:2. tr = {O tr ,L tr}、Validation set D v = {O v ,L v} and test set D te = {O te ,L te};

[0087] Step 2: Build a multimodal target detection network;

[0088] Step 201: The training set D generated in step 102 tr = {O tr ,L tr} as input to the multimodal target detection network;

[0089] Step 202: construct a multimodal target detection network, including an input network Input, a backbone network Backbone, a neck network Neck and a detection head Head;

[0090] The input network Input uses the Mosaic algorithm to process the training set D input in step 201. tr = {O tr ,L tr}Perform data enhancement operations to obtain enhanced training sets Use clustering algorithm to enhance the training set In The anchor box is optimized to obtain the optimal anchor box size; the robustness of the network is improved through data enhancement and anchor box optimization;

[0091] The backbone network Backbone is used for the extraction, interaction and fusion of infrared modality and visible light modality features. The backbone network Backbone includes a dual-branch CSPDarket53 network, four cross-modal feature enhancement modules (CFMs) and four dual-modal cross-guided attention modules (DMCTs). The infrared modality and visible light modality semantic feature maps with channel numbers of 64, 128, 256, 512 and 1024 are obtained through the dual-branch CSPDarket53 network, and the infrared modality and visible light modality features corresponding to the channel numbers of 64, 128, 256, and 512 are respectively input into the four A cross-modal feature enhancement module (CFM) is built. The cross-modal feature enhancement module (CFM) combines the characteristics of the attention mechanism to cross-enhance the infrared modality and visible light modality features in space and channels to avoid information loss. The cross-enhanced infrared modality and visible light modality features are input into four dual-modal cross-guided attention modules (DMCT) to achieve long-distance feature interaction between infrared modality and visible light modality features. Among them, the features processed by the dual-modal cross-guided attention module (DMCT) are added to the features extracted by the dual-branch CSPDarket53 network to obtain fused features with different numbers of channels.

[0092] Step 2021: Build a dual-branch CSPDarket53 network;

[0093] The dual-branch CSPDarket53 network is used to extract multi-scale feature information of infrared modality and visible light modality. The dual-branch CSPDarket53 network includes four CBS modules and four C3 modules. Both the CBS module and the C3 module are composed of convolutional layers Conv, activation functions and batch normalization operations, and the number of sampling channels is 64, 128, 256, 512 and 1024. A series of convolution operations are performed in the backbone network Backbone to extract semantic feature maps of infrared modality and visible light modality with channel numbers of 64, 128, 256, 512 and 1024.

[0094] Step 2022: construct a cross-modal feature enhancement module (CFM); Figure 3As shown in the figure, several 1×1 convolutions, several different activation functions (Sigmoid, Softmax), layer normalization (LayerNorm), several global pooling (GlobalPooling) and element-by-element point multiplication operations and multiplication operations are combined to form a channel branch and a space branch. The channel branch and the space branch are connected in series to form a cross-modal feature enhancement module (CFM). For the channel branch, the present invention uses 1x1 convolution to convert the input infrared image features and visible light image features into a query vector and a numerical vector, and then performs element-by-element point multiplication on the query vectors of the infrared image features and the visible light image features, and then adds the activation function Softmax to the back and performs matrix multiplication with the numerical vectors of the infrared image features and the visible light image features respectively, and then connects to 1x1 convolution, layer normalization (LayerNorm), activation function Sigmoid and combines with the infrared image features and visible light image features. The visible light image features are element-wise multiplied, and then cross-added with the infrared image features and the visible light image features, and the channel branch results are output; for the spatial branch, the present invention uses 1x1 convolution to convert the results of the channel branch into a query vector and a numerical vector, performs a global pooling operation on the query vector, and then performs element-wise dot multiplication on the query vector of the infrared image features and the visible light image features of the result, followed by 1x1 convolution, layer normalization (LayerNorm), activation function (Sigmoid) and matrix multiplication with the numerical vector of the channel branch result, followed by 1x1 convolution and layer normalization (LayerNorm), and element-wise dot multiplication with the results of the channel branch, and then cross-added with the results of the channel branch, and output the cross-modal feature enhancement module (CFM) result; the expression of the cross-modal feature enhancement module (CFM) is as follows:

[0095]

[0096]

[0097] where F i R , F i R is the input feature of the i-th (i=1,2,3,4) stage, represents a 1×1 convolution, σ1, σ2, σ3 represent the change tensor operation, and F SG , F SM , F GP They represent the activation function Sigmod, the activation function SoftMax, and the global pooling operation respectively, and · represents the element-by-element dot product of the tensor.

[0098] Step 2023: construct a dual-modal cross-guided attention module (DMCT); as shown in FIG4(a), the dual-modal cross-guided attention module (DMCT) consists of a cross-guided attention mechanism and a self-attention mechanism. The shared modality is formed by multiplying the infrared modality and the visible light modality features. The shared modality is used as the basic modality for guidance to guide the infrared modality and the visible light modality to achieve deep information interaction between the two modalities; it is expressed as the following formula:

[0099]

[0100] As shown in Figure 4(c), the self-attention layer mechanism is expressed as:

[0101] Q=FW q ,K=FW k ,V=FW v

[0102]

[0103] As shown in Figure 4(b), the cross-guided attention mechanism (Guided-attention Layer Mechanisms) is expressed as:

[0104]

[0105]

[0106] Among them, Q, K, V represent the query vector, key vector and value vector respectively, F represents the infrared image feature map and the visible light image feature map, W q ,W k ,W v represents the trainable weight matrix, and B represents the bias matrix;

[0107] The neck network Neck fuses the fused features of different numbers of channels obtained by the backbone network Backbone through upsampling, feature enhancement module (FEM) and fully connected layer (Concat), extracts the effective structural information of the network, and obtains the fused high-level semantic feature map; wherein the feature enhancement module (FEM) includes several traditional convolutions and void convolutions with different convolution kernels.

[0108] like Figure 5As shown in the figure, the structure of the feature enhancement module (FEM) includes channel attention (Channelattention), spatial attention (Spatial Attention), and four convolution branches, and the four convolution branches include: two conventional convolution series branches, three conventional convolutions and one hole convolution series branches, three conventional convolutions and one hole convolution branch, and one conventional convolution branch; the features processed by the dual-modal cross-guided attention module (DMCT) are added to the features extracted by the dual-branch CSPDarket53 network to obtain fusion features with different channel numbers, and the infrared mode and visible light mode fusion features with channel numbers of 256, 512 and 1024 are input into the feature enhancement module (FEM) of the neck network Neck, and the fusion features are obtained through channel attention (Channel Attention and spatial attention enhance features in channels and spaces, input the enhanced features into four convolution branches, and perform feature addition operation on the output feature maps of the convolution to obtain enhanced fusion features. The enhanced fusion features outputted at each stage are fused through upsampling, full connection layer (Concat), and conventional convolution in the neck network Neck to obtain high-level semantic feature maps of the fusion features. This reduces the noise generated during the fusion process of traditional methods.

[0109] The detection head, including an anchor frame, a convolution layer, a prediction layer and non-maximum suppression, is responsible for performing multi-scale target detection on the high-level semantic feature map fused by the neck network Neck and outputting the prediction results.

[0110] The detection heads are small target detection heads of three different scales, which are used to detect targets of different scales and sizes, thereby improving the generalization ability of the model and the accuracy of target detection and recognition.

[0111] Step 3: Set network training parameters;

[0112] Set the training round epoch ≥ 100, batch size batch_size ≥ 16, and learning rate ≤ 10 -5 , loss threshold ≤ 0.001, correlation coefficient conf-thres ≤ 0.5, intersection-over-union coefficient iou-thres ≤ 0.5, momentum = 0.937.

[0113] Step 4: Based on the network training parameters set in step 3, use the enhanced training set obtained in step 202 The multimodal target detection network constructed in step 2 is trained and optimized with the optimal anchor box size. 100 rounds of training are designed. After every five rounds of training, a training weight file pt is obtained. i , where i represents the number of training rounds; the validation set D obtained in step 102v = {O v ,L v}For training weight file pt i Perform verification and select the weight file with the highest verification accuracy as the optimal weight file pt;

[0114] Step 5: The test set D obtained in step 102 te = {O te ,L te} and the optimal weight file pt obtained in step 4 are input into the multimodal target detection network constructed in step 2. te = {O te ,L te}Perform target recognition, obtain target recognition results, and output target category, detection box with target location, recall rate, accuracy and other data.

[0115] Set network test parameters: set batch size batch_size ≥ 16, correlation coefficient conf-thres ≤ 0.5, intersection-over-union coefficient iou-thres ≤ 0.5, momentum = 0.937.

[0116] like Figure 2 As shown in the figure, the multimodal target detection network of the present invention consists of four parts. The first part is the input network Input, including Mosaic data enhancement and clustering algorithm to optimize the anchor frame to obtain a 512×512 size picture; the second part is the backbone network Backbone, which uses a dual-branch CSPDarknet53 network. Each branch extracts features of infrared images and visible light images through the CBS layer and the C3 layer, thereby obtaining the semantic features of infrared images and visible light images, and uses the cross-modal feature enhancement module (CFM) and the dual-modal cross-guided attention module (DMCT) to fuse the features of visible light images and infrared images, realizing two-way convergence at the channel and space levels. The third part is the neck. It adopts the multi-scale detection method, adds a feature enhancement module (FEM) to the shallow feature part of the network, and fuses and enhances the three pairs of infrared image and visible light image feature maps in the feature extraction network to reduce the loss caused by the noise generated by the fusion process and improve the detection accuracy of the target. The fourth part is the detection head, which mainly includes anchor frame, convolution layer, prediction layer and non-maximum suppression. It is responsible for multi-scale target detection of the fused and enhanced features of the neck, so as to improve the generalization ability of the model and the target detection and recognition accuracy.

[0117] The effect of the present invention is further described below in conjunction with simulation experiments:

[0118] 1. Simulation Experiment Conditions

[0119] The hardware platform of the simulation experiment of the present invention is: the processor is Intel i7-8750H, and the GPU is NUVIDA GeForceRTX 4060.

[0120] The software platform of the simulation experiment of the present invention is: Ubuntu20.04 operating system and PyCharm 2022, PyTorch1.12.0, and CUDA11.2.

[0121] 2. Simulation steps

[0122] The training set D of the visible light-infrared image pairing dataset tr = {O tr ,L tr} and validation set D v = {O v ,L v} is input into the multimodal target detection network model for optimization training. The training process is: Mosaic algorithm is used to train the training set D tr = {O tr ,L tr} Perform data enhancement and use clustering algorithm to enhance the training set Clustering is performed to obtain the optimal size of the anchor frame; the input image will be subjected to feature extraction through a dual-branch CSPDarknet53 network to obtain feature maps of infrared images and visible light images at different scales, and then these feature maps are cross-modulated and fused at the spatial and channel levels, interact with deep information and enhance the fused features to obtain the most suitable fused feature map, and then these feature maps are classified and regressed, and the regression results are subjected to feature reconstruction operations to obtain more refined feature maps, on this basis, classification and regression operations are performed again, and the loss is calculated to complete the multimodal target detection based on the present invention.

[0123] 3. Simulation content and results analysis

[0124] The simulation experiment of the present invention uses four pairs of infrared images and visible light images for target recognition processing, and the results are as follows: Figure 6 shown.

[0125] Combine the following Figure 6 And Table 1 further describe the simulation effect of the present invention.

[0126] Figure 6 (a) is the original infrared image and visible light image containing the target to be detected. Figure 6 (b) In order to adopt the present invention Figure 6 (a) Image after target recognition, Figure 6(c) is the original infrared image and visible light image containing the target to be detected, Figure 6 (d) In order to adopt the present invention Figure 6 (c) Image after target recognition.

[0127] like Figure 6 (a) and Figure 6 As shown in (c), due to factors such as overexposure, low light, and feature overlap, it is difficult to capture the color and texture information of the target to be detected, while the thermal infrared modality can still provide foreground information about pedestrians, vehicle movement, and vehicle position. Figure 6 (b) and Figure 6 As shown in (d), it can be seen that the present invention can clearly and accurately mark the task target position and size range in the infrared image and the visible light image in the form of a rectangular box, proving the effectiveness of the multimodal target detection network proposed in the present invention. In summary, the experimental results show that the target detection method based on cross-modal fusion and guided attention mechanism proposed in the present invention can accurately identify the target to be detected in infrared images and visible light images.

[0128] Table 1, Table 2, and Table 3 are comparison tables of target detection efficiency between the present invention and the existing methods. The test data sets include three public data sets: VEDAI, FLIR, and LLVIP.

[0129] Table 1 is a comparison table of the efficiency of the present invention and the existing methods in detecting the VEDAI public dataset.

[0130]

[0131] Table 2 is a comparison table of the efficiency of the present invention and the existing method in detecting the FLIR public data set.

[0132]

[0133] Table 3 is a comparison table of the efficiency of the present invention and the existing methods in detecting the LLVIP public dataset.

[0134]

[0135]

[0136] As shown in Table 1, Table 2, and Table 3, it can be seen that, compared with Faster RCNN, EfficientDet, and YOLOv5, the present invention makes full use of the target information of the visible light modality and the infrared modality to achieve excellent accuracy. During the test, the present invention uses mAP (%), mAP50 (%), and mAP75 (%) as evaluation indicators to evaluate the quality of the model. mAP (%) reflects the average detection accuracy of the model under different IOU (intersection over union) thresholds, mAP50 (%) represents the average accuracy when the IOU threshold is 0.5, and mAP75 (%) represents the average accuracy when the IOU threshold is 0.5; in the process of testing using the VEDAI dataset, compared with the existing method using a single modality, the method of the present invention is 14.7% and 14.7% higher than the existing method YOLOV5 when only using visible light images (RGB) and only using infrared images (Thermal), respectively. In the process of testing with FLIR dataset, the mAP(%) of the method of the present invention is improved by 11.6% and 4.1% respectively when compared with the existing method YOLOV5 using only visible light images (RGB) and only infrared images (Thermal). In the process of testing with LLVIP dataset, the mAP(%) of the method of the present invention is improved by 15.4% and 3.5% respectively when compared with the existing method YOLOV5 using only visible light images (RGB) and only infrared images (Thermal). At the same time, compared with Faster R-CNN using only visible light images (RGB) and only infrared images (Thermal), the evaluation indicators mAP(%), mAP50(%) and mAP75(%) of the method of the present invention are all improved. Therefore, the present invention makes full use of the complementary information provided by the visible light modality and the infrared modality by introducing a cross-modal feature enhancement module (CFM), a dual-modal cross-guided attention module (DMCT) and a feature enhancement module (FEM), thereby improving the target detection accuracy under lighting changes and adverse weather conditions.

[0137] Simulation results show that the target detection method based on cross-modal fusion and guided attention mechanism proposed in this invention can achieve target detection requirements under lighting changes and adverse weather conditions.

[0138] The present invention also provides a target detection system based on cross-modal fusion and guided attention mechanism, comprising:

[0139] A data set processing module is used to obtain a visible light-infrared image pairing data set in step 1, process and divide the visible light-infrared image pairing data set to obtain a training set, a validation set, and a test set;

[0140] A network construction module, used to implement the construction of a multimodal target detection network in step 2, including an input network Input, a backbone network Backbone, a neck network Neck and a detection head Head, wherein the backbone network Backbone includes a dual-branch CSPDarket53 network, four cross-modal feature enhancement modules (CFMs), and four bimodal cross-guided attention modules (DMCTs), and the neck network Neck includes a feature enhancement module (FEM);

[0141] The network training module is used to implement the network training parameters set in step 3 in step 4, train and optimize the multimodal target detection network constructed in step 2 using the training set obtained in step 1, output the training weight file after the training is completed, and verify the trained weight file using the verification set obtained in step 1, and select the weight file with the highest accuracy as the optimal weight file;

[0142] The target detection module is used to implement step 5 in which the test set obtained in step 1 and the optimal weight file selected in step 4 are loaded into the multimodal target detection network constructed in step 2, perform target detection on the test set, and obtain the target detection result.

[0143] The present invention also provides a target detection device based on cross-modal fusion and guided attention mechanism, comprising:

[0144] Memory: a computer program storing the above-mentioned target detection method based on cross-modal fusion and guided attention mechanism, which is a computer-readable device;

[0145] Processor: used to implement the target detection method based on cross-modal fusion and guided attention mechanism when executing the computer program.

[0146] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it can implement the target detection method based on cross-modal fusion and guided attention mechanism.

Claims

1. A target detection method based on cross-modal fusion and guided attention mechanism, characterized in that: The following steps are involved: Step 1: Obtain a visible light-infrared image pairing dataset, process and divide the visible light-infrared image pairing dataset to obtain a training set, a validation set, and a test set; Step 2: Construct a multimodal target detection network, including the input network Input, the backbone network Backbone, the neck network Neck and the detection head Head. The backbone network Backbone includes a dual-branch CSPDarket53 network, four cross-modal feature enhancement modules (CFMs), and four bimodal cross-guided attention modules (DMCTs). The neck network Neck includes a feature enhancement module (FEM). Step 3: Set network training parameters; Step 4: According to the network training parameters set in step 3, the multimodal target detection network constructed in step 2 is trained and optimized using the training set obtained in step 1. After the training, the training weight file is output, and the trained weight file is verified using the verification set obtained in step 1. The weight file with the highest accuracy is selected as the optimal weight file. Step 5: Load the test set obtained in step 1 and the optimal weight file selected in step 4 into the multimodal target detection network constructed in step 2, perform target detection on the test set, and obtain the target detection result.

2. According to claim 1, a target detection method based on cross-modal fusion and guided attention mechanism is characterized in that: The implementation method of step 1 includes: Step 101: Obtain a visible light-infrared image pairing dataset; use infrared and visible light modality cameras carried by a drone to capture target images in a harsh environment, align the captured images, and put them into an image set D; The adverse environment includes but is not limited to nighttime, poor lighting conditions, and foggy environment; Step 102: annotate the images in the image set D obtained in step 101 with target categories and location information to obtain an original image dataset O and an image label dataset L; Step 103: The original image dataset O and the image label dataset L obtained in step 102 are partitioned together to generate a training set, a validation set, and a test set.

3. The target detection method based on cross-modal fusion and guided attention mechanism according to claim 1, characterized in that: The implementation method of step 2 includes: Step 201: using the training set generated in step 103 as input to a multimodal object detection network; Step 202: construct a multimodal target detection network, including an input network Input, a backbone network Backbone, a neck network Neck and a detection head Head; The input network Input uses the Mosaic algorithm to perform data enhancement processing on the training set input in step 201 to obtain an enhanced training set; uses a clustering algorithm to optimize the anchor frame of the enhanced training set to obtain an optimal anchor frame size; The backbone network Backbone is used for the extraction, interaction and fusion of infrared modality and visible light modality features. The backbone network Backbone includes a dual-branch CSPDarket53 network, four cross-modal feature enhancement modules (CFMs) and four dual-modal cross-guided attention modules (DMCTs). The dual-branch CSPDarket53 network obtains infrared modality and visible light modality semantic feature maps with channel numbers of 64, 128, 256, 512 and 1024. The infrared modality corresponding to the channel numbers of 64, 128, 256, 512 is converted into visible light modality semantic feature maps. The infrared modality and visible light modality features are respectively input into four cross-modal feature enhancement modules (CFM), and the infrared modality and visible light modality features are cross-enhanced in space and channels. The cross-enhanced infrared modality and visible light modality features are input into four dual-modal cross-guided attention modules (DMCT) to achieve long-distance feature interaction of infrared modality and visible light modality features. Among them, the features processed by the dual-modal cross-guided attention module (DMCT) are added to the features extracted by the dual-branch CSPDarket53 network to obtain fusion features with different numbers of channels. The neck network Neck fuses the fusion features of different channel numbers obtained by the backbone network Backbone through upsampling, feature enhancement module (FEM) and full connection layer (Concat) to obtain a fused high-level semantic feature map; The detection head Head is used to perform multi-scale target detection on the high-level semantic feature map after fusion by the neck network Neck, including anchor boxes, convolution layers, prediction layers and non-maximum suppression, and output prediction results.

4. The target detection method based on cross-modal fusion and guided attention mechanism according to claim 3, characterized in that: The construction of the backbone network Backbone in step 202 specifically includes: Step 2021: Build a dual-branch CSPDarket53 network; The dual-branch CSPDarket53 network is used to extract multi-scale feature information of infrared modality and visible light modality. The dual-branch CSPDarket53 network includes four CBS modules and four C3 modules. Both the CBS module and the C3 module are composed of convolutional layers Conv, activation functions and batch normalization operations, and the number of sampling channels is 64, 128, 256, 512 and 1024. A series of convolution operations are performed in the backbone network Backbone to extract semantic feature maps of infrared modality and visible light modality with channel numbers of 64, 128, 256, 512 and 1024. Step 2022: construct a cross-modal feature enhancement module (CFM); A number of 1×1 convolutions, a number of different activation functions (Sigmoid, Softmax), layer normalization (LayerNorm), and a number of global pooling (Global Pooling) are connected in series and parallel, and the dual-modal features are enhanced by cross-modulation at the spatial and channel levels; the infrared modality and visible light modality semantic feature maps with 64, 128, 256, and 512 channels extracted by step 2021 are input into a cross-modal feature enhancement module (CFM). The cross-modal feature enhancement module (CFM) uses the auxiliary modality to enhance the feature representation of another modality. The expression of the cross-modal feature enhancement module (CFM) is: in is the input feature of the i-th (i=1,2,3,4) stage, W q- , W v- represents a 1×1 convolution, σ1, σ2, σ3 represent the change tensor operation, and F SG , F SM , F GP They represent the activation function Sigmod, the activation function SoftMax, and the global pooling operation, respectively, and represent the element-by-element dot product of the tensor; Add the constructed cross-modal feature enhancement module (CFM) to the back of each C3 module of the dual-branch CSPDarket53 network in step 2021; Step 2023: construct a bimodal cross-guided attention module (DMCT); The dual-modal cross-guided attention module (DMCT) is composed of a cross-guided attention mechanism and a self-attention mechanism. The feature map processed by the cross-modal feature enhancement module (CFM) constructed in step 2022 is input into the dual-modal cross-guided attention module (DMCT). The infrared modality and the visible light modality features are multiplied to form a shared modality. The shared modality is used as the basic modality for guidance to guide the infrared modality and the visible light modality to achieve deep information interaction between the two modalities. It is expressed as the following formula: Among them, the self-attention layer mechanism is expressed as: Q=FW q ,K=FW k ,V=FW v The cross-guided attention mechanism (Guided-attention Layer Mechanisms) is expressed as: Among them, Q, K, V represent the query vector, key vector and value vector respectively, F represents the infrared image feature map and the visible light image feature map, W q ,W k ,W v represents the trainable weight matrix, and B represents the bias matrix; Connect the constructed bimodal cross-guided attention module (DMCT) to the cross-modal feature enhancement module (CFM) constructed in step 2022, add the features processed by the bimodal cross-guided attention module (DMCT) to the features extracted by the dual-branch CSPDarket53 network, and obtain fused features with different numbers of channels.

5. The target detection method based on cross-modal fusion and guided attention mechanism according to claim 3, characterized in that: The feature enhancement module (FEM) in step 202 specifically includes: channel attention (Channelattention), spatial attention (SpatialAttention), and four convolution branches; the features processed by the dual-modal cross-guided attention module (DMCT) are added to the features extracted by the dual-branch CSPDarket53 network to obtain fused features with different channel numbers, and the infrared mode and visible light mode fusion features with channel numbers of 256, 512 and 1024 are input into the feature enhancement module (FEM) of the neck network Neck, and the fused features are enhanced in channels and space through channel attention (Channellattention) and spatial attention (SpatialAttention), and the enhanced features are input into four convolution branches, and the output feature map of the convolution is subjected to feature addition operation to obtain the enhanced fused features, and the enhanced fused features outputted at each stage are fused through upsampling, a fully connected layer (Concat), and conventional convolution in the neck network Neck to obtain a high-level semantic feature map of the fused features; The four convolution branches include: two conventional convolution series branches, three conventional convolutions and one dilated convolution series branch, three conventional convolutions and one dilated convolution branch, and one conventional convolution branch.

6. The object detection method based on cross-modal fusion and guided attention mechanism according to claim 1, characterized in that: The network training parameter settings in step 3 are: training round epoch ≥ 100, batch size batch_size ≥ 16, learning rate ≤ 10 -5 , loss threshold ≤ 0.001, 0.9 ≤ momentum ≤ 1.

7. The object detection method based on cross-modal fusion and guided attention mechanism according to claim 1, characterized in that: The training parameter settings in step 4 are: batch size batch_size ≥ 16, correlation coefficient conf-thres ≤ 0.5, intersection-over-union coefficient iou-thres ≤ 0.5; The test parameter settings in step 5 are as follows: batch size batch_size ≥ 24, correlation coefficient conf-thres ≤ 0.5, and intersection-over-union coefficient iou-thres ≤ 0.

5.

8. An object detection system based on cross-modal fusion and guided attention mechanism based on the method according to any one of claims 1 to 7, characterized in that: include: A data set processing module is used to obtain a visible light-infrared image pairing data set, process and divide the visible light-infrared image pairing data set to obtain a training set, a validation set, and a test set; A network construction module is used to construct a multimodal target detection network, including an input network Input, a backbone network Backbone, a neck network Neck and a detection head Head. The backbone network Backbone includes a dual-branch CSPDarket53 network, four cross-modal feature enhancement modules (CFMs), and four bimodal cross-guided attention modules (DMCTs). The neck network Neck includes a feature enhancement module (FEM). The network training module is used to train and optimize the multimodal target detection network using the training set according to the set network training parameters. After the training, the training weight file is output, and the trained weight file is verified using the verification set, and the weight file with the highest accuracy is selected as the optimal weight file. The target detection module is used to load the test set and the optimal weight file into the multimodal target detection network, perform target detection on the test set, and obtain the target detection result.

9. An object detection device based on cross-modal fusion and guided attention mechanism, characterized in that: include: Memory: a computer program storing a target detection method based on cross-modal fusion and guided attention mechanism as described in any one of claims 1 to 7, which is a computer-readable device; Processor: used to implement the target detection method based on cross-modal fusion and guided attention mechanism as described in any one of claims 1-7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, can implement a target detection method based on cross-modal fusion and guided attention mechanism as described in any one of claims 1 to 7.

Images