Small target detection method based on multi-modal information fusion driving

Abstract

The invention discloses a small target detection method based on multi-modal information fusion driving, and relates to the technical field of visual detection, and the method comprises the steps: sending collected infrared and visible light image data into a network model for training, parameter fine tuning and other operations, and outputting a detection result; the network model comprises a multi-modal fusion module, a lightweight backbone network and a self-adaptive cascade query detection head network structure; using the prepared training data set to train the network model until the training frequency reaches a set threshold value or the value of the loss function reaches a set threshold value range; a prepared verification set is used to train and finely adjust the model, model parameters are optimized, and the performance of the network is further improved; according to the method, small target detection with higher precision and higher speed can be realized, and the information complementation effect is realized by utilizing the characteristics of the image in different modes.

Description

Technical Field

[0001] The present invention relates to the field of visual detection technology, and in particular to a small target detection method based on multimodal information fusion drive. Background Art

[0002] Target detection aims to detect the desired target in complex scenes and is an important research field in computer vision. It is increasingly used in areas such as unmanned driving and video surveillance. However, this type of visible light single-modal detection algorithm cannot cope with insufficient lighting conditions, which makes the small target detection model perform poorly at night. Infrared images reflect the thermal radiation information of objects and can highlight significant targets in the scene, but they are limited by their imaging principles and have low resolution. Therefore, how to effectively combine the two to achieve multi-modal fusion target detection tasks is still a huge challenge for researchers in this field.

[0003] The Chinese authorization announcement number is "CN118470557A", and the name is "A small target detection method and system based on visible light and infrared multimodal fusion". This method realizes the fusion of visible light and infrared images through pixel-level fusion, and then sends them to the improved YOLOV5 network to improve the detection accuracy; but the fusion ability of pixel-level fusion is weak and cannot fully utilize the complementary characteristics of multimodality; in addition, this method is a two-stage operation, first performing pixel-level fusion preprocessing, and then sending it to the detection model training, which is time-consuming and the model is complex.

[0004] In summary, how to make full use of the complementarity between different modalities to achieve efficient image fusion and accurate target detection to cope with complex and changing scenarios and needs is an urgent problem that researchers in this field need to solve. Summary of the invention

[0005] The technical solution of the present invention to solve the above technical problem is to provide a small target detection method based on multimodal information fusion drive, comprising the following steps:

[0006] Step 1. Prepare the data set: Prepare infrared image and visible light image data sets, and divide the data sets into training set, validation set, and test set;

[0007] Step 2. Build a network model: The network model includes a multimodal fusion module, a lightweight backbone network and an adaptive cascade query detection head;

[0008] The multimodal fusion module includes a shared encoder, an adaptive fusion module and a decoder;

[0009] The lightweight backbone network includes multiple convolution blocks, a CSP-G module, a CSP-L module, a neuron activation module and an SPPF module;

[0010] The adaptive cascade query detection head includes at least four identical detection heads and classification and regression networks, each detection head includes sparse convolution, point-by-point convolution and four prediction branches, and the prediction branches include category prediction, confidence prediction, position prediction and adaptive binary mask prediction;

[0011] Step 3. Train the network model: input the training set into the constructed detection network model for training;

[0012] Step 4. Select the loss function: including pixel loss, perceptual loss, gradient loss, category loss and regression loss; set the training loss threshold, and continuously iterate and optimize the model until the training number threshold is reached or the loss function value reaches the set threshold range;

[0013] Step 5. Fine-tune the model: Use the validation set to train and fine-tune the model and optimize the model parameters;

[0014] Step 6. Save the model: solidify the fine-tuned network parameters and determine the final detection network model.

[0015] Further, in step 2, the shared encoder is composed of a semantic branch and a detail branch, the detail branch includes at least two dense blocks, and the semantic branch includes at least two residual blocks;

[0016] The adaptive fusion module includes a feature purification attention module and a Kalman filter fusion module;

[0017] The decoder includes at least three convolutional layers for reconstructing a fused image.

[0018] Furthermore, the feature purification attention module is composed of a channel purification module and a space purification module; it is used to obtain important feature information in the channel dimension and the space dimension.

[0019] The Kalman filter fusion module consists of two Kalman filters and a Softmax activation function, and is used to achieve weighted fusion of different modal features.

[0020] Further, in step 2, the CSP-G module includes convolution block 1, convolution block 2, convolution block 3 and a Ghost bottleneck block;

[0021] The CSP-L module includes convolution block 1, convolution block 2, convolution block 3 and a learnable group convolution bottleneck block;

[0022] The SPPF module includes three serially connected maximum pooling blocks and two convolutional blocks to capture feature maps at different scales.

[0023] Furthermore, in step 2, the adaptive binary mask prediction uses the Gumbel-Softmax activation function to convert the feature map into a binary mask matrix; the classification and regression network is used to fuse the detection results of the four branches to obtain the final detection result.

[0024] Furthermore, in step 1, the labels of the dataset include three categories: person, people, and cyclist; image enhancement and data enhancement are performed on the source images before the fusion images.

[0025] Furthermore, in step 5, when the model is trained using the validation set, the number of training times is set to be smaller than the number of training times in step 3.

[0026] Compared with the prior art, it has the following beneficial effects:

[0027] (1) The present invention provides a new feature-level multimodal fusion small target detection task processing framework. Different from other multimodal fusion detection task processing methods, this framework integrates fusion and detection to achieve end-to-end multimodal fusion detection tasks. At the same time, it improves the detection network structure, can reduce the model complexity and obtain more feature information, and achieve higher accuracy and faster detection.

[0028] (2) The present invention designs a multimodal fusion module with an adaptive fusion module as the core. A feature purification attention module is designed in this module to highlight the important features of infrared and visible light. At the same time, a Kalman filter fusion is designed to solve the shortcoming of not fully utilizing the information characteristics between different modalities and improve the detection accuracy.

[0029] (3) The present invention uses the Ghost module to improve CSP to obtain CSP-G, and replaces part of the convolution with linear transformation to achieve lightweight while ensuring unchanged accuracy. It uses learnable group convolution to improve CSP-L, and prunes redundant information by screening important features. The two work together to extract deep semantic information and shallow feature information, and achieve efficient small target detection.

[0030] (4) The present invention designs an adaptive cascade query detection head. By designing a prediction branch, a prediction mask is generated to quickly locate the local target position of the lower-level detection feature, thereby solving the problem of complex detection head structure. At the same time, a fusion loss function is added on the basis of the original target loss, and the feedback of the fusion loss is used to improve the feature fusion effect, thereby promoting more accurate downstream detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.

[0032] Figure 1 It is a flowchart of the steps of the small target detection method driven by multimodal information fusion;

[0033] Figure 2 This is a schematic diagram of the network model structure of the present invention;

[0034] Figure 3 Schematic diagram of the dense block and residual block structure of the present invention;

[0035] Figure 4 It is a structural schematic diagram of the feature purification attention module in the adaptive fusion module of the present invention;

[0036] Figure 5 This is a schematic diagram of the Kalman filter fusion structure in the adaptive fusion module of the present invention;

[0037] Figure 6 This is a schematic diagram of the standard convolution and learnable group convolution block structure of the present invention;

[0038] Figure 7 This is a schematic diagram of the Ghost bottleneck block structure of the present invention;

[0039] Figure 8 This is a schematic diagram of the CSP-L module structure of the present invention;

[0040] Fig. 9 This is a schematic diagram of the CSP-G module structure of the present invention;

[0041] Fig.10 This is a schematic diagram of the structure of the adaptive cascade query detection head described in the present invention. DETAILED DESCRIPTION

[0042] The present invention proposes a small target detection method based on multimodal information fusion drive, aiming to design a small target detection method based on multimodal information fusion drive, which is used to solve the problems of low detection efficiency caused by insufficient fusion of infrared and visible light features and complex network models in existing methods.

[0043] The small target detection method based on multimodal information fusion drive proposed by the present invention will be described in the following specific embodiments:

[0044] In the technical solution of this embodiment, Figure 1As shown, a small target detection method based on multimodal information fusion drive includes the following steps:

[0045] Step 1. Prepare the data set: Prepare infrared image and visible light image data sets, and divide the data sets into training set, validation set, and test set;

[0046] Specifically, the dataset is the KAIST dataset, which is divided into training set, validation set and test set in a ratio of 8:1:1, and the dataset is enhanced using mosaic enhancement, adaptive scaling, inversion and other methods.

[0047] Step 2. Build a network model: The network model includes a multimodal fusion module, a lightweight backbone network and an adaptive cascade query detection head;

[0048] The multimodal fusion module includes a shared encoder, an adaptive fusion module and a decoder;

[0049] The lightweight backbone network includes multiple convolution blocks, a CSP-G module, a CSP-L module, a neuron activation module and an SPPF module;

[0050] The adaptive cascade query detection head includes at least four identical detection heads and classification and regression networks, each detection head includes sparse convolution, point-by-point convolution and four prediction branches, and the prediction branches include category prediction, confidence prediction, position prediction and adaptive binary mask prediction;

[0051] Specifically, the shared encoder is used to extract deep semantic information and shallow feature information from infrared and visible light images; the adaptive fusion module is used to effectively fuse infrared feature maps with visible light feature maps; the decoder is used to reconstruct the fused image and supervise the quality of the reconstructed image with the fused image loss; the convolution block is used to extract image features; the CSP-G module is used to deepen feature extraction and improve detection accuracy; the CSP-L module is used to extract shallow features; the SPPF is used to capture feature maps of different scales; the neuron activation module is used to increase the focus on target information. Sparse convolution is used to sparsely convolve the output of the feature expression network with the mask matrix of the previous level, accurately query the target position and reduce the amount of calculation; point-by-point convolution is used to enhance global context information; four branches are used to predict category, confidence, position and mask matrix.

[0052] Step 3. Train the network model: input the training set into the constructed detection network model for training;

[0053] Step 4. Select the loss function: including pixel loss, perceptual loss, gradient loss, category loss and regression loss; set the training loss threshold, and continuously iterate and optimize the model until the training number threshold is reached or the loss function value reaches the set threshold range;

[0054] Specifically, category loss is used to solve the imbalance of positive and negative samples of the target; regression loss is used to calculate angle, distance, shape and IOU loss to improve the accuracy of bounding box regression; gradient loss is used to adapt the specific convolution kernel of the convolution operation to perform edge detection and retain the gradient information of the image; pixel loss is used to calculate the pixel error between the network input and output images, and minimize the pixel loss between the reconstructed image and the original image; perceptual loss is used to help the network pay more attention to the content and structure of the image, pay more attention to the texture information of the image, and reconstruct a more realistic original image.

[0055] Step 5. Fine-tune the model: Use the validation set to train and fine-tune the model and optimize the model parameters;

[0056] Step 6. Save the model: solidify the fine-tuned network parameters and determine the final detection network model.

[0057] Further, in step 2, the shared encoder is composed of a semantic branch and a detail branch, the detail branch includes at least two dense blocks, and the semantic branch includes at least two residual blocks;

[0058] The adaptive fusion module includes a feature purification attention module and a Kalman filter fusion module;

[0059] The decoder includes at least three convolutional layers for reconstructing a fused image.

[0060] Furthermore, the feature purification attention module is composed of a channel purification module and a space purification module; and is used to obtain important feature information in the channel dimension and the space dimension;

[0061] like Figure 5 As shown, the Kalman filter fusion module consists of two Kalman filters and a Softmax activation function, and is used to achieve weighted fusion of different modal features.

[0062] Further, in step 2, the CSP-G module includes convolution block 1, convolution block 2, convolution block 3 and a Ghost bottleneck block;

[0063] The CSP-L module includes convolution block 1, convolution block 2, convolution block 3 and a learnable group convolution bottleneck block;

[0064] The SPPF module includes three serially connected maximum pooling blocks and two convolution blocks to capture feature maps at different scales;

[0065] Furthermore, in step 2, the adaptive binary mask prediction uses a Gumbel-Softmax activation function to convert the feature map into a binary mask matrix; the classification and regression network is used to fuse the detection results of the four branches to obtain the final detection result;

[0066] Furthermore, in step 1, the labels of the dataset include three categories: person, people, and cyclist; image enhancement and data enhancement are performed on the source images before the fusion images.

[0067] Furthermore, in step 5, when the model is trained using the validation set, the number of training times is set to be smaller than the number of training times in step 3.

[0068] Embodiment 2:

[0069] A small target detection method based on multimodal information fusion drive includes the following steps:

[0070] Step 1, prepare the data set: prepare the KAIST data set for network training. The KAIST data set includes a total of 95,328 pictures. The data set captures various regular traffic scenes including campuses, streets and the countryside during the day and at night. It is divided into training set, validation set and test set in a ratio of 8:1:1. The labels of the data set include three categories: person, people and cyclist. In order to solve the problems existing in visible light and infrared images, the source images are enhanced and data enhanced before the fusion images. The image enhancement methods mainly include histogram equalization, smoothing, sharpening, edge enhancement, etc. The data enhancement methods include mosaic, rotation, scaling, cropping, etc.

[0071] Step 2: Build a network model. The structure of the network model is as follows: Figure 2 As shown in , it mainly includes a multimodal fusion module, a lightweight backbone network and an adaptive cascade query detection head; the multimodal fusion module includes a shared encoder, an adaptive fusion module and a decoder. The shared encoder consists of a dense block and a residual block, as shown in Figure 3 As shown in the figure, the dense block includes three layers of convolution, and the output of each layer of convolution is concatenated with the input; the residual block contains three layers of convolution, and the input is added to the final output through jump connections; the adaptive fusion module first processes the output feature map of the encoder using dilated convolutions with different dilation rates (1, 3, 5) to obtain rich global information, and then concatenates it with the cascade feature map, so that the feature information of different scales can be used to determine whether to retain important information and ignore redundant information; then the cascade features are integrated through 1×1 convolution and sent to the channel purification module and the spatial purification module respectively. Figure 4 The GAP and GMP in represent global average pooling and global maximum pooling respectively;

[0072] Channel purification module: Assume that a feature map M∈R is given H×W×CFirst, GAP and GMP are applied jointly in the spatial dimension to obtain global context information, and the two feature vectors obtained are added to obtain a channel feature vector d∈R 1 ×1×C ; Secondly, adaptive one-dimensional convolution is used to capture the information between different channels. Finally, the Sigmoid function is used to activate the channel feature vector e∈R 1×1×C ;

[0073] Adaptive one-dimensional convolution can adaptively calculate the convolution kernel size k according to the number of channels. k is a key parameter. The selection of k may be different for different numbers of channels. Although it can be fine-tuned slowly, it is more computationally expensive. There is a mapping relationship between k and C:

[0074] C=φ(k)

[0075]

[0076] Among them, γ and b represent hyperparameters, and odd is rounded down to the nearest odd number;

[0077] Spatial purification module: Assume that a feature map N∈R is given H×W×C , first apply GAP and GMP operations to the input features independently along the channel dimension and concatenate the outputs to form the feature f∈R H×W×2 ; Secondly, the feature f is subjected to channel dimension reduction using 1×1 convolution. Finally, the Sigmoid function is used to activate the feature g∈R H×W×1 ;

[0078] The two feature maps K carrying different modalities R , K T As the current observation feature map, K R , K T The feature map added by elements is used as the fusion state feature map of the previous moment, and then the feature map K R , K T The fusion state feature map at the previous moment is sent to the respective Kalman filters, and the current state prediction and covariance prediction are first performed. The formula is as follows:

[0079]

[0080] P k|k-1 =A·P k-1 ·A T +Q

[0081] Among them, A is the state transfer matrix, F k-1 is the fusion feature map of the previous moment, B,u k To control input and noise, Q is the process noise covariance;

[0082] Then the Kalman gain is calculated, and finally the current observed feature map and the predicted feature map are weighted to update the current feature map. That is:

[0083] K k =P k|k-1 ·H T ·(H·P k|k-1 ·H T +R) -1

[0084]

[0085] Among them, Z k The current observation value, H is the observation matrix, and R is the observation noise covariance;

[0086] Finally, the updated feature map uses the Softmax activation function to generate weights and feature maps K R , K T Multiply them and then add them element-wise;

[0087] The decoder is composed of convolutional layers 2, 3, and 4. After the fused features are convolved three times, the number of channels in the decoder is gradually reduced to one, and a reconstructed image or a fused image can be obtained. The convolution operation of the decoder uses a 3×3 padding kernel. The purpose of the decoder to reconstruct the image is not to achieve the fusion task, but to use the quality of the fused image as supervision to promote the downstream target detection to improve accuracy.

[0088] like Figure 2 As shown in the figure, the lightweight backbone network consists of convolution block 1, convolution block 2, convolution block 3, convolution block 4, 4 CSP-G modules, 4 CSP-L modules, a neuron activation module and an SPPF module. The convolution block uses a standard two-dimensional convolution, and the convolution block also includes normalization and activation functions. Figure 6 It is shown that the expressiveness of features is enhanced through batch normalization and activation function;

[0089] Learnable group convolution uses a mask matrix to filter out a subset of less important input features from each group during the entire training period. Sparse operations are achieved by setting the convolution kernel to zero; each input channel can be assigned a learnable weight (or mask), and these weights determine which group the channel belongs to. In this way, the grouping is no longer predefined, but is dynamically adjusted through these weights. Specific steps:

[0090] Initialize the weight vector W;

[0091] Use discretization methods such as softmax to convert the weights into the assignment probability of each channel to determine which group it belongs to;

[0092] According to these weights, the input channels are divided into different groups and convolution operations are performed separately;

[0093] During the training process, the weights are gradually updated through back-propagation to learn the optimal grouping method;

[0094] Ghost module (such as Figure 7 ) Use standard convolution blocks to obtain some feature maps, and then generate more feature maps through linear operations (1×1 point-by-point convolution, batch normalization, and RELU activation function); finally, concatenate two sets of feature maps in the specified dimension to obtain more feature maps with fewer parameters and calculations;

[0095] The neuron activation module is a lightweight, parameter-free convolutional neural network attention mechanism that defines an energy function based on findings in neuroscience, specifically the phenomenon of spatial inhibition, whereby the activation of certain neurons inhibits the activity of surrounding neurons in visual processing. The energy function is designed to quantify the linear separability between the target neuron and other neurons.

[0096] The energy function is defined as minimizing the distance between the target neuron and other neurons while maximizing the distance between the target neuron and a fixed value y t The degree of closeness is:

[0097]

[0098] Combining Ghost modules and learnable group convolutions in CSP, such as Figure 8 and Fig. 9 As shown in the figure, the adaptive cascade query detection head uses a decoupled head to predict the category, location and confidence information respectively, as shown in the figure. Fig.10 As shown in the figure, adaptive mask prediction is added to the prediction branch, and the feature map is converted into a mask matrix using the Gumbel-Softmax activation function, and sent to the feature map of the next level for sparse convolution. Since the sparse convolution performs convolution on the positions where the target may exist according to the mask matrix, the context information is ignored. Therefore, the global context information generated by point-by-point convolution is added after the sparse convolution to enrich the feature information; in addition, a 3×3 and a 1×1 convolution layer are used in the prediction branches of category, position and confidence.

[0099] Step 3, training the network model: training the detection network model, performing data enhancement on the training set prepared in step S1, and inputting the preprocessed data set into the network model constructed in step S2 for training; setting the number of training rounds to 300, the batch size to 16, the initial learning rate to 0.01, the parameter optimizer using the stochastic gradient descent algorithm, and the target detection loss function threshold to 0.05. When the loss value is lower than the threshold, the model training is basically completed;

[0100] Step 4, select a suitable loss function: the common loss functions of the target detection algorithm under a single modality include category loss and regression loss (position loss). Since the present invention is aimed at small target detection of infrared and visible light fusion images, the image comprehensive loss is introduced for the fusion image, including pixel loss, gradient loss and perception loss;

[0101] The pixel loss function can calculate the pixel error between the reconstructed image and the input image. The formula is as follows:

[0102] L 像素 =MSE(I re ,I in )

[0103] Where Ire is the reconstructed image, Iin is the input image, and MSE(x,y) is the mean square error function of x and y.

[0104] The gradient loss function can calculate the edge information loss of the reconstructed image and the input image. The formula is as follows:

[0105] L 梯度 =MSE(Gradient(I re ),Gradient(I in ))

[0106] Wherein, Gradient(x) is the image sharpening using the Laplacian operator to obtain the gradient map.

[0107] The perceptual loss function can calculate the feature error between the reconstructed image and the input image;

[0108]

[0109] Where φi(x) represents the features of the i-th layer obtained by inputting image I into a specific trained network;

[0110] In addition, the existing target detection algorithm is improved. For category loss, there is often an imbalance of positive and negative samples of the target in target detection. In order to solve this problem, the Focal loss function is introduced. The specific formula is as follows:

[0111] FL(pt )=-α t (1-p t ) γ log(p t )

[0112] where p t is the predicted probability of the sample;

[0113] In addition α t Is a balancing factor used to balance the importance of positive and negative samples; for positive samples, α t =α, for negative samples, α t =1-α; γ is a tuning parameter (often called focusing parameter) used to adjust the weight of easy-to-classify samples;

[0114] The position loss uses SIoU Loss, which not only considers the intersection-over-union ratio of the traditional IoU, but also introduces the shape and angle information of the bounding box to better align the prediction box and the target box; the regression loss function is defined as follows:

[0115]

[0116] Where Δ is:

[0117]

[0118] Ω is:

[0119]

[0120] Step 5, fine-tune the model: Use the validation set to train the model, set the number of training times to 150, keep other parameters unchanged, and optimize the model weights based on the training;

[0121] Step 6, save the model: after the fine-tuning training in step 5 is completed, solidify the fine-tuned network parameters and determine the final detection network model;

[0122] Among them, the implementation of convolution, pooling, activation function, channel splicing and feature map addition operations is a method well known to those skilled in the art, and the principles thereof will not be repeated here.

[0123] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A small target detection method based on multimodal information fusion drive, characterized in that: The following steps are involved: Step 1. Prepare the data set: Prepare infrared image and visible light image data sets, and divide the data sets into training set, validation set, and test set; Step 2. Build a network model: The network model includes a multimodal fusion module, a lightweight backbone network and an adaptive cascade query detection head; The multimodal fusion module includes a shared encoder, an adaptive fusion module and a decoder; The lightweight backbone network includes multiple convolution blocks, a CSP-G module, a CSP-L module, a neuron activation module and an SPPF module; The adaptive cascade query detection head includes at least four identical detection heads and classification and regression networks, each detection head includes sparse convolution, point-by-point convolution and four prediction branches, and the prediction branches include category prediction, confidence prediction, position prediction and adaptive binary mask prediction; Step 3. Train the network model: input the training set into the constructed detection network model for training; Step 4. Select the loss function: including pixel loss, perceptual loss, gradient loss, category loss and regression loss; set the training loss threshold, and continuously iterate and optimize the model until the training number threshold is reached or the loss function value reaches the set threshold range; Step 5. Fine-tune the model: Use the validation set to train and fine-tune the model and optimize the model parameters; Step 6. Save the model: solidify the fine-tuned network parameters and determine the final detection network model.

2. The small target detection method based on multimodal information fusion drive according to claim 1 is characterized in that: In step 2, the shared encoder consists of a semantic branch and a detail branch, the detail branch includes at least two dense blocks, and the semantic branch includes at least two residual blocks; The adaptive fusion module includes a feature purification attention module and a Kalman filter fusion module; The decoder includes at least three convolutional layers for reconstructing a fused image.

3. The small target detection method based on multimodal information fusion drive according to claim 2 is characterized in that: The feature purification attention module consists of a channel purification module and a space purification module; it is used to obtain important feature information in the channel dimension and the space dimension; The Kalman filter fusion module is composed of two Kalman filters and a Softmax activation function, and is used to achieve weighted fusion of different modal features.

4. The small target detection method based on multimodal information fusion drive according to claim 1 is characterized in that: In step 2, the CSP-G module includes convolution block 1, convolution block 2, convolution block 3 and Ghost bottleneck block; The CSP-L module includes convolution block 1, convolution block 2, convolution block 3 and a learnable group convolution bottleneck block; The SPPF module includes three serially connected maximum pooling blocks and two convolutional blocks to capture feature maps at different scales.

5. The small target detection method based on multimodal information fusion drive according to claim 1 is characterized in that: In step 2, the adaptive binary mask prediction uses the Gumbel-Softmax activation function to convert the feature map into a binary mask matrix; the classification and regression network is used to fuse the detection results of the four branches to obtain the final detection result.

6. The small target detection method based on multimodal information fusion drive according to claim 1 is characterized in that: In step 1, the labels of the dataset include three categories: person, people, and cyclist; image enhancement and data enhancement are performed on the source images before fusion.

7. The small target detection method based on multimodal information fusion drive according to claim 1 is characterized in that: In step 5, when the model is trained using the validation set, the number of training times is set to be smaller than the number of training times in step 3.

Images