Air-to-ground small target multi-modal detection method based on adaptive modal selection strategy

Abstract

The invention discloses an air-to-ground small target multi-modal detection method based on an adaptive modal selection strategy, and the method comprises the following steps: S1, constructing a deep learning convolutional network initial frame model which comprises a two-way convolutional neural network, a strategy module, a fusion module and a detection head which are sequentially arranged; s2, training the convolutional network initial framework model by using a training set in a visible light-infrared image pair-based unmanned aerial vehicle crowd detection data set to obtain a target detection model; and S3, obtaining a to-be-detected image, inputting the to-be-detected image into the target detection model, carrying out feature extraction, screening and fusion on the to-be-detected image through the target detection model, and finally outputting a target position result. According to the method, the problems of small size of the target to be detected, unbalanced modal space and redundant modal information in the existing air-to-ground small target detection technology are solved, and the influence of interference factors such as poor illumination conditions, partial shielding and overlapping can be resisted.

Description

Technical Field

[0001] The present invention belongs to the technical field of target detection, and relates to a method and device for multimodal detection of small air-to-ground targets, and in particular to a method for multimodal detection of small air-to-ground targets based on an adaptive mode selection strategy. Background Art

[0002] Object detection is a basic technology in various computer vision applications such as surveillance, search and rescue, and fault monitoring. In recent years, with the development of neural networks and artificial intelligence technologies, object detection technology based on convolutional neural networks (CNNs) has made great progress. At present, mainstream object detectors can be divided into two categories: two-stage detectors and single-stage detectors. Generally speaking, two-stage detectors have higher detection accuracy but slower execution speed, while single-stage detectors have slightly lower detection accuracy although they execute faster. However, these detectors are generally based on single-modal visible light images, so their detection performance will be greatly reduced under conditions such as low light or bad weather (such as rain, fog, and haze).

[0003] Therefore, the target detector that combines infrared and visible light dual-modal information has gradually been favored by researchers as a feasible and effective improvement solution. Experimental results show that the fusion of visible light and infrared dual-modal information can usually improve the target detection performance and better resist the influence of adverse factors such as insufficient light and bad weather. However, the visible light and infrared multi-modal target detection technology for small air-to-ground targets is still in the exploratory stage and currently faces the following three major challenges:

[0004] 1. The scale of the target to be detected is small. Since the images used in air-to-ground small target detection are all collected by drones, and drones collect images at a high altitude and with a wide field of view, the captured target objects usually appear small, contain fewer details, and are greatly affected by the cluttered background. Therefore, it is quite challenging to overcome various interference factors in the air-to-ground small target detection task and accurately detect tiny targets with limited information.

[0005] 2. Imbalance in modal space. The information contained in visible light images and infrared images has significantly different characteristics and can provide complementary clues for target detection tasks. For example, visible light sensors can capture images with rich details, color information, and textures under well-lit conditions. In contrast, infrared detectors can capture the temperature difference between the target and its surroundings and show strong performance under insufficient lighting conditions. Therefore, in different scenarios, the importance of visible light modalities and infrared modalities for target detection tasks is not the same. How to effectively balance or fuse multimodal features remains a difficult problem to be solved.

[0006] 3. Modal information redundancy. In a typical air-to-ground small target detection scenario, the visible light and infrared images captured by drones contain a lot of invalid information from the cluttered background. In order to improve the accuracy of detecting small targets, it is reasonable to remove these invalid inputs. In addition, small targets usually look similar in visible light and infrared images captured using drone platforms. Therefore, it is feasible to reduce redundant information in multimodal data without reducing the overall detection performance. However, how to design an effective scheme to remove invalid inputs and reduce redundant information remains to be explored. Summary of the invention

[0007] In view of the above technical problems, the purpose of the present invention is to provide an air-to-ground small target multimodal detection method based on an adaptive modal selection strategy, which selects effective modal information for fusion through an adaptive modal selection mechanism to improve the accuracy and robustness of small target detection.

[0008] The technical solution adopted by the present invention is as follows:

[0009] A multi-modal detection method for small air-to-ground targets based on an adaptive modality selection strategy comprises the following steps:

[0010] S1. constructing an initial framework model of a deep learning convolutional network, wherein the initial framework model of the convolutional network includes a dual-path convolutional neural network, a strategy module, a fusion module and a detection head arranged in sequence;

[0011] S2. Use the training set in the drone crowd detection dataset based on visible light-infrared image pairs (RGBTDronePerson) to train the initial framework model of the convolutional network to obtain the target detection model;

[0012] S3. Obtain the image to be detected and input it into the target detection model. The target detection model extracts, filters and fuses the features of the image to be detected, and finally outputs the target position result.

[0013] Furthermore, the dual-path convolutional neural network processes the visible light image and the infrared image respectively, extracts feature maps of the two modalities, and obtains a dual-path feature map.

[0014] Furthermore, the strategy module is used to perform modal screening on the dual-path feature maps extracted by the dual-path convolutional neural network to remove invalid or highly interfering modal information.

[0015] Furthermore, the fusion module is used to integrate the complementary information of the two-way feature map after being filtered by the strategy module.

[0016] Furthermore, the convolutional layer parameters of the dual-path convolutional neural network are initialized using the weights and biases of a residual network model (ResNet50) pre-trained on a large-scale image recognition dataset (ImageNet), and other convolutional layer parameters are randomly initialized using a Gaussian normal distribution.

[0017] Furthermore, in step S2, when the initial framework model of the convolutional network is trained, a strategy of sequentially generating small batches of data is adopted for data input; a gradient descent method is used for parameter update, and gradient clipping is used to adjust the model parameters.

[0018] Furthermore, the fusion module identifies key areas in the feature map through a spatial attention mechanism and reduces the influence of irrelevant areas.

[0019] An air-to-ground small target multi-modal detection system based on an adaptive modal selection strategy comprises:

[0020] Model construction module: used to construct an initial framework model of a deep learning convolutional network, wherein the initial framework model of the convolutional network includes a dual-path convolutional neural network, a strategy module, a fusion module, and a detection head arranged in sequence;

[0021] Model training module: used to train the initial framework model of the convolutional network using the training set in the drone crowd detection dataset based on visible light-infrared image pairs to obtain the target detection model;

[0022] Image detection module: used to obtain the image to be detected and input it into the target detection model. The target detection model is used to extract, filter and fuse the features of the image to be detected, and finally output the target position result.

[0023] A computer device, comprising:

[0024] one or more processors;

[0025] A memory for storing one or more programs;

[0026] When the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned air-to-ground small target multi-modal detection method based on the adaptive modality selection strategy.

[0027] A computer-readable storage medium storing computer instructions, when the computer instructions are executed by one or more processors, causes the one or more processors to execute the steps in the above method.

[0028] The beneficial effects of the present invention are:

[0029] 1. The present invention can make full use of the complementary information of visible light and infrared modes to achieve accurate detection of small targets at a long distance, and can still stably detect small targets under poor lighting conditions such as cloudy days and nights, and has strong robustness.

[0030] 2. The present invention introduces a strategy module and utilizes an adaptive modal screening mechanism to remove invalid or interfering modal information, thereby greatly overcoming the problems of modal space imbalance and modal information redundancy and improving detection performance.

[0031] 3. The present invention fuses the feature maps of visible light and infrared modalities by designing a specific fusion module, highlighting the target area and suppressing interference factors such as background, while utilizing the spatial attention mechanism to enhance the detection performance for small targets.

[0032] 4. The present invention not only has excellent detection performance, but also has a relatively simple model. It is suitable for deployment on devices with limited computing power such as drones, and can meet the application scenarios of air-to-ground small target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a flowchart of the algorithm steps of an embodiment of the present invention.

[0034] Figure 2 It is a schematic diagram of the overall structure of the convolutional neural network model used in an embodiment of the present invention.

[0035] Figure 3 It is a schematic diagram of the structure of the strategy module in the convolutional neural network model of an embodiment of the present invention.

[0036] Figure 4 It is a structural schematic diagram of the fusion module in the convolutional neural network model of an embodiment of the present invention.

[0037] Figure 5 It is the target detection result (infrared channel display) of an embodiment of the present invention in a residential area scene in a drone crowd detection dataset based on visible light-infrared image pairs.

[0038] Figure 6 This is the target detection result (infrared channel display) of an embodiment of the present invention in an urban road scene in a drone crowd detection dataset based on visible light-infrared image pairs.

[0039] Figure 7 This is the target detection result (infrared channel display) of an embodiment of the present invention in a park scene in a drone crowd detection dataset based on visible light-infrared image pairs.

[0040] Figure 8This is the target detection result (infrared channel display) of an embodiment of the present invention in a school playground scene in a drone crowd detection dataset based on visible light-infrared image pairs. DETAILED DESCRIPTION

[0041] The technical solution of the present invention will be further explained in detail below in conjunction with the accompanying drawings and specific embodiments.

[0042] The present invention provides an air-to-ground small target multi-modal detection method based on an adaptive modal selection strategy, comprising the following steps:

[0043] S1. Construct an initial framework model of a deep learning convolutional network, which includes a dual-channel convolutional neural network, a strategy module, a fusion module and a detection head arranged in sequence. The dual-channel convolutional neural network processes visible light images and infrared images respectively, performs dual-channel feature extraction, outputs feature maps corresponding to the two modalities, and obtains a dual-channel feature map. The dual-channel feature map is input into the strategy module for modality screening to remove invalid or interfering modal information. The screened dual-channel feature map is input into the fusion module for complementary information integration to obtain the fused feature map, and finally the fused feature map is input into the subsequent detection head to output the target position result finally predicted by the target detection model.

[0044] S2. The training set in the drone crowd detection dataset based on visible light-infrared image pairs is used to train the initial framework model of the convolutional network to obtain the target detection model; specifically, the training set in the drone crowd detection dataset based on visible light-infrared image pairs is used as training data to input into the initial framework model of the convolutional network for training, and the detection labels in the dataset are used as supervision information for supervision, so as to obtain the target detection model. During training, the convolutional layer parameters of the two-way convolutional neural network in the visible light image and infrared image feature extraction channels are initialized with the weights and biases of the residual network model (ResNet50) pre-trained on the large-scale image recognition dataset (ImageNet), while all other convolutional layers are randomly initialized using Gaussian normal distribution. When the training data of the visible light image and infrared image in the dataset are input into the convolutional neural network for training, the strategy of sequentially generating small batches of data is adopted, and the batch size is 2. The initial framework model of the convolutional network uses the gradient descent method to update parameters during the training process, and needs to be trained for several rounds. During training, the size of the learning rate will gradually decay with the increase of training rounds, and the model parameters are adjusted by gradient clipping.

[0045] S3. Obtain the image to be detected and input it into the target detection model, extract, filter and fuse the features of the image to be detected through the target detection model, and finally output the target position result. Among them, the input image to be detected needs to include both visible light image and infrared image.

[0046] Example

[0047] See also Figure 1 The specific implementation of the air-to-ground small target multi-modal detection method based on the adaptive mode selection strategy of the embodiment of the present invention includes the following steps:

[0048] Step 1: Build the initial framework model of deep learning convolutional network.

[0049] The initial framework model structure of the convolutional network can be found in Figure 2 The initial architecture of the convolutional network includes a dual-channel convolutional neural network (C1-C5), a policy module, a fusion module, and a detection head. The dual-channel convolutional neural network processes visible light images and infrared images respectively, extracts dual-channel features, and then inputs them into the policy module, fusion module, and detection head to obtain the output results.

[0050] The strategy module is used to perform modal filtering operations on the feature graphs of the two modalities. The structure of the strategy module can be found in Figure 3 , its workflow is as follows:

[0051] First, the dual-path feature map is input into the adaptive average pooling layer and the output is cascaded and flattened to obtain a one-dimensional vector :

[0052]

[0053] in and Represents the feature maps corresponding to the input visible light and infrared modes respectively; and Indicates the adaptive average pooling layer that each of the two feature maps passes through; Indicates a cascade operation; Indicates flattening the result into a one-dimensional vector.

[0054] Then the vector Input a series of fully connected layers to perform classification operations and obtain discrete classification results:

[0055]

[0056]

[0057] in correspond Figure 3 A series of fully connected layers in ; and They correspond to the classification results of the fully connected layer for the two modalities; Indicates that and The result after cascading.

[0058] Since the classification results output by the fully connected layer are discrete, the model will not be differentiable in subsequent training steps, and the model parameters cannot be updated through back propagation. Therefore, the Gumbel-Softmax distribution strategy is introduced in this module to approximate the discrete classification results with a continuous distribution. The process can be expressed as:

[0059]

[0060] in Represents the use of Gumbel normalized distribution for discrete results Make an approximation; and The strategy outputs corresponding to visible light and infrared modalities respectively; Indicates that the results are combined and output together.

[0061] Finally, the feature maps of the two modal inputs ( and ) and its corresponding strategy output ( and ) is multiplied to obtain the feature map of the final output of the strategy module ( and ):

[0062]

[0063] The fusion module is used to integrate the complementary information of the filtered dual-path feature maps. The structure of the fusion module can be found in Figure 4 , which consists of two parts: the Attention Module and the Multi-convolution Module (C3 Module). The workflow of the fusion module is as follows:

[0064] First, the dual-path feature map of visible light and infrared modalities output by the strategy module ( and ) is cascaded and input into the attention module. The process can be expressed as:

[0065]

[0066]

[0067] in Indicates that the dual-path feature map and The result after cascading; and Represent the maximum pooling operation and the average pooling operation respectively; Represents a two-dimensional convolution operation; represents the activation function; Represents a multiplication operation; Represents the feature map output after the attention module.

[0068] Then After further feature extraction is performed in the input multi-convolution module, the feature map output by the fusion module is obtained. , the process can be expressed as:

[0069]

[0070] in Represents addition operation; express Figure 4 The operations corresponding to each normalized convolution (CBS) module in, ; Represents a joint operation consisting of multiple normalized convolution modules.

[0071] Step 2: Use the training set in the drone crowd detection dataset based on visible light-infrared image pairs as training data and input it into the convolutional network initial framework model for training, and use the detection labels in the dataset as supervision information for supervision, so as to obtain the target detection model.

[0072] The dataset used in the embodiment of the present invention is a drone crowd detection dataset based on visible light-infrared image pairs. Its training set consists of 4900 pairs of visible light and infrared image pairs, including 43006 pedestrian objects, 4869 rider objects and 7316 crowd objects.

[0073] When the training data of visible light images and infrared images in the dataset are input into the initial framework model of the convolutional network for training, the following specific operations are included but not limited to:

[0074] 1. During training, the input of visible light images and infrared images uses a method of sequentially generating small batches of data, with a batch size of 2.

[0075] 2. During training, the convolutional layer parameters in the dual-path feature extraction channels of visible light and infrared images are initialized using the weights and biases of the residual network model pre-trained on a large-scale image recognition dataset, while all other convolutional layers are randomly initialized using a Gaussian normal distribution.

[0076] 3. The initial framework model of the convolutional network uses the gradient descent method to update parameters during the training process, and requires several rounds of training. During training, the learning rate will gradually decay with the increase of training rounds, and the model parameters are adjusted by gradient clipping.

[0077] The overall loss function used when training the model It can be expressed as:

[0078]

[0079] in represents the classification loss used to determine the category to which the target belongs; represents the regression loss for determining the target location and size; represents the policy loss used to supervise the policy module; The proportional factor that represents the weight of the balance strategy loss and the other two losses.

[0080] The classification loss selected in the embodiment of the present invention is the cross entropy loss, which is expressed as:

[0081]

[0082] in represents the true value of the category distribution probability, The predicted value representing the probability of the class distribution.

[0083] The regression loss selected in the embodiment of the present invention is the smooth mean absolute error loss (smooth L1 loss), which is expressed as:

[0084]

[0085] in Represents the difference between the predicted box and the true box.

[0086] In the embodiment of the present invention, a strategy loss is specially designed to embed the strategy module into the overall model to perform end-to-end joint training. The strategy loss can be expressed as:

[0087]

[0088] in represents the total number of modes, which is set to 2 in the formula; Corresponding to visible light and infrared modes; Indicates the output of the strategy module corresponding to the two modes; Represents the weight factors of the visible light mode and the infrared mode respectively. Since the visible light mode and the infrared mode are considered to be equivalent, the weight factors corresponding to the two modes are set to 1; Indicates the detection accuracy of the model; Represents the proportional factor between the first and second terms of the equilibrium strategy loss.

[0089] Step 3: Obtain the image to be detected and input it into the target detection model, extract, filter and fuse the features of the image to be detected through the target detection model, and finally output the target position result. The input image to be detected needs to include both visible light image and infrared image.

[0090] Step 3.1: Get the image to be detected and input it into the trained target detection model.

[0091] Since the deep learning convolutional network initial framework model constructed in the embodiment of the present invention receives both visible light images and infrared images as input, the image to be detected needs to contain images of both visible light and infrared modalities.

[0092] In step 3.2, the target detection model performs dual-channel feature extraction and outputs feature maps corresponding to the two modalities.

[0093] See also Figure 2 , the dual-channel convolutional neural network (C2 to C5) processes the input visible light image and infrared image respectively, and outputs the feature maps corresponding to the two modalities, the purpose of which is to integrate the features in the visible light image and the infrared image. In the process of extracting features from the two modal images, if no effective features are extracted from the image of one modality (for example, due to insufficient light in the night environment, the visibility of the visible light image taken is low and the target information is unclear), but the image of the other modality happens to provide the corresponding complementary features (because the temperature of the target to be detected at night is higher than the ambient temperature, the target features in the infrared image are more obvious), by integrating the features extracted from the two modalities, more comprehensive and complete target feature information can be obtained.

[0094] Step 3.3: Input the extracted feature maps of the visible light and infrared modes into the strategy module for mode screening.

[0095] For the feature map extracted by the two-way convolutional neural network, the strategy module proposed in the embodiment of the present invention will adaptively select valid modal features to retain and discard invalid modal features. For example, if the feature map corresponding to the input visible light modality does not contain clear and effective target information, while the feature map corresponding to the infrared modality contains clear target information, then after the input two-way feature map passes through the strategy module, the feature information corresponding to the visible light modality will be discarded, while the feature information corresponding to the infrared modality will be retained. It should be noted that the strategy module does not select only one modality feature map every time, and its output may also be all retained or all discarded.

[0096] Step 3.4: Input the filtered feature map into the fusion module to obtain the fused feature map.

[0097] The fusion module used in the embodiment of the present invention identifies the key areas in the feature map through the spatial attention mechanism and reduces the influence of irrelevant areas, thereby effectively highlighting the characteristics of small targets and suppressing background interference, thereby improving the detection performance of small targets. In addition, the fusion module extracts target features of different scales by deploying multiple groups of convolutional layers, achieving better robustness and generalization ability.

[0098] Step 3.5: Input the fused feature map into the subsequent detection head to output the target position result finally predicted by the target detection model.

[0099] Figures 5 to 8 The following sections show some of the detection results of the architecture of the present invention on the test set of the drone crowd detection dataset based on visible light-infrared image pairs, which includes different lighting conditions (such as day and night) and different detection scenarios. The green box in the figure shows the true positive detection result, the red box shows the false positive detection result, and the blue box shows the false negative detection result. According to the results, it is not difficult to find that the algorithm proposed by the architecture of the present invention has high detection accuracy, fewer false detections and missed detections, and has excellent detection performance for multi-scale targets (especially small targets). For example, in Figure 8 In the figure, the architecture of the present invention detects most of the tiny and dense targets, which shows that it has a strong ability to recognize distant tiny targets and partially occluded targets, and is highly robust to background interference.

[0100] The embodiment of the present invention solves the three problems of small scale of the target to be detected, imbalance of modal space and redundancy of modal information in the existing air-to-ground small target detection technology by constructing an air-to-ground small target multimodal detection method based on an adaptive modal selection strategy, and can resist the influence of interference factors such as poor lighting conditions, partial occlusion and overlap. The architecture of the present invention can also be used for other multi-sensor-based visual analysis tasks, and promote high-quality path planning and target tracking for drone-based surveillance, search and rescue, etc.

[0101] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0102] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 Process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0103] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0104] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0105] Those skilled in the art will readily appreciate other embodiments of the present application after considering the description and practicing the contents disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The description and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the claims.

[0106] The above specific implementation modes are used to explain the present invention rather than to limit the present invention. Any modification and change made to the present invention within the spirit of the present invention and the protection scope of the claims shall fall within the protection scope of the present invention.

Claims

1. A multi-modal detection method for small air-to-ground targets based on an adaptive modal selection strategy, characterized in that: The following steps are involved: S1. constructing an initial framework model of a deep learning convolutional network, wherein the initial framework model of the convolutional network includes a dual-path convolutional neural network, a strategy module, a fusion module and a detection head arranged in sequence; S2. Use the training set in the drone crowd detection dataset based on visible light-infrared image pairs to train the initial framework model of the convolutional network to obtain the target detection model; S3. Obtain the image to be detected and input it into the target detection model. The target detection model extracts, filters and fuses the features of the image to be detected, and finally outputs the target position result.

2. The multi-modal detection method for small air-to-ground targets based on an adaptive modality selection strategy according to claim 1 is characterized in that: The dual-path convolutional neural network processes the visible light image and the infrared image respectively, extracts feature maps of the two modalities, and obtains a dual-path feature map.

3. The multi-modal detection method for small air-to-ground targets based on an adaptive modality selection strategy according to claim 1 is characterized in that: The strategy module is used to perform modal screening on the dual-path feature graph extracted by the dual-path convolutional neural network to remove invalid or highly interfering modal information.

4. The multi-modal detection method for small air-to-ground targets based on an adaptive modality selection strategy according to claim 1 is characterized in that: The fusion module is used to integrate the complementary information of the two-way feature map after being filtered by the strategy module.

5. The multi-modal detection method for small air-to-ground targets based on an adaptive modality selection strategy according to claim 1 is characterized in that: The convolutional layer parameters of the dual-path convolutional neural network are initialized using the weights and biases of a residual network model pre-trained on a large-scale image recognition dataset, and other convolutional layer parameters are randomly initialized using a Gaussian normal distribution.

6. The multi-modal detection method for small air-to-ground targets based on an adaptive modality selection strategy according to claim 1 is characterized in that: When the initial framework model of the convolutional network is trained, a strategy of sequentially generating small batches of data is adopted for data input; a gradient descent method is used for parameter update, and gradient clipping is used to adjust the model parameters.

7. The multi-modal detection method for small air-to-ground targets based on an adaptive modality selection strategy according to claim 1 is characterized in that: The fusion module identifies key areas in the feature map through a spatial attention mechanism and reduces the influence of irrelevant areas.

8. An air-to-ground small target multi-modal detection system based on an adaptive mode selection strategy, characterized in that: include: Model construction module: used to construct an initial framework model of a deep learning convolutional network, wherein the initial framework model of the convolutional network includes a dual-path convolutional neural network, a strategy module, a fusion module, and a detection head arranged in sequence; Model training module: used to train the initial framework model of the convolutional network using the training set in the drone crowd detection dataset based on visible light-infrared image pairs to obtain the target detection model; Image detection module: used to obtain the image to be detected and input it into the target detection model. The target detection model is used to extract, filter and fuse the features of the image to be detected, and finally output the target position result.

9. A computer device, characterized in that: The computer device comprises: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the multi-modal detection method for air-to-ground small targets based on an adaptive modal selection strategy as described in any one of claims 1-7.

10. A computer-readable storage medium storing computer instructions, characterized in that: When the computer instructions are executed by one or more processors, the one or more processors are caused to execute the steps in the method according to any one of claims 1 to 7.

Images