Multi-modal target detection method of intelligent reconnaissance equipment

Abstract

The invention relates to the technical field of computer vision, and provides a multi-modal target detection method for intelligent reconnaissance equipment, which comprises the following steps of: acquiring a visible light image, an infrared image and a single photon image of a to-be-detected target main body through the intelligent reconnaissance equipment; inputting the visible light image, the infrared image and the single photon image into a YOLOv5-based multi-modal target detection model to obtain a target detection result; wherein the multi-modal target detection model comprises a first feature extraction module, a second feature extraction module and a third feature extraction module which are respectively used for performing feature extraction on an input visible light image, an input infrared image and an input single-photon image to obtain a visible light feature map, an infrared feature map and a single-photon feature map; the multi-modal target detection model further comprises a fusion module and a head network. And the fusion module is used for fusing the visible light feature map, the infrared feature map and the single photon feature map to obtain a fused feature map, and obtaining the target position and category information of the to-be-detected target body through a head network.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and more specifically, to a multimodal target detection method for intelligent reconnaissance equipment. Background Art

[0002] Intelligent reconnaissance equipment is an important component of optoelectronic reconnaissance and warning technology and its equipment, and is also the core equipment of drone reconnaissance. Target detection is an important task in the field of computer vision research, and is widely used in military, urban security monitoring, smart agriculture and other fields. The edge device of intelligent reconnaissance equipment is equipped with supercomputing modules and various types of cameras and sensors (such as infrared, visible light, single photon, etc.). It uses the multimodal information provided by various sensors to complete real-time target detection tasks in complex environments.

[0003] The existing target detection algorithms based on deep learning have made some progress, but multimodal target detection for edge device scenarios of intelligent reconnaissance equipment still faces greater challenges. The current intelligent reconnaissance equipment is equipped with various types of cameras such as infrared, visible light, and single photon to obtain multimodal images to improve the ability of reconnaissance, but most of the existing target detection algorithms focus on visible light scenes, and the detection effect of directly using different modalities is not good. Secondly, the existing multimodal target detection algorithms are mainly divided into two categories. One type fuses images of different modalities and inputs them into the target detection model, so the features extracted by the model may be insufficient; the other type adds a branch feature extraction module to the target detection model. Although this improves the accuracy of detection, the inference speed of the model also decreases. In addition, the imaging distance of the image collected by intelligent reconnaissance equipment is far, and the targets in the image are small, while the existing multimodal target detection algorithms have poor detection capabilities for long-distance targets and cannot effectively identify these special targets. Summary of the invention

[0004] In order to overcome the defects of the multimodal target detection algorithm described in the above-mentioned prior art, that is, the detection accuracy is low and it is difficult to be deployed on the edge device of the pod, the present invention provides a multimodal target detection method for intelligent reconnaissance equipment.

[0005] In order to solve the above technical problems, the technical solution of the present invention is as follows:

[0006] A multi-modal target detection method for intelligent reconnaissance equipment comprises the following steps:

[0007] Collect visible light images, infrared images and single photon images of the target subject through intelligent reconnaissance equipment;

[0008] Inputting the visible light image, infrared image and single photon image into a multimodal target detection model based on YOLOv5 (You Only LookOnce version Five) to obtain a target detection result;

[0009] The multimodal target detection model includes a first feature extraction module, a second feature extraction module and a third feature extraction module, which are respectively used to extract features from the input visible light image, infrared image and single photon image to obtain a visible light feature map, an infrared feature map and a single photon feature map;

[0010] The multimodal target detection model also includes a fusion module and a head network; the fusion module is used to fuse the visible light feature map, the infrared feature map and the single photon feature map to obtain a fused feature map, and obtain the target position and category information of the target subject to be detected through the head network.

[0011] Furthermore, the present invention also proposes a device, including intelligent reconnaissance equipment, on which a memory and a processor are deployed, and the memory stores computer-readable instructions, wherein when the computer-readable instructions are executed by the processor, the processor performs the steps of the multimodal target detection method proposed in the present invention.

[0012] Furthermore, the present invention also proposes a storage medium having computer-readable instructions stored thereon, wherein the computer-readable instructions, when executed by a processor, implement the steps of the multimodal target detection method proposed in the present invention.

[0013] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0014] The present invention provides an effective real-time multimodal target detection method for intelligent reconnaissance equipment, which is particularly suitable for multimodal target detection of pod edge devices. It can effectively fuse high-resolution multimodal information from infrared, visible light, and single-photon sensors, and at the same time provide more comprehensive target information by combining multimodal data, thereby increasing the accuracy and reliability of target detection of the pod in low-light conditions and under different meteorological conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 The figure is a flow chart of a multimodal target detection method according to an embodiment of the present invention.

[0016] Figure 2 The figure is an architecture diagram of a multimodal target detection model according to one embodiment of the present invention.

[0017] Figure 3 The diagram is an architecture diagram of a backbone network and a neck network according to an embodiment of the present invention.

[0018] Figure 4 The figure is an architecture diagram of a CBAM attention module according to one embodiment of the present invention.

[0019] Figure 5 The figure is an architecture diagram of a channel attention layer according to one embodiment of the present invention.

[0020] Figure 6 An architectural diagram of a spatial attention layer according to one embodiment of the present invention.

[0021] Figure 7 FIG. 4 is an architecture diagram of a CARAFE upsampling operator according to an embodiment of the present invention.

[0022] Figure 8 The figure is an architecture diagram of a fusion module according to an embodiment of the present invention. DETAILED DESCRIPTION

[0023] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.

[0024] The terms used in the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The singular forms "a", "the" and "the" used in the present invention and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0025] It should be understood that although the terms first, second, third, etc. may be used in the present invention to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0026] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0027] Example 1

[0028] This embodiment proposes a multi-modal target detection method for intelligent reconnaissance equipment, such as Figure 1 , which is a flow chart of the multimodal target detection method of this embodiment.

[0029] The multimodal target detection method proposed in this embodiment includes the following steps:

[0030] S1. Collect visible light images, infrared images and single photon images of the target subject through intelligent reconnaissance equipment;

[0031] S2. Inputting the visible light image, infrared image and single photon image into a multimodal target detection model based on YOLOv5 to obtain a target detection result;

[0032] The multimodal target detection model includes a first feature extraction module, a second feature extraction module and a third feature extraction module, which are respectively used to extract features from the input visible light image, infrared image and single photon image to obtain a visible light feature map, an infrared feature map and a single photon feature map;

[0033] The multimodal target detection model also includes a fusion module and a head network; the fusion module is used to fuse the visible light feature map, the infrared feature map and the single photon feature map to obtain a fused feature map, and obtain the target position and category information of the target subject to be detected through the head network.

[0034] like Figure 2 , which is an architecture diagram of the multimodal target detection model of this embodiment.

[0035] In this embodiment, a multimodal target detection model (YOLOv5-RGBTS) is built based on the YOLOv5 network for RGB, infrared and single-photon multimodal target detection. After the visible light image, infrared image and single-photon image of the target subject to be detected collected by the intelligent reconnaissance equipment are input into the multimodal target detection model, feature extraction is performed through the first feature extraction module, the second feature extraction module and the third feature extraction module respectively to obtain a visible light feature map, an infrared feature map and a single-photon feature map, and then the feature maps are fused through the fusion module to obtain a fused feature map, and then the head network is used to perform target detection to obtain the target position and category information of the target subject to be detected.

[0036] This embodiment provides an effective real-time multimodal target detection method for intelligent reconnaissance equipment, which is particularly suitable for multimodal target detection of pod edge devices. It can effectively fuse high-resolution multimodal information from infrared, visible light, and single-photon sensors, and at the same time provide more comprehensive target information by combining multimodal data, thereby increasing the accuracy and reliability of target detection of the pod in low light conditions and under different meteorological conditions.

[0037] Exemplarily, after the camera alignment device in the intelligent reconnaissance equipment collects infrared images, visible light images, and single-photon images, it transmits the images to the edge device in the pod for image processing and multi-mode target detection.

[0038] Exemplarily, the intelligent reconnaissance equipment is equipped with a visible light camera, an infrared camera and a single-photon camera. Among them, the visible light camera can optionally use the MV-CH650-90XM ultra-clear long-distance camera with a resolution of 9216×7000 pixels, which can capture clear, high-quality images at a long distance. The infrared camera can optionally use the Tigris 640 medium-wave infrared camera with a resolution of 640×512, which can provide thermal imaging information of the target. The single-photon camera can optionally use a SPADλ camera, which can achieve 555,000fps photon counting acquisition with zero readout noise. Its global shutter can achieve nanosecond exposure and picosecond gated delay accuracy, and can provide high-sensitivity images under low-light conditions.

[0039] In an optional embodiment, the first feature extraction module, the second feature extraction module, and the third feature extraction module all include a backbone network and a neck network.

[0040] Furthermore, the backbone network is built based on CSPDarkNet (Cross Stage Partial Dark Net), including a convolution module, a C3_CBAM attention module and a SPPF (Spatial PyramidPooling-Fast) module; wherein the output end of the convolution module is connected to the input end of the C3_CBAM attention module, and the last C3_CBAM attention module in the backbone network is connected to an SPPF module.

[0041] The backbone network of this embodiment is built based on CSPDarkNet, and uses the characteristics of cross-stage partial connections and residual connections of CSP DarkNet to enable YOLOv5 to improve the effect of feature extraction and the accuracy of target detection without increasing too much computation.

[0042] Furthermore, the neck network is built based on PAFPN (Path Aggregation Feature Pyramid Network) to enhance the feature fusion effect and improve the detection accuracy and robustness. The neck network includes C3_CBAM attention module, convolution module and upsampling module.

[0043] like Figure 3 The figure shows the architecture diagram of the backbone network and the neck network of this embodiment.

[0044] Furthermore, in an optional embodiment, the C3_CBAM attention module includes a C3 module, and the last convolution module in the C3 module is connected to a CBAM (Convolutional Block Attention Module) module for obtaining more feature information of small targets and improving their detection rate.

[0045] Further, in an optional embodiment, the CBAM module includes a channel attention layer and a spatial attention layer.

[0046] The channel attention layer includes a global maximum pooling layer and a global average pooling layer based on width and height, and a two-layer neural network.

[0047] Exemplarily, in the neural network in the channel attention layer, the number of neurons in the first layer is C / r, the activation function is the Relu function, and the number of neurons in the second layer is C.

[0048] The input feature map is passed through the global maximum pooling layer and the global average pooling layer to obtain two 1×1×C feature maps, and then the channel attention feature is extracted through the neural network. The output feature is then summed based on element perception and then passed through the sigmoid activation function to generate the channel attention feature M. c ; Finally, the channel attention feature M c Perform element-wise multiplication with the input feature to obtain the channel attention feature M c (F) Input the spatial attention layer; its expression is:

[0049]

[0050] Where F is the input feature map, σ(·) represents the sigmoid function, MLP(·) represents the multi-layer perceptron, AvgPool(·) represents the global average pooling operation, MaxPool(·) represents the global maximum pooling layer; W0 represents the hidden layer weight of the multi-layer perceptron, W0 is a two-dimensional vector with C / r rows and C columns, W1 is the output layer weight of MLP, Indicates that W1 is a two-dimensional vector with C rows and C / r columns; the weights W0 and W1 of the MLP are shared, and the ReLU activation function is connected after W0; represents the channel average pooling feature, Represents the channel maximum pooling feature.

[0051] The calculation formula of the Sigmoid function is as follows:

[0052]

[0053] Where x represents the input independent variable.

[0054] The calculation formula of the ReLU function is as follows:

[0055] f(x)=max(0,x)

[0056] Where x represents the input independent variable.

[0057] The spatial attention layer includes a channel-based global maximum pooling layer and a global average pooling layer connected in sequence, and a convolution layer; the input feature map is passed through the channel-based global maximum pooling layer and the global average pooling layer to obtain two H×W×1 feature maps, and then after the channel-based connection operation, the input convolution layer is subjected to a 7×7 convolution operation, the dimension is reduced to one channel, and then the spatial attention feature is generated through sigmoid; its expression is:

[0058]

[0059] Where σ(·) represents the sigmoid function; f 7×7 (·) represents a 7×7 convolution operation; They represent the spatial average pooling features and the spatial maximum pooling features respectively.

[0060] Finally, the channel attention feature M c (F) and spatial attention feature M s (F) Perform multiplication to obtain the final generated feature map.

[0061] For example, Figures 4 to 6 As shown, it is the architecture diagram of the CBAM attention module, channel attention layer and spatial attention layer of this embodiment.

[0062] In this embodiment, the CBAM module adjusts the feature map through the channel attention layer and the spatial attention layer, so that the network can better handle targets of various sizes.

[0063] Furthermore, in an optional embodiment, the neck network adopts a PAFPN structure, including a C3_CBAM module, a convolution module and an upsampling module.

[0064] The neck network is used to upsample and fuse the feature maps of the 3rd to 5th layers output by the backbone network from bottom to top to obtain multi-scale visible light feature maps, infrared feature maps and single photon feature maps.

[0065] Furthermore, in an optional embodiment, in order to better preserve and improve the quality of features, the upsampling module includes a CARAFE (Content-Aware Relevance and Feature Enrichment) upsampling operator.

[0066] For example, Figure 7 , which is an architecture diagram of the CARAFE upsampling operator of this embodiment.

[0067] The up-sampling module includes the following steps when performing the up-sampling operation:

[0068] For an input feature map of size H×W×C, its channel number is compressed to H×W×C through a 1×1 convolution m ; where C m <C;

[0069] Using a k encoder ×k encoder The convolution layer performs upsampling kernel prediction; its input channel is C m , the number of output channels is Then expand the channel dimension in the space dimension to get the shape of The upsampling kernel of

[0070] Normalizing the upsampling kernel so that the sum of its convolution kernel weights is 1;

[0071] Feature reorganization is performed based on the normalized upsampling kernel, where for any feature point in the output feature map, it is mapped back to the input feature map, and k features centered at the feature point are taken out. up ×k up , and dot product the region with the normalized upsampling kernel to obtain an upsampling operation output value.

[0072] Exemplarily, when the upsampling kernel is normalized, the Softmax function is used for normalization so that the sum of the convolution kernel weights is 1. The calculation formula of the Softmax function is as follows:

[0073]

[0074] In this embodiment, a lightweight CARAFE is used as an upsampling operator, and the input feature map is used to predict the upsampling kernel at each position. These upsampling kernels are then used to reorganize the feature map. By utilizing its larger receptive field, the surrounding information can be better utilized, thereby improving the upsampling effect, as well as improving the resolution of the feature map and the performance of the model.

[0075] In an optional embodiment, the fusion module includes a feature weight network, which is used to calculate the visible light weight, infrared weight and single photon weight of the visible light feature map, infrared feature map and single photon feature map respectively through a multi-layer convolutional neural network, and then convolve and fuse the weight with the corresponding feature map to obtain a fused feature map. The expression of feature fusion is:

[0076] F fusion =F RGB *W RGB +F IR *W IR +F S *W S

[0077] [W RGB ,W IR ,W S ]=FWN([F RGB ,F IR ,F S ])

[0078] Among them, F RGB 、F IR 、F S Respectively represent the visible light feature map, infrared feature map and single photon feature map; W RGB , W IR , W S denote visible light weight, infrared weight and single photon weight respectively; FWN(·) denotes feature weight network.

[0079] For example, Figure 8 The figure shows the architecture diagram of the fusion module of this embodiment.

[0080] The head network of this embodiment outputs predicted target information and outputs two three-dimensional tensors, each of which contains the position, confidence and target category of the detected target.

[0081] In an optional embodiment, the method further comprises the following steps: pre-training the multimodal target detection model based on YOLOv5 using distillation learning; the steps include:

[0082] The multimodal target detection model based on YOLOv5l is used as the teacher model, and the multimodal target detection model based on YOLOv5s is used as the student model;

[0083] Inputting the training data set into the teacher model and the student model at the same time and performing forward propagation respectively to obtain the output of the teacher model and the student model; the output includes a predicted bounding box and a category probability of each predicted bounding box;

[0084] According to the outputs of the teacher model and the student model, the classification loss, the bounding box regression loss and the object confidence loss are calculated, and the total distillation loss function is calculated by objectness scaled distiallation. Then, according to the total distillation loss function, the parameters of the student model are optimized by back propagation; wherein, the total distillation loss function L final The expression is:

[0085]

[0086] In the formula, represents the bounding box regression loss, represents the classification loss, represents the target confidence loss; and They represent the objectness value (the score of the predicted bounding box covering the target), the category probability, and the true value of the bounding box position, respectively. and Represents the predicted values ​​of target values, class probabilities, and bounding box locations output by the student model; and The predicted values ​​of target value, category probability and bounding box position output by the teacher model; λ D is the distillation temperature; represents the loss of the student model prediction, which is weighted by the true value and the target value of the student model prediction result; represents the distillation loss, which is weighted by the target value of the teacher model prediction result and the target value of the student model prediction result;

[0087] The student model that completes the distillation learning is used as the multimodal target detection model.

[0088] It should be noted that YOLOv5l is the larger model in the YOLOv5 series, with relatively high accuracy, but slow detection speed and requires devices with strong computing power and resources; while YOLOv5s is the smallest model in the YOLOv5 series, which performs best on devices with limited computing resources in the pod and has the fastest detection speed, but relatively low accuracy. In addition, YOLOv5l and YOLOv5s have the same network structure, but different model sizes and complexities.

[0089] In order to complete high-accuracy real-time target detection on the pod edge computing device, this embodiment uses YOLOv5-RGBTS built based on YOLOv5l as the teacher model, and YOLOv5-RGBTS built based on YOLOv5s as the student model, and then uses distillation learning to transfer the knowledge learned by the teacher model to the student model.

[0090] Exemplarily, the trained student model is deployed on the pod edge device as the .pt file of the multimodal target detection model to achieve high-accuracy real-time target detection on the pod edge computing device.

[0091] This embodiment uses a pre-made multimodal image dataset as a training dataset and uses a teacher model to perform distillation learning training on the student model.

[0092] Furthermore, in an optional embodiment, the training data set is prepared as follows:

[0093] The infrared images, visible light images and single photon images obtained by taking photos of different detection subjects at different time periods, locations and scenes using an infrared camera, a visible light camera and a single photon camera constitute an image set; wherein the scene includes at least one of low light illumination, occlusion, rain and fog, and camouflage;

[0094] Performing adaptive image segmentation on the image set, normalizing the size of the segmented images to the output size of the multimodal target detection model, and obtaining a preprocessed image set;

[0095] The preprocessed image set is annotated with target categories and information locations to obtain the training data set.

[0096] This embodiment uses the distillation learning method to compress the complex multimodal target detection model into a lighter version, making the model smaller and faster inference, reducing the demand for computing and storage resources. It is suitable for environments with limited resources of pod edge devices and can give full play to the computing power of edge computing devices to implement multimodal target detection.

[0097] Example 2

[0098] This embodiment proposes a device, including intelligent reconnaissance equipment, on which a memory and a processor are deployed, and the memory stores computer-readable instructions. When the computer-readable instructions are executed by the processor, the processor executes all or part of the steps of the multimodal target detection method described in Example 1.

[0099] It can be understood that the device of this embodiment corresponds to the method of the above-mentioned embodiment 1, and the options in the above-mentioned embodiment 1 are also applicable to this embodiment, so they are not described repeatedly here.

[0100] Example 3

[0101] This embodiment proposes a storage medium on which computer-readable instructions are stored, wherein the computer-readable instructions, when executed by a processor, implement all or part of the steps of the multimodal target detection method proposed in Embodiment 1.

[0102] Exemplarily, the storage medium includes, but is not limited to, a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0103] Exemplarily, the instructions, programs, code sets or instruction sets may be implemented using conventional programming languages.

[0104] Exemplarily, the processor includes but is not limited to a smart phone, a personal computer, a server, a network device, etc., and is used to execute all or part of the steps of the multimodal target detection method described in Example 1.

[0105] The terms in the drawings are only used for illustrative purposes and are not to be construed as limiting the present invention;

[0106] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A multimodal target detection method for intelligent reconnaissance equipment, characterized in that: The following steps are involved: Collect visible light images, infrared images and single photon images of the target subject through intelligent reconnaissance equipment; Inputting the visible light image, infrared image and single photon image into a multimodal target detection model based on YOLOv5 to obtain a target detection result; The multimodal target detection model includes a first feature extraction module, a second feature extraction module and a third feature extraction module, which are respectively used to extract features from the input visible light image, infrared image and single photon image to obtain a visible light feature map, an infrared feature map and a single photon feature map; The multimodal target detection model also includes a fusion module and a head network; the fusion module is used to fuse the visible light feature map, the infrared feature map and the single photon feature map to obtain a fused feature map, and obtain the target position and category information of the target subject to be detected through the head network.

2. The multimodal target detection method according to claim 1, characterized in that: The first feature extraction module, the second feature extraction module, and the third feature extraction module all include a backbone network and a neck network; The backbone network is built based on CSPDarkNet, including a convolution module, a C3_CBAM attention module and an SPPF module; wherein the output end of the convolution module is connected to the input end of the C3_CBAM attention module, and the last C3_CBAM attention module in the backbone network is connected to an SPPF module; The neck network is built based on PAFPN, including a C3_CBAM attention module, a convolution module and an upsampling module.

3. The multimodal target detection method according to claim 2, characterized in that: The C3_CBAM attention module includes a C3 module, and the last convolution module in the C3 module is connected to a CBAM module; the CBAM module includes a channel attention layer and a spatial attention layer; The channel attention layer includes a global maximum pooling layer and a global average pooling layer based on width and height, and a two-layer neural network; the input feature map is respectively passed through the global maximum pooling layer and the global average pooling layer to obtain two 1×1×C feature maps, and then the channel attention feature is extracted through the neural network, and then the output feature is added based on element perception, and then passed through the sigmoid activation function to generate the channel attention feature M c ; Finally, the channel attention feature M c Perform element-wise multiplication with the input feature to obtain the channel attention feature M c (F) Input the spatial attention layer; its expression is: Where F is the input feature map, σ(·) represents the sigmoid function, MLP(·) represents the multi-layer perceptron, AvgPool(·) represents the global average pooling operation, MaxPool(·) represents the global maximum pooling layer; W0 represents the hidden layer weight of the multi-layer perceptron, W0 is a two-dimensional vector with C / r rows and C columns, W1 is the output layer weight of MLP, Indicates that W1 is a two-dimensional vector with C rows and C / r columns; the weights W0 and W1 of the MLP are shared, and the ReLU activation function is connected after W0; represents the channel average pooling feature, Represents the channel maximum pooling feature; The spatial attention layer includes a channel-based global maximum pooling layer and a global average pooling layer connected in sequence, and a convolution layer; the input feature map is passed through the channel-based global maximum pooling layer and the global average pooling layer to obtain two H×W×1 feature maps, and then after the channel-based connection operation, the input convolution layer is subjected to a 7×7 convolution operation, the dimension is reduced to one channel, and then the spatial attention feature is generated through sigmoid; its expression is: Where σ(·) represents the sigmoid function; f 7×7 (·) represents a 7×7 convolution operation; They represent the spatial average pooling features and the spatial maximum pooling features respectively.

4. The multimodal target detection method according to claim 2, characterized in that: The neck network adopts a path aggregation feature pyramid network structure, including a C3_CBAM module, a convolution module and an upsampling module; the neck network is used to upsample and fuse the feature maps of the 3rd to 5th layers output by the backbone network from bottom to top to obtain multi-scale visible light feature maps, infrared feature maps and single photon feature maps.

5. The multimodal target detection method according to claim 4, characterized in that: The upsampling module includes content-aware relevance and feature-enriching upsampling operators; wherein the upsampling module includes the following steps when performing the upsampling operation: For an input feature map of size H×W×C, its channel number is compressed to H×W×C through a 1×1 convolution m ; where C m <C; Using a k encoder ×k encoder The convolution layer performs upsampling kernel prediction; its input channel is C m , the number of output channels is Then expand the channel dimension in the space dimension to get the shape of The upsampling kernel of Normalizing the upsampling kernel so that the sum of its convolution kernel weights is 1; Feature reorganization is performed based on the normalized upsampling kernel, where for any feature point in the output feature map, it is mapped back to the input feature map, and k features centered at the feature point are taken out. up ×k up , and dot product the region with the normalized upsampling kernel to obtain an upsampling operation output value.

6. The multimodal target detection method according to claim 1, characterized in that: The fusion module includes a feature weight network, which is used to calculate the visible light weight, infrared weight and single photon weight of the visible light feature map, infrared feature map and single photon feature map respectively through a multi-layer convolutional neural network, and then convolve and fuse the weights with the corresponding feature maps to obtain a fused feature map; Its expression is: F fusion =F RGB *W RGB +F IR *W IR +F S *W S [W RGB ,W IR ,W S ]=FWN([F RGB ,F IR ,F S ]) Among them, F RGB 、F IR 、F S Respectively represent the visible light feature map, infrared feature map and single photon feature map; W RGB , W IR , W S denote visible light weight, infrared weight and single photon weight respectively; FWN(·) denotes feature weight network.

7. The multimodal target detection method according to any one of claims 1 to 6, characterized in that: The method further comprises the following steps: pre-training the multimodal target detection model based on YOLOv5 by using distillation learning; the steps include: The multimodal target detection model based on YOLOv5l is used as the teacher model, and the multimodal target detection model based on YOLOv5s is used as the student model; Inputting the training data set into the teacher model and the student model at the same time and performing forward propagation respectively to obtain the output of the teacher model and the student model; the output includes a predicted bounding box and a category probability of each predicted bounding box; The classification loss, bounding box regression loss and target confidence loss are calculated according to the outputs of the teacher model and the student model, and the total distillation loss function is calculated by target weighted distillation. Then, according to the total distillation loss function, the parameters of the student model are optimized by back propagation; wherein the total distillation loss function L final The expression is: In the formula, represents the bounding box regression loss, represents the classification loss, represents the target confidence loss; and Represent the true value of the target value, category probability and bounding box position respectively, and Represents the predicted values ​​of target values, class probabilities, and bounding box locations output by the student model; and The predicted values ​​of target value, category probability and bounding box position output by the teacher model; λ D is the distillation temperature; represents the loss of the student model prediction, which is weighted by the true value and the target value of the student model prediction result; represents the distillation loss, which is weighted by the target value of the teacher model prediction result and the target value of the student model prediction result; The student model that completes the distillation learning is used as the multimodal target detection model.

8. The multimodal target detection method according to claim 7, characterized in that: The training data set includes: The infrared images, visible light images and single photon images obtained by taking photos of different detection subjects at different time periods, locations and scenes using an infrared camera, a visible light camera and a single photon camera constitute an image set; wherein the scene includes at least one of low light illumination, occlusion, rain and fog, and camouflage; Performing adaptive image segmentation on the image set, normalizing the size of the segmented images to the output size of the multimodal target detection model, and obtaining a preprocessed image set; The preprocessed image set is annotated with target categories and information locations to obtain the training data set.

9. A device, comprising intelligent reconnaissance equipment, characterized in that: The intelligent reconnaissance equipment is deployed with a memory and a processor, the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor executes all or part of the steps of the multimodal target detection method according to any one of claims 1 to 8.

10. A storage medium having computer-readable instructions stored thereon, characterized in that: When the computer-readable instructions are executed by a processor, all or part of the steps of the multimodal target detection method according to any one of claims 1 to 8 are implemented.

Images