A cross-modal target detection system and method based on infrared and visible light fusion of DUMYOLO

Abstract

The application provides an infrared and visible light fusion cross-modal target detection system and method based on DUMYOLO, comprising: a video acquisition module, which is used for acquiring continuous infrared video stream and visible light video stream of a detection area to be detected, and performing frame sampling and image preprocessing on the infrared video stream and the visible light video stream; and a target detection module, which is used for performing target detection on the processed infrared video stream and visible light video stream by using a target detection model based on a DUMYOLO network to obtain a target detection result containing target position information and a category label. The technical scheme of the application can be widely used in various complex target detection scenes such as automatic driving, intelligent security, industrial manufacturing, aerospace and the like.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to a cross-modal target detection system and method based on DUMYOLO for infrared and visible light fusion. Background Technology

[0002] Object detection, a core technology in computer vision, plays a crucial role in numerous fields, including autonomous driving, intelligent security, industrial manufacturing, and aerospace. Visible light images have consistently been a primary research focus in object detection. With the continuous advancement of deep learning technology, significant progress has been made in object detection. However, due to the limitations of the visible light wavelength range, in complex environments (such as insufficient lighting, fog, and smoke), visible light-based object detectors struggle to acquire effective object information. Based on these challenges, infrared images have begun to be incorporated into object detection research. Unlike visible light images, infrared images accurately capture object information by capturing the thermal radiation of objects, thus reducing or even eliminating the impact of complex environments on object detection. However, because infrared images lack information such as object color and texture, accurate object detection is difficult to achieve in certain specific scenarios (such as distinguishing similarly shaped car models). To achieve better performance in object detection, increasing research is integrating the complementary characteristics of infrared and visible light to acquire more object information. Given these differences in characteristics and technological complementarity, cross-modal object detection technology combining infrared and visible light has gradually become a research hotspot in recent years.

[0003] Existing infrared and visible light cross-modal target detection methods still have the following problems:

[0004] Their model architecture is too cumbersome. For example, the module for infrared and visible light mode fusion and the module for target detection are two separate models. This redundant model architecture leads to an increase in computation and consequently an incalculable training cost.

[0005] The problem of mutual interference between modes is that the fusion of infrared and visible light modes is intended to make full use of the texture, color and other information of the visible light mode and the structural information of the infrared mode. However, directly fusing the information of infrared and visible light modes will cause the visible light mode to inject irrelevant interference information into the infrared mode, resulting in feature confusion and thus reducing the performance of target detection.

[0006] In the feature extraction stage, as the network layers deepen, the low-resolution semantic information of the deep network becomes increasingly mismatched with the high-resolution spatial information. This makes it difficult to retain the low-level spatial information in the deep network, resulting in feature mismatch and hindering multimodal fusion. Summary of the Invention

[0007] To address the problems existing in the prior art, this invention proposes a cross-modal target detection system and method based on DUMYOLO infrared and visible light fusion, which can be widely used in various complex target detection scenarios such as autonomous driving, intelligent security, industrial manufacturing, and aerospace.

[0008] To achieve the above objectives, the present invention provides the following solution:

[0009] A cross-modal target detection system based on DUMYOLO infrared and visible light fusion includes:

[0010] The video acquisition module is used to acquire continuous infrared video streams and visible light video streams of the area to be detected, and to perform frame sampling and image preprocessing on the infrared video streams and visible light video streams.

[0011] The target detection module is used to perform target detection on the processed infrared video stream and visible light video stream using a target detection model based on the DUMYOLO network, and obtain target detection results containing target location information and category labels.

[0012] Preferably, in the target detection module, the target detection model includes: a backbone network, a neck network, and a detection head;

[0013] The backbone network is used to extract and fuse multimodal features from the processed infrared video stream and the image sequence in the visible light video stream to obtain fused multimodal features.

[0014] The neck network is used to enhance the fused multimodal features, and to fuse the multimodal features and the enhanced multimodal features at multiple scales to obtain a multiscale fused feature map, and to filter the effective modal information in the multiscale fused feature map that meets preset requirements.

[0015] The detection head is used to output a target detection result containing target location information and category label based on the effective modal information.

[0016] Preferably, the backbone network sequence includes: convolutional layers, a first feature interaction coupling module, a first C3k2 module, a second feature interaction coupling module, a second C3k2 module, a third feature interaction coupling module, a first A2C2f module, a fourth feature interaction coupling module, and a second A2C2f module; wherein,

[0017] The convolutional layer is used to extract multimodal features from the image sequence and adjust the dimension and spatial size of the multimodal features;

[0018] Both the first C3k2 module and the second C3k2 module are used to refine the multimodal features into discriminative features;

[0019] The first feature interaction coupling module, the second feature interaction coupling module, the third feature interaction coupling module, and the fourth feature interaction coupling module are all used to encode the multimodal features spatial information, fuse spatial information and semantic information, and obtain the fused multimodal feature map.

[0020] Both the first A2C2f module and the second A2C2f module incorporate an attention mechanism to capture target features from the fused multimodal feature map.

[0021] Preferably, the neck network includes:

[0022] A U-shaped multi-kernel perceptron is used to perform multi-scale feature extraction, convolution operation, and edge information enhancement processing on the multimodal feature map to obtain an enhanced multimodal feature map. The Neck network is then used to fuse the multimodal feature map and the enhanced multimodal feature map at multiple scales to obtain a fused multi-scale feature map.

[0023] A multispectral cross-domain collaborative unit is used to perform soft attention selection on the fused multi-scale feature map, capture global context information, suppress irrelevant modal information that does not meet preset requirements, and obtain effective modal information that meets preset requirements.

[0024] Preferably, the U-shaped multi-core perceptron includes five convolutions with kernel sizes of 3, 1, 5, 1, and 7, respectively, and each pair of convolutions is connected by residuals to form a U-shaped structure.

[0025] Preferably, the multispectral cross-domain collaborative unit includes three branches:

[0026] The first branch consists of parallel global pooling layers, average pooling layers, softmax normalization layers, and 1×1 convolutional layers, which are used to model global spatial context information.

[0027] The second branch includes parallel 1×1 convolutions, softmax operations, and cross product operations, used to extract discriminative features from the original image sequence;

[0028] The third branch includes a pooling downsampling layer and introduces a non-linear attention layer for image compression and feature selection to obtain effective modal information that meets preset requirements.

[0029] This invention also provides a cross-modal target detection method based on DUMYOLO infrared and visible light fusion, for implementing the system, comprising:

[0030] Continuous infrared video streams and visible light video streams of the area to be detected are acquired, and frame sampling and image preprocessing are performed on the infrared video streams and visible light video streams.

[0031] A target detection model based on the DUMYOLO network is used to perform target detection on the processed infrared video stream and visible light video stream, and the target detection results containing target location information and category labels are obtained.

[0032] Preferably, the target detection model includes: a backbone network, a neck network, and a detection head;

[0033] The backbone network is used to extract and fuse multimodal features from the processed infrared video stream and the image sequence in the visible light video stream to obtain fused multimodal features.

[0034] The neck network is used to enhance the fused multimodal features, and to fuse the multimodal features and the enhanced multimodal features at multiple scales to obtain a multiscale fused feature map, and to filter the effective modal information in the multiscale fused feature map that meets preset requirements.

[0035] The detection head is used to output a target detection result containing target location information and category label based on the effective modal information.

[0036] Compared with existing technologies, the beneficial effects of this invention are as follows: The technical solution of this invention reconstructs the backbone network by introducing a feature interaction coupling module. This module reduces network loss during deep integration by encoding spatial information, promoting the integration of spatial and semantic information. Applying the feature interaction coupling module at different stages alleviates the feature mismatch problem in the backbone network. A U-shaped multi-core perceptron is used to enhance the features of the fused multimodal information, better extracting fine-grained information from the feature map and providing more reliable feature information for subsequent soft attention operations. Multispectral cross-domain collaborative units are used for feature selection, emphasizing useful information and suppressing irrelevant information, thus solving the problem of interference between modalities. DUMYOLO can be widely used in various complex target detection scenarios such as autonomous driving, intelligent security, industrial manufacturing, and aerospace. Attached Figure Description

[0037] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This is a flowchart of the cross-modal target detection method based on infrared and visible light fusion according to an embodiment of the present invention;

[0039] Figure 2This is a schematic diagram of the target detection model structure according to an embodiment of the present invention;

[0040] Figure 3 This is a schematic diagram of the backbone network structure according to an embodiment of the present invention;

[0041] Figure 4 This is a flowchart illustrating the workflow of the feature interaction coupling module in an embodiment of the present invention.

[0042] Figure 5 This is a flowchart illustrating the operation of the U-shaped multi-core sensor according to an embodiment of the present invention.

[0043] Figure 6 This is a flowchart illustrating the workflow of the multispectral cross-domain collaborative unit in an embodiment of the present invention.

[0044] Figure 7 This is a schematic diagram of the complete hardware structure of an embodiment of the present invention;

[0045] Figure 8 This is a schematic diagram illustrating target detection using a drone according to an embodiment of the present invention;

[0046] Figure 9 This is a schematic diagram illustrating target detection using a robot according to an embodiment of the present invention;

[0047] Figure 10 This is a schematic diagram illustrating target detection using an unmanned vehicle, as an embodiment of the present invention. Detailed Implementation

[0048] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0049] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0050] Example 1

[0051] A cross-modal target detection system based on DUMYOLO, integrating infrared and visible light, is deployed on platforms such as drones, unmanned vehicles, and robots, including:

[0052] The video acquisition module is used to acquire continuous infrared and visible light video streams of the area to be detected, and to perform frame sampling and image preprocessing on the infrared and visible light video streams. Specifically, a fixed frame rate sampling operation is performed on the continuous video stream, and then the resolution of the sampled continuous video stream is uniformly adjusted to obtain a standardized image sequence. In this embodiment, cameras from platforms such as drones, unmanned vehicles, and robots are used to capture continuous video streams of the area to be detected, and frame sampling and image preprocessing are performed on the infrared and visible light video streams. The infrared video stream frame sampling uses a fixed frequency of 30fps, and the image preprocessing uniformly adjusts all image sequences to a resolution of 640×512. The visible light video stream sampling uses a fixed frequency of 60fps, and the image preprocessing uniformly adjusts all image sequences to a resolution of 1080×768, and normalizes the pixel values ​​to the [0,1] range.

[0053] The target detection module is used to perform target detection on the processed infrared video stream and visible light video stream using a target detection model based on the DUMYOLO network, and obtain target detection results containing target location information and category labels.

[0054] A further implementation method is, such as Figure 2 As shown, the target detection module includes a target detection model comprising a backbone network, a neck network, and a detection head.

[0055] The backbone network is used to extract and fuse multimodal features from image sequences in the processed infrared video stream and visible light video stream to obtain fused multimodal features.

[0056] A further implementation method involves using, for both infrared and visible light modes, the feature extraction stage as follows: Figure 3 The backbone network shown includes the following modules in the following order: convolutional layer, first feature interaction coupling module, first C3k2 module, second feature interaction coupling module, second C3k2 module, third feature interaction coupling module, first A2C2f module, fourth feature interaction coupling module, and second A2C2f module. In this embodiment, the feature extraction backbone network for infrared and visible light modes first downsamples the input image to half its size using a convolutional module (Conv). Then, it passes through a feature interaction coupling module and a C3k2 module twice to obtain a set of infrared and visible light feature maps. Finally, it passes through a feature interaction coupling module and an A2C2f module twice to obtain two more sets of infrared and visible light feature maps.

[0057] Convolutional layers are used to extract multimodal features from image sequences and adjust the dimensionality and spatial size of the multimodal features;

[0058] Both the first C3k2 module and the second C3k2 module are used to refine multimodal features into discriminative features;

[0059] The first feature interaction coupling module, the second feature interaction coupling module, the third feature interaction coupling module, and the fourth feature interaction coupling module are all used to encode the spatial information of multimodal features, fuse spatial information and semantic information, and obtain a fused multimodal feature map to solve the feature mismatch problem.

[0060] Specifically, such as Figure 4 As shown, the feature interaction coupling module includes: an SPD depth conversion unit and a spatial information guidance unit;

[0061] The SPD depth transformation unit converts spatial information of features into depth information, thereby reducing the loss of spatial information. The spatial information guidance unit implicitly encodes the depth information generated by the SPD depth transformation unit into a high-level vector, guiding the fusion of spatial and semantic information. The feature interaction coupling module guides the combination of spatial and semantic information using the following method:

[0062] f 0,0 =X[0:S:2,0:S:2],f 1,0 =X[1:S:2,0:S:2],

[0063] f 0,1 =X[0:S:2,1:S:2],f 1,1 =X[1:S:2,1:S:2],

[0064] f input =Concat(f 0,0 ,f 1,0 ,f 0,1 ,f 1,1 )

[0065] (X1,X2)=Split(f input )

[0066]

[0067] ω2=R(F(X S ))

[0068]

[0069] Where f 0,0 ,f 1,0 ,f 0,1 ,f 1,1 f represents the sub-feature map after SPD segmentation. input This is the spliced ​​feature map. Representing the mapping relationship between spatial and semantic information, X1 consists of branches composed of standard 3×3 convolutions, extracting richer feature information in each channel, denoted as X. C X2, composed of pointwise convolutions, extracts relatively weak information while retaining a large amount of shallow spatial location information, denoted as X. S , k represents the mapping of each feature layer channel. i For the i-th convolutional kernel, This represents the i-th input channel. For each corresponding single output channel, R represents the activation function, and F represents the convolutional mapping with spatial clustering. This represents element-wise multiplication. X represents the feature map of the original input, ω1 represents the channel information weights, ω2 represents the spatial information weights, and X... input X represents the depth information generated by the SPD depth conversion unit. output H represents the final output of the feature interaction coupling module, where H represents the height of the feature map and W represents the width of the feature map.

[0070] Both the first A2C2f module and the second A2C2f module incorporate an attention mechanism to capture target features from the fused multimodal feature map.

[0071] The neck network is used to enhance the fused multimodal features and fuse the multimodal features and enhanced multimodal features at multiple scales to obtain a multiscale fused feature map. Valid modal information that meets preset requirements is then selected from the multiscale fused feature map.

[0072] A further implementation method is, such as Figure 5 As shown, the neck network includes a U-shaped multi-kernel perceptron, which consists of multiple convolutions of different kernel sizes, pointwise convolutions, and multiple skip connections. Convolutional kernels of various scales are sequentially connected to detect targets of different scales. Specifically, multi-scale feature extraction, convolution operations, and edge information enhancement processing are performed on the multimodal feature map to obtain an enhanced multimodal feature map. The Neck network is then used to fuse the multimodal feature map and the enhanced multimodal feature map at multiple scales to obtain a fused multimodal feature map. In this embodiment, the feature maps of layers 5, 7, and 10 of the two branches of the feature extraction stage are concatenated. After the concatenated layers 5 and 7, a U-shaped multi-kernel perceptron is added to perform multi-scale feature extraction on the fused features. After the feature maps of the three levels are sampled by the Neck layer, a multispectral cross-domain collaborative unit is used to select features, enhance the required features, and suppress useless features.

[0073] A further implementation involves a U-shaped multi-kernel perceptron comprising five convolutions with kernel sizes of 3, 1, 5, 1, and 7, respectively. Each pair of convolutions is connected using residual connections to form a U-shaped structure. The U-shaped multi-kernel perceptron enhances features as follows:

[0074] X'=C 2k+1 (A(…A(C k (X rgb+ir ))+C k (X rgb+ir )…))+X rgb+ir ,

[0075] Where X' represents the feature processed by the U-shaped multi-core perceptron, X rgb+ir C represents the characteristics of the input infrared and visible light after fusion. k This represents a convolution with a kernel size of k, and A represents a pointwise convolution transformation.

[0076] Multispectral cross-domain collaborative units are used to perform soft attention selection on fused multi-scale feature maps, capture global contextual information, suppress irrelevant modal information that does not meet preset requirements, and obtain effective modal information that meets preset requirements. For example... Figure 6 As shown.

[0077] A further implementation method is that the multispectral cross-domain collaborative unit includes three branches:

[0078] The first branch consists of parallel global pooling layers, average pooling layers, softmax normalization layers, and 1×1 convolutional layers, used to model global spatial context information. It emphasizes the difference between the target and irrelevant information through normalization operations and element-wise multiplication. The method of the first branch is as follows:

[0079] Q left =Conv1×1(P)⊙Softmax(Conv 1×1 ([GMP(P);GAP(P)]))

[0080] Where P represents the feature input to the model, P∈R C×H×W Where C is the number of channels, H and W are the height and width of the feature map, and Q is the width of the feature map. left This represents the feature after processing through the first branch. ⊙ represents the Hadamard product, Softmax represents Softmax normalization, GMP represents global max pooling, and GAP represents global average pooling.

[0081] The second branch includes parallel 1×1 convolutions, softmax operations, and cross product operations to extract discriminative features from the original image sequences, reducing the computational cost of the model while avoiding excessive redundant information from interfering with subsequent processing. The methods of the two branches are as follows:

[0082] Q mid =Conv 1×1 (P)⊙Softmax(Conv 1×1 (P))

[0083] Q mid This represents the features of the feature map after processing through the second branch.

[0084] The third branch includes a pooling downsampling layer and introduces a non-linear attention layer for image compression and feature selection to obtain effective modal information that meets preset requirements. Specifically, the third branch uses pooling downsampling to compress the image size to retain key feature information and introduces a non-linear attention operation to increase the model's ability to fit complex relationships. The method of the third branch is as follows:

[0085] Q right =Expand(Relu(Conv) 1×1 (GMP(P)⊕GAP(P))))

[0086] Q right This represents the features of the feature map after processing by the third branch. Expand indicates the dimensionality expansion operation. In this branch, the initial feature dimension is R. C×1×1 After the Expand operation, the feature dimension is expanded to R. C×H×W ⊕ indicates element-wise addition, and Relu represents the Relu activation function.

[0087] The multispectral cross-domain cooperative unit method can be expressed as:

[0088]

[0089] Here, Q represents the output of the feature map after being processed by the three branches.

[0090] The detection head is used to output target detection results, including target location information and category labels, based on valid modal information.

[0091] In this embodiment, as Figure 7 As shown, the system also includes a device mounting module, an information acquisition module, a lighting auxiliary module, a power supply module, a data processing module, an information storage module, an edge computing module, and a communication module.

[0092] The equipment mounting module is located at the bottom of the drone platform, the top of the unmanned vehicle platform, and the head of the robot. It connects the information acquisition module and the lighting assistance module through a three-axis mechanical stabilization gimbal, aiming to reduce the negative impact on imaging quality during movement.

[0093] The information acquisition module includes a binocular visible light camera and an infrared thermal imager, enabling multimodal imaging and distance perception of the target scene.

[0094] The lighting auxiliary module features a small night light with IPX4 waterproof rating. It is mounted on both sides of a three-axis mechanical stabilization gimbal and is used to provide high-brightness LED supplemental lighting at night or in low-light environments.

[0095] Power modules, data processing modules, edge computing modules, information storage modules, and communication modules are placed inside drones, unmanned vehicles, robots, etc.

[0096] The power module provides power to all functional modules to ensure stable operation.

[0097] The data processing module encapsulates the detection results in a structured manner, generating encapsulated data with timestamps, corresponding video frame identifiers, target location information, and category information. This encapsulated data is then transmitted to the communication module for real-time image transmission and to the information storage module for historical record saving.

[0098] The edge computing module uses the K3588 embedded computing unit, which internally deploys the DUMYOLO model (object detection module) to achieve real-time end-to-end object detection.

[0099] The information storage module is used to store algorithm models and model parameters, and to save detection logs, detection results and image data;

[0100] The communication module is used to send identification information to the user terminal in real time.

[0101] Figure 8 This is a schematic diagram illustrating the use of drones for target detection.

[0102] Figure 9 This is a schematic diagram illustrating the use of robots for object detection.

[0103] Figure 10 This is a schematic diagram illustrating the use of unmanned vehicles for target detection.

[0104] Example 2

[0105] like Figure 1 As shown, the present invention also provides a cross-modal target detection method based on DUMYOLO infrared and visible light fusion, for implementing a system, comprising:

[0106] Continuous infrared and visible light video streams of the area to be detected are acquired, and frame sampling and image preprocessing are performed on the infrared and visible light video streams.

[0107] A target detection model based on the DUMYOLO network is used to perform target detection on the processed infrared video stream and visible light video stream, and the target detection results containing target location information and category labels are obtained.

[0108] A further implementation method is that the target detection model includes: a backbone network, a neck network, and a detection head;

[0109] The backbone network is used to extract and fuse multimodal features from image sequences in the processed infrared video stream and visible light video stream to obtain fused multimodal features.

[0110] The neck network is used to enhance the fused multimodal features and fuse the multimodal features and enhanced multimodal features at multiple scales to obtain a multiscale fused feature map. Valid modal information that meets preset requirements is then selected from the multiscale fused feature map.

[0111] The detection head is used to output target detection results, including target location information and category labels, based on valid modal information.

[0112] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A cross-modal target detection system based on DUMYOLO infrared and visible light fusion, characterized in that, include: The video acquisition module is used to acquire continuous infrared video streams and visible light video streams of the area to be detected, and to perform frame sampling and image preprocessing on the infrared video streams and visible light video streams. The target detection module is used to perform target detection on the processed infrared video stream and visible light video stream using a target detection model based on the DUMYOLO network, and obtain target detection results containing target location information and category labels. In the target detection module, the target detection model includes: a backbone network, a neck network, and a detection head; The backbone network is used to extract and fuse multimodal features from the processed infrared video stream and the image sequence in the visible light video stream to obtain fused multimodal features. The neck network is used to enhance the fused multimodal features, and to fuse the multimodal features and the enhanced multimodal features at multiple scales to obtain a multiscale fused feature map, and to filter the effective modal information in the multiscale fused feature map that meets preset requirements. The detection head is used to output a target detection result containing target location information and category label based on the effective modal information; The backbone network sequence includes: convolutional layers, a first feature interaction coupling module, a first C3k2 module, a second feature interaction coupling module, a second C3k2 module, a third feature interaction coupling module, a first A2C2f module, a fourth feature interaction coupling module, and a second A2C2f module; wherein, The convolutional layer is used to extract multimodal features from the image sequence and adjust the dimension and spatial size of the multimodal features; Both the first C3k2 module and the second C3k2 module are used to refine the multimodal features into discriminative features; The first feature interaction coupling module, the second feature interaction coupling module, the third feature interaction coupling module, and the fourth feature interaction coupling module are all used to encode the multimodal features spatial information, fuse spatial information and semantic information, and obtain the fused multimodal feature map. Both the first A2C2f module and the second A2C2f module incorporate an attention mechanism to capture target features from the fused multimodal feature map. The neck network includes: A U-shaped multi-kernel perceptron is used to perform multi-scale feature extraction, convolution operation, and edge information enhancement processing on the multimodal feature map to obtain an enhanced multimodal feature map. The Neck network is then used to fuse the multimodal feature map and the enhanced multimodal feature map at multiple scales to obtain a fused multi-scale feature map. A multispectral cross-domain collaborative unit is used to perform soft attention selection on the fused multi-scale feature map, capture global context information, suppress irrelevant modal information that does not meet preset requirements, and obtain effective modal information that meets preset requirements. The U-shaped multi-core perceptron includes five convolutions with kernel sizes of 3, 1, 5, 1, and 7, respectively. Each pair of convolutions is connected by a residual to form a U-shaped structure. The multispectral cross-domain collaborative unit comprises three branches: The first branch consists of parallel global pooling layers, average pooling layers, softmax normalization layers, and 1×1 convolutional layers, which are used to model global spatial context information. The second branch includes parallel 1×1 convolutions, softmax operations, and cross product operations, used to extract discriminative features from the original image sequence; The third branch includes a pooling downsampling layer and introduces a non-linear attention layer for image compression and feature selection to obtain effective modal information that meets preset requirements.

2. A cross-modal target detection method based on DUMYOLO infrared and visible light fusion, used to implement the system described in claim 1, characterized in that, include: Continuous infrared video streams and visible light video streams of the area to be detected are acquired, and frame sampling and image preprocessing are performed on the infrared video streams and visible light video streams. A target detection model based on the DUMYOLO network is used to perform target detection on the processed infrared video stream and visible light video stream, and the target detection results containing target location information and category labels are obtained.

3. The method according to claim 2, characterized in that, The target detection model includes: a backbone network, a neck network, and a detection head; The backbone network is used to extract and fuse multimodal features from the processed infrared video stream and the image sequence in the visible light video stream to obtain fused multimodal features. The neck network is used to enhance the fused multimodal features, and to fuse the multimodal features and the enhanced multimodal features at multiple scales to obtain a multiscale fused feature map, and to filter the effective modal information in the multiscale fused feature map that meets preset requirements. The detection head is used to output a target detection result containing target location information and category label based on the effective modal information.

Images