Visible light, infrared and radar fusion target detection method based on deep learning

Abstract

The present invention discloses a visible light, infrared and radar fusion target detection method based on deep learning, comprising: automatically analyzing and processing data collected by a visible light camera, a thermal infrared camera and a laser radar, firstly fusing the pre-processing stage of visible light and infrared to realize the conversion of the laser radar point cloud to a bird's-eye view, and then automatically extracting and bifurcating pyramid-type cross-stage features of the fusion image and the bird's-eye view, and then performing bifurcated target detection, and then performing "graph" matching, and finally realizing target detection by fusing the three modes of visible light, infrared and radar, thereby improving the detection accuracy, having good robustness, and being able to adapt to the scene changes and dynamic changes of the load. With the increase of the types of targets in the training set and the increase of the training set, the present invention has a broader detection performance.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence target detection and matching technology, and specifically to a visible light, infrared and radar fusion target detection method based on deep learning. Background Art

[0002] Target detection is the main research problem in the field of computer vision. Image segmentation, target tracking, target behavior analysis, etc. are all based on target detection. With the development of deep learning algorithms based on convolutional neural networks, target detection has made great breakthroughs. Target detection algorithms based on visible light camera images have been widely used. As a type of sensor, visible light cameras have advantages such as realistic geometric shapes, strong stereoscopic perception, and good resolution. However, they also have their limitations. For example, visible light cameras are passive cameras. When they are disturbed by low light or rain, snow, fog, etc., the imaging quality is significantly reduced, and even the algorithm fails. Thermal infrared cameras are another type of passive cameras. They rely on normal light and can be used to supplement the imaging performance of visible light cameras in low light. LiDAR is an active detector that measures the distance, reflectivity and other parameters of the target by emitting a beam of light (usually a pulsed laser) to the target. LiDAR can supplement the distance information lacking in visible light cameras and can work effectively in low light.

[0003] The information obtained by single, unintegrated target detection in the scene is not rich enough, nor can it help to better understand the target and the surrounding environment. It cannot adapt to complex scene changes and dynamic changes. With the increase in the number of target types in the training set and the increase in the training set, the efficiency and performance of detection may gradually decrease.

[0004] Therefore, people need a visible light, infrared and radar fusion target detection method based on deep learning to solve the above problems. Summary of the invention

[0005] The purpose of the present invention is to provide a visible light, infrared and radar fusion target detection method based on deep learning to solve the problems raised in the above background technology.

[0006] In order to solve the above technical problems, the present invention provides the following technical solution: a visible light, infrared and radar fusion target detection method based on deep learning, characterized in that it includes the following steps:

[0007] S1: Build N = 6 typical experimental scenarios, select 2N = 12 targets (dynamic targets + static targets) in different scenarios, use the self-developed multimodal fusion sensor SensorBox mounted on the unmanned vehicle to randomly walk in different experimental scenarios to collect multimodal data sets, and store the multimodal data as rosbag packages;

[0008] S2: Generate a fusion image and a bird's-eye view image, and annotate the target matching truth values ​​of the fusion image and the bird's-eye view image;

[0009] S3: Divide the dataset into training set, validation set and test set;

[0010] S4: Load the visible light + infrared fusion image and radar bird's-eye view image;

[0011] S5: Build the fusion detection network FusionNet, extract and match the detection targets, train the training set divided in step S2, and obtain the trained weight file;

[0012] S6: Use the fusion detection network FusionNet and the obtained weight file to detect the test set, obtain the detection results and analyze the accuracy of fusion target detection.

[0013] Furthermore, the multimodal data set refers to a visible light, thermal infrared and lidar data set containing N = 6 typical scenes and 2N = 12 typical targets, wherein the 6 typical scenes are indoor, open (daytime), open (nighttime), semi-open (daytime), semi-open (nighttime) and underground, and different environments are quite different. For example, the illumination in underground and nighttime environments is weak, and the visible light data is darker; the radar data in indoor and underground environments is limited to a certain height range; the thermal infrared camera data in different scenes is less affected by illumination, but is more strongly affected by the dynamic interference of moving targets, and the imaging resolution and field of view of the thermal infrared camera are small. In indoor scenes: this scene mainly uses the RGB data obtained by the camera to perform target detection on the obtained data, and the data collected by the infrared sensor, the millimeter-wave radar data and the lidar data are important supplements to target detection;

[0014] In open scenes (daytime): This scene mainly uses RGB data obtained by the camera for target detection. Thermal infrared data and lidar data are important supplements to RGB target detection. In addition, millimeter-wave radar data is an important supplement to target position detection.

[0015] In open scenes (night): This scene mainly uses infrared data obtained by infrared sensors for target detection. The RGB data obtained by the camera and the lidar data are important supplements for infrared target detection.

[0016] In semi-open scenes (daytime): This scene mainly uses RGB data obtained by the camera and data obtained by the infrared sensor for target detection. Millimeter-wave radar data and lidar data are important supplements for target detection.

[0017] In semi-open scenes (at night): This scene mainly uses the laser radar point cloud data and the data obtained by the infrared sensor for target detection. The millimeter wave radar data and infrared data are important supplements to target detection.

[0018] In underground scenarios: In low-light underground garage scenarios, thermal infrared data is an important supplement to visual target detection; in the case of only forward-facing cameras, lidar data is an important supplement to target detection in visual blind spots;

[0019] Among them, the 12 typical targets refer to the 12 selected targets to be detected: people, cars, wheels, tank models, chairs, small stools, air conditioners, laptops, tables, cyclists, traffic lights, and trucks. The dynamic targets include cyclists, people, cars, and tank models. People and cars also have static situations. The various targets are unevenly distributed in different scenes. Among them, the underground garage contains people, cars, and wheels; the indoor scene contains people, tank models, chairs, small stools, air conditioners, laptops, and tables; the semi-open and open outdoor scenes include people, trucks, cyclists, and traffic lights.

[0020] Furthermore, in step S2: the visible light and infrared images of the corresponding frames are aligned according to the calibration files of the visible light and infrared cameras, and merged to generate a fusion image. The Label Image tool is used to annotate the fusion image with 12 types of targets, and the annotations are saved in txt format. The Cloud2BEV algorithm is designed to convert the 3D lidar point cloud of each frame into a bird's-eye view containing radar information. The bird's-eye view image is annotated with targets using Label Image and saved in txt format. According to the visible light and radar calibration files, the target annotations of the visible light and radar are manually aligned, and the visible light-radar matching truth value is saved. According to the visible light and radar calibration files, the target matching truth value of the fusion image and the bird's-eye view image is annotated: the fusion image annotation, the bird's-eye view annotation and the original point cloud data are read respectively, and the point cloud contained in the annotated target is determined according to the annotation box of the bird's-eye view. The contained point cloud is projected to the pixel coordinates of the fusion image according to the calibrated internal and external parameters, and the matching of the bird's-eye view and the fusion image with the same target is completed. Due to the inaccurate automatic registration caused by the annotation error, the internal and external parameter error, etc., manual matching can be performed to improve the accuracy.

[0021] Furthermore, the Cloud2BEV algorithm refers to an algorithm for converting three-dimensional point cloud data into a bird's-eye view. The region of interest of the radar point cloud is set to the region from 10 meters on the left to 10 meters on the right and from 10 meters behind the radar to 10 meters in front. The aspect ratio of the region is 1:1. The pixels of the bird's-eye view after conversion are 640×640. Each pixel point of the bird's-eye view has 3 channels, namely: the average reflection intensity of all points contained in the pixel point, the average height, that is, the z-axis value and the normalized density. Among them, the region of interest refers to the area to be processed that is outlined by the processed image in the form of a box, circle, ellipse, irregular polygon, etc.

[0022] Furthermore, in step S4: a fusion data loading program is written to load the visible light + infrared fusion image and the radar bird's-eye view image at the same time, so as to implement various types of random data enhancement methods and parallel loading.

[0023] Furthermore, the data enhancement method is performed on a fusion image containing visible light + infrared information and a bird's-eye view image containing radar information, including: color gamut transformation, illumination distortion, random scaling, random left-right flipping, random up-down flipping and mosaic data enhancement. Mosaic data enhancement refers to: using 4 pictures, randomly cropping and scaling the target and background, and synthesizing one picture, updating the target annotation, enriching the background information of the data, and enhancing the robustness.

[0024] Furthermore, in step S5: build a fusion detection network FusionNet, based on YOLOv5, use CSPNet, SPPNet and PANet packages, modify the network, introduce SE module to realize channel attention mechanism, design Leg layer to realize separation of fusion features, introduce Graph layer according to graph neural network GNN, perform graph matching and graph convolution on the detection results of fusion graph and bird's-eye view, and finally realize the fusion of three modalities, use the training set divided in step S3 for training, use the validation set to display the effect of each generation of training in real time, and obtain the trained weight file after training.

[0025] Furthermore, the FusionNet refers to a deep neural network designed to perform fusion target detection on visible light, infrared and lidar data. The input of the network is (bs, 6, 640, 640), where bs refers to the batch size batch. size, 640 is the height and width of the image, 6 refers to the two images with 6 channels in total, the two images are: a picture after the visible light and infrared are merged and the bird's-eye view after the radar point cloud is converted to BEV. The network adopts residual network, cross-stage local network, compression and excitation block, spatial pyramid pooling network and path aggregation network and other technologies to solve the problem of gradient information duplication in network optimization in other large convolutional neural network frameworks, and integrates the gradient changes from beginning to end into the feature map, which can effectively alleviate the gradient disappearance problem, support feature propagation, encourage the network to reuse features, and reduce the number of model parameters and FLOPS value. Among them, the residual network used is the Bottleneck residual network structure, which increases the network depth and makes the network have more powerful feature extraction capabilities. Before each detection head, the feature bifurcation network is used to split the feature into two parts, and then the detection head is regressed. The regression results are respectively compared with the annotation files of the visible light + infrared image and the annotation files of the BEV image to calculate the loss function. The output matching result is compared with the annotation true value to calculate the loss function. The loss function adopts L1 loss for network learning.

[0026] Furthermore, the Leg layer refers to a layer composed of a bifurcation network that processes features of three different scales, with scales of 20×20, 40×40, and 80×80, respectively, which separates the merged visible light infrared fusion image features and radar bird's-eye view image features, which is conducive to target detection in the two images separately.

[0027] Furthermore, the Graph layer refers to a network layer built using a graph neural network GNN for matching visible light + infrared detection results and radar detection results. First, the two results are embedded into a graph Graph using a graph embedding method, that is, a "graph" of node matching and edge matching is built based on the n targets of the fusion graph result and the m targets of the bird's-eye view. Then, the "graph" is processed using graph convolution, and the matching results are output. The loss function is calculated between the matching results and the actual labeled matching data, which is conducive to updating the network weights.

[0028] Compared with the prior art, the beneficial effects achieved by the present invention are:

[0029] 1. Compared with the previous target detection algorithms, the present invention can realize fusion target detection for the fusion data collected by multi-modal sensors, realize the algorithm and network that can process multi-modal information at the same time, solve the problem of fusing information of different modalities, make full use of the complementary characteristics of information of different modalities, and has the advantages of high stability, strong anti-interference ability, good robustness, etc.;

[0030] 2. The present invention is based on the target detection algorithm YOLOv5, which has the strongest performance of single-modal detection in the current target detection field. YOLOv5 is further expanded to enable it to have the ability to process fusion information, and the SE module is introduced to increase a small amount of calculation while improving the accuracy of target detection. The Leg layer is introduced to achieve the separation of different modal information, and the graph neural network GNN is used to achieve the fusion matching of different modal information, thereby improving the comprehensive performance of target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0032] Figure 1 It is a flow chart of the visible light, infrared and radar fusion target detection method based on deep learning of the present invention;

[0033] Figure 2 is an example diagram of three modal sensor data of the present invention;

[0034] Figure 3 It is the overall deep learning network model diagram of the present invention;

[0035] Figure 4 This is a multi-scene, multi-target, multi-modal fusion target detection result diagram of the present invention. DETAILED DESCRIPTION

[0036] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0037] See also Figure 1-4 The present invention provides a technical solution: a visible light, infrared and radar fusion target detection method based on deep learning, characterized in that it includes the following steps:

[0038] S1: Build N = 6 typical experimental scenarios, select 2N = 12 targets in different scenarios, and use multimodal fusion sensors to collect multimodal data sets;

[0039] S2: Generate a fusion image and a bird's-eye view image, and annotate the target matching truth values ​​of the fusion image and the bird's-eye view image;

[0040] S3: Divide the dataset into training set, validation set and test set;

[0041] S4: Load the visible light + infrared fusion image and radar bird's-eye view image;

[0042] S5: Build the fusion detection network FusionNet, extract and match the detection targets, train the training set divided in step S2, and obtain the trained weight file;

[0043] S6: Use the fusion detection network FusionNet and the obtained weight file to detect the test set, obtain the detection results and analyze the accuracy of fusion target detection.

[0044] The multimodal dataset refers to the visible light, thermal infrared and lidar datasets containing N = 6 typical scenes and 2N = 12 typical targets.

[0045] In step S2: the visible light and infrared images of the corresponding frames are aligned according to the calibration files of the visible light and infrared cameras, and a fusion image containing visible light and infrared information is generated, and the fusion image is labeled with targets. The Cloud2BEV algorithm is designed to convert the 3D lidar point cloud of each frame into a bird's-eye view containing radar information, and the bird's-eye view is labeled with targets. The target matching truth values ​​of the fusion image and the bird's-eye view are labeled according to the visible light and radar calibration files.

[0046] The Cloud2BEV algorithm refers to an algorithm that converts three-dimensional point cloud data into a bird's-eye view. The range is [-10, 10] meters to the left and right of the radar and [-10, 10] meters in front and behind. After conversion, the pixel size of the bird's-eye view is 640×640. Each pixel point in the bird's-eye view has three channels, namely: the average reflection intensity of all points contained in the pixel point, the average height, that is, the value of the z-axis, and the normalized density.

[0047] In step S4: write a fusion data loading program to load the visible light + infrared fusion image and the radar bird's-eye view image at the same time, and implement various types of random data enhancement methods and parallel loading.

[0048] The data enhancement method is performed on the fusion image and the bird's-eye view image, including: color gamut transformation, illumination distortion, random scaling, random left-right flipping, random up-down flipping and mosaic data enhancement.

[0049] In step S5: build a fusion detection network FusionNet, use the convolutional neural network CNN to perform unified feature extraction and bifurcation target detection on the fusion image and the bird's-eye view image, and then use the graph neural network GNN to match the detection targets of the fusion image and the bird's-eye view image. Use the training set divided in step S3 for training, use the validation set to display the effect of each generation of training in real time, and obtain the trained weight file after the training is completed.

[0050] FusionNet refers to a deep neural network designed to process fusion images and bird's-eye views at the same time. It modifies the input layer based on YOLOv5, introduces the SE module into the cross-stage layout network, introduces the channel attention mechanism SE into Bottleneck to form SE-Bottleneck and then SE-CSP network, introduces the bifurcation layer Leg layer, sets a double detection head, and introduces the Graph layer of the graph neural network to match the two target detection results, so as to improve the accuracy of target detection.

[0051] The Leg layer refers to a layer composed of a bifurcated network that processes features of three different scales, with scales of 20×20, 40×40, and 80×80 respectively.

[0052] The Graph layer refers to a network layer built using the graph neural network GNN to match the detection results of visible light + infrared and the detection results of radar.

[0053] Embodiment 1: First, in the data collection and data pre-processing stage: collect Figure 2 The visible light, thermal infrared and lidar data shown are time synchronized, as shown in Figure 2 As shown in (a) and (b), the visible light data is a 3-channel grayscale image with a resolution of 1280*720, and the 3 channels have the same values. The data of the thermal infrared camera has been processed by the sensor and is a 3-channel color image with a resolution of 384*288. The two cameras have different fields of view. The two images need to be merged into a fusion image based on the camera calibration data. The radar is a 16-line Velodyne lidar, and the radar data is 3D point cloud data. The bird's-eye view after conversion by the Cloud2BEV algorithm is shown in the figure below. Figure 2 As shown in (c), the fusion image and the bird's-eye view image are resized to (640, 640, 3), and then the tensor Tensor resized to (640, 640, 6) is stacked and sent to the model. The fusion image and the bird's-eye view image are labeled respectively. The labeled data includes: category, bbox, that is, the normalized coordinates of 12 types of targets in the pixel coordinate system, saved in YOLO format (txt), and the annotation file path corresponds to the original image file path one by one. The data is written to load the visible light + infrared fusion image and the radar bird's-eye view image and the data enhancement algorithm to achieve color gamut transformation, illumination distortion, random scaling, random left and right flipping, random up and down flipping and mosaic data enhancement;

[0054] Secondly, in the network design phase: the network architecture is as follows: Figure 3 As shown in Figure 2, the Leg layer is added to the classic Backbone+Neck+Head target detection architecture to form the Backbone+Neck+Leg+Head architecture, as shown in Figure 2 Figure 3As shown in (b), the powerful feature extraction capability mainly comes from the backbone network Backbone. Backbone uses the spatial attention mechanism Foucus to increase the receptive field by 4 times; uses convolution blocks containing convolution, batch normalization and activation functions instead of pooling as the intermediate link between different layers; uses spatial pyramid pooling to adapt to sub-images of different sizes; the most important layer in Backbone is SE-CSP, which is based on the cross-stage local network CSP. CSP is based on the idea of ​​Densnet, which copies the feature map of the base layer and passes dense The block sends a copy to the next stage, thereby separating the feature map of the base layer. Inside the CSP, the Bottleneck residual network structure is used to increase the network depth. The channel attention mechanism SE is introduced into the Bottleneck to form SE-Bottleneck and then SE-CSP. Another route is added on the basis of the original residual block. The global pooling is used to obtain the initial channel weights, and then two complete connection layers and sigmoid activation functions are used to update the weight of each channel. Finally, the original channel is multiplied by the weight of each channel. In this way, during the network training process, the weight of each channel is learned by gradient descent to improve the detection accuracy; Figure 3 As shown in (c), feature enhancement and feature extraction for objects of different sizes are mainly completed by the Neck layer; the Neck layer adopts the path aggregation network PANet, and the feature extractor of PANet adopts a new feature pyramid network FPN that enhances the bottom-up path, improves the feature propagation of the bottom layer, and adopts raw sampling Upsample for top-down feature propagation, and convolutional blocks CONV for bottom-up feature propagation. Through lateral connection and stacking, the features of the previous stage are added to the feature map, and these features provide information for the features of the next stage. Each stage has three features of different scales, thereby realizing the detection of the same object of different sizes and scales, as shown in Figure 1. Figure 3 As shown in (d), the bifurcation of the fusion feature mainly depends on the Leg layer. The three different scale features output by the Neck layer include the features of the fusion image and the bird's-eye view. The Leg layer uses a slicing method to separate the features, as shown in Figure 3 As shown in (e), the detection is mainly realized by the detection head. The separated fusion image and the bird's-eye view, a total of 6 features, are sent to the detection heads of 3 scales for detection. The detection head is mainly implemented using Conv2d, as shown in Figure 3 As shown in (f), the target detection results of visible light + infrared and radar are mainly fused by the Graph layer. After the detection results of the detection head are subjected to non-maximum suppression (NMS), the targets detected by the fusion graph and the bird's-eye view are formed into a "graph" through the graph embedding method. The graph is processed by the graph neural network and the matching results are output;

[0055] Then, in the network training stage: after the data and network of the fusion image and the bird's-eye view are prepared, the stacked tensors of the fusion image and the radar image are input into the model, so that the detection head outputs the detection result, and the loss function is calculated by comparing it with the labeled true value. The loss function includes classification loss, positioning loss and confidence loss. The classification loss is calculated according to the difference between the target prediction category and the true value category, the positioning loss is calculated according to the IoU of the predicted bbox and the true value bbox, and the confidence loss is calculated according to the presence or absence of the true value mark. , update the model by gradient descent according to the loss function, obtain the weight file after training, test it on the test set, check the target detection results of the model, and calculate the confusion matrix (Confusion matrix) according to the detection results and the true value, and then calculate the performance indicators, including: Precision Precision, Recall Recall, mAP average AP of each category, and F1 score F1_score. If the effect is not ideal, use the network and the weight file to automatically annotate more data sets, and then perform manual fine-tuning, adjust the network parameters, and retrain until the F1 score reaches 90%. The weights of the Graph layer need to be trained separately. After the target matching data and network are prepared, the detected targets are input into the graph neural network that constitutes the Graph layer, and graph embedding and graph convolution calculations are performed. The matching results are output and compared with the annotated true value to calculate the loss function. The loss function uses L1 loss. Update the model by gradient descent according to the loss function, obtain the weight file after training, test it on the test set, and check the target matching results, including: mAP and F1 score;

[0056] Finally, in the model deployment and inference phase: visible light and infrared image fusion program, laser point cloud to bird's eye view program, data loading program, Figure 3 The network architecture shown in the figure, the trained weight file and the source code are deployed to the computer, and the multimodal data set collected by the multimodal sensor is input to automatically realize data pre-processing, target detection and target fusion, and output the detection results in real time. The detection results of the experimental test are shown in Figure 4 shown.

[0057] Finally, it should be noted that the above description is only a preferred example of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A visible light, infrared and radar fusion target detection method based on deep learning, characterized by: The following steps are involved: S1: Build N typical experimental scenarios, select 2N targets in different scenarios, and use multimodal fusion sensors to collect multimodal data sets; S2: Generate a fusion image and a bird's-eye view image, and annotate the target matching truth values ​​of the fusion image and the bird's-eye view image; S3: Divide the dataset into training set, validation set and test set; S4: Load the visible light + infrared fusion image and radar bird's-eye view image; S5: Build the fusion detection network FusionNet, extract and match the detection targets, train the training set divided in step S2, and obtain the trained weight file; S6: Use the fusion detection network FusionNet and the obtained weight file to detect the test set, obtain the detection results and analyze the fusion target detection accuracy; In step S5: build a fusion detection network FusionNet, use the convolutional neural network CNN to perform unified feature extraction and bifurcation target detection on the fusion image and the bird's-eye view image, and then use the graph neural network GNN to match the detection targets of the fusion image and the bird's-eye view image. Use the training set divided in step S3 for training, use the validation set to display the effect of each generation of training in real time, and obtain the trained weight file after the training is completed.

2. The visible light, infrared and radar fusion target detection method based on deep learning according to claim 1 is characterized in that: The multimodal dataset refers to a visible light, thermal infrared and lidar dataset containing N typical scenes and 2N typical targets.

3. The visible light, infrared and radar fusion target detection method based on deep learning according to claim 1 is characterized in that: In step S2: align the visible light and infrared images of the corresponding frames according to the calibration files of the visible light and infrared cameras, merge them to generate a fusion image containing visible light and infrared information, annotate the fusion image with targets, design the Cloud2BEV algorithm, convert each frame of the 3D lidar point cloud into a bird's-eye view containing radar information, annotate the bird's-eye view with targets, and annotate the target matching truth values ​​of the fusion image and the bird's-eye view according to the visible light and radar calibration files: read the fusion image annotation, the bird's-eye view annotation and the original point cloud data respectively, determine the point cloud contained in the annotated target according to the annotation box of the bird's-eye view, project the contained point cloud to the pixel coordinates of the fusion image according to the calibrated internal and external parameters, and complete the matching of the bird's-eye view and the fusion image with the same target.

4. The visible light, infrared and radar fusion target detection method based on deep learning according to claim 3 is characterized in that: The Cloud2BEV algorithm refers to an algorithm that converts three-dimensional point cloud data into a bird's-eye view. The range is [-10, 10] meters to the left and right of the radar and [-10, 10] meters in front and behind. The pixel size of the bird's-eye view after conversion is 640×640. Each pixel point of the bird's-eye view has three channels, namely: the average reflection intensity of all lines contained in the pixel point, the average height, that is, the value of the z-axis, and the normalized density.

5. The visible light, infrared and radar fusion target detection method based on deep learning according to claim 1, characterized in that: In step S4: write a fusion data loading program to load the visible light + infrared fusion image and the radar bird's-eye view image at the same time, and implement various types of random data enhancement methods and parallel loading.

6. The visible light, infrared and radar fusion target detection method based on deep learning according to claim 5 is characterized in that: The data enhancement method is performed on the fusion image and the bird's-eye view image, including: color gamut transformation, illumination distortion, random scaling, random left-right flipping, random up-down flipping and mosaic data enhancement.

7. The visible light, infrared and radar fusion target detection method based on deep learning according to claim 1, characterized in that: In step S6: the FusionNet refers to a deep neural network designed to simultaneously process a fusion graph and a bird's-eye view. The input layer is modified based on YOLOv5, and the SE module is introduced into the cross-stage local network CSP to form a SE-CSP network. The bifurcation layer Leg layer is introduced, and a double detection head is set. The Graph layer of the graph neural network is introduced to match the two target detection results. After obtaining the detection results, a confusion matrix is ​​calculated based on the detection results and the true values, and the mAP and F1 scores are calculated based on the confusion matrix. The accuracy of the fusion target detection is judged by the obtained mAP and F1 scores.

8. The visible light, infrared and radar fusion target detection method based on deep learning according to claim 7 is characterized in that: The Leg layer refers to a layer composed of a bifurcated network that processes three different scale features.

9. The visible light, infrared and radar fusion target detection method based on deep learning according to claim 8, characterized in that: The Graph layer refers to a network layer built using a graph neural network (GNN) for matching the detection results of visible light + infrared and the detection results of radar.

Images