A night pedestrian detection method

Abstract

The present application provides a kind of night pedestrian detection method, the present application is based on the network structure of existing YOLOv5 algorithm to improve part details, make network have better detection performance, further guarantee the precision and speed of fusion algorithm.The present application carries out pre-processing to radar initial data, reduces the interference of invalid target to radar data, improves algorithm running speed, uses unscented Kalman filter algorithm to detect and track radar data, reduces radar detection false alarm rate, improves the reliability and stability of radar data.For the problem that single sensor cannot accurately obtain the comprehensive state information of night pedestrian target, the present application makes full use of the performance advantages of infrared camera and millimeter wave radar, avoids the missed detection and false detection of single sensor to target detection pedestrian etc.Using fusion algorithm can quickly and accurately detect pedestrian target in different scenes at night.

Description

Technical Field

[0001] This invention belongs to the field of sensor fusion technology, and more specifically, relates to a method for detecting pedestrians at night. Background Technology

[0002] Research on autonomous driving technology is becoming increasingly in-depth. Object detection, as a crucial component of autonomous driving environmental perception systems, has received widespread attention and research, particularly deep learning-based object detection technologies. Studies show that traffic accidents are more frequent and severe at night or in adverse weather conditions, causing significant loss of life and property. Therefore, nighttime pedestrian detection is of great importance for improving the safety of intelligent driving.

[0003] Application number CN202210507603.5 discloses a nighttime pedestrian detection method based on YOLOv3 deep learning. It utilizes an improved YOLOv3 algorithm to detect pedestrian targets in preprocessed infrared images. However, this method relies solely on a visual sensor for target detection, leading to missed detections of partially occluded pedestrian targets. Application number CN202010699510.8 discloses a nighttime target detection method based on millimeter-wave radar and visual fusion. It projects millimeter-wave radar data onto infrared images to generate regions of interest (ROIs) and uses deep learning algorithms to detect targets within the ROIs. However, this method overly depends on the prior ROIs generated by the millimeter-wave radar, resulting in a high rate of false positives and false negatives in pedestrian detection.

[0004] Based on the above analysis, research on target detection and recognition based on millimeter-wave radar and visual fusion has the following problems and shortcomings: In dark, unlit nighttime scenes, the detection performance of visible light cameras is significantly affected, and visual information cannot provide effective information for sensor fusion. Infrared cameras, employing thermal imaging technology, have the advantages of strong anti-interference capabilities and are not easily affected by harsh environments and lighting conditions, which can greatly improve the accuracy of pedestrian detection at night. Millimeter-wave radar has strong penetration capabilities through smoke and dust, good weather adaptability, and can acquire target relative position and velocity information, but it cannot acquire target category, outline, size, and other information, and is greatly affected by noise during operation, resulting in many false targets. If only radar is used to detect target position and status information to define the region of interest, valid targets may be affected by redundant targets. If information from different sensors is directly fused, the fusion detection performance is weak and biases may occur. Summary of the Invention

[0005] The purpose of this invention is to provide a method for detecting pedestrians at night, which overcomes the shortcomings of single-sensor detection and effectively improves the ability to detect pedestrians at night while ensuring detection speed.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is: a method for detecting pedestrians at night, comprising the following steps:

[0007] Step 1: Obtain the infrared pedestrian image dataset and perform noise reduction preprocessing on the infrared pedestrian images;

[0008] Step 2: Improve the deep learning object detection model YOLOv5 to obtain the YOLOv5-ours algorithm. The improvements include adding an end-side neural network to the CBL convolutional block to enhance the feature map; replacing the SPP module with the SimCSPSPPF module, which is similar to the CSP structure; adding an SE attention mechanism module; and using the YOLOv5-ours algorithm to train the dataset to obtain the weight file.

[0009] Step 3: Use the weight file to predict the infrared pedestrian image to obtain the visual detection target box including pedestrian category label and confidence information;

[0010] Step 4: Acquire pedestrian radar data and preprocess the pedestrian radar data to obtain effective pedestrian targets for radar.

[0011] Step 5: Perform unscented Kalman filtering detection and tracking on effective pedestrian targets identified by the radar;

[0012] Step 6: Using the infrared camera and millimeter-wave radar target matching strategy, the filtered pedestrian target is projected onto the pedestrian image and a radar detection target bounding box is generated;

[0013] Step 7: Calculate the intersection-union ratio (IU) of the visual target detection bounding box and the radar target detection bounding box, and output the sensor fusion target detection result according to the sensor fusion algorithm.

[0014] Preferably, the SPPF module in step 2 consists of three 5×5 maxpool layers in sequence. The SimCSPSPPF module divides the image feature layer into two parts, one part performs regular convolution operations, and the other part performs SPPF processing after convolution. Finally, the two parts are merged together. The SE module is added before the SimCSPSPPF structure to integrate the input image channel features and input them into the enhanced feature extraction network.

[0015] Preferably, the millimeter-wave radar data preprocessing in step 4 specifically includes: for airborne and stationary targets detected by the radar, they can be eliminated by checking whether the values ​​of relative angle, relative distance, and relative velocity are 0; for invalid targets, a target lifecycle L can be set. t >4, and velocity V t Target points that are not equal to 0 are considered as real target points; otherwise, they are discarded as false target points.

[0016] Preferably, in step 5, the unscented Kalman filter is used to detect and track valid pedestrian targets acquired by the radar. The UKF uses a deterministic form to approximate the posterior distribution of the state, generating more accurate estimates of the system state mean and covariance.

[0017] Preferably, the target matching strategy using infrared camera and millimeter-wave radar in step 6 includes spatial matching and temporal matching, specifically including:

[0018] For spatial fusion, the relationship between the radar and the world coordinate system can be established based on the radar's installation location. Using the camera's internal and external parameters and installation location, the transformation relationships between three sets of coordinate systems can be obtained: pixel coordinate system and image coordinate system, image coordinate system and camera coordinate system, and camera coordinate system and world coordinate system. Finally, the transformation relationship between the radar coordinate system and the pixel coordinate system can be obtained.

[0019]

[0020] In the formula, the position of the radar-detected target is (r, θ), and its position in the world coordinate system is (X). W ,Y W Z W R is the rotation matrix between the camera coordinate system and the world coordinate system, T is the translation matrix between the camera coordinate system and the world coordinate system, and Z is the rotation matrix between the camera coordinate system and the world coordinate system. c Let f be the Z-axis coordinate of the radar observation point in the camera coordinate system, and d be the focal length of the camera. x and d y This represents the physical size of each pixel on the x and y axes, and (p0, q0) is the origin of the image coordinate system in the pixel coordinate system.

[0021] The millimeter-wave radar has a sampling frequency of 20Hz (sampling period of 50ms), and the infrared camera has a sampling frequency of 30Hz (sampling period of 33.33ms). To ensure sensor time synchronization, the least common multiple of the sampling periods of the two sensors, 100ms, is selected as the sampling period of the fusion system. The millimeter-wave radar and the camera complete time fusion by sampling at intervals.

[0022] Preferably, the specific method for outputting the fused detection result in step 7 using a sensor fusion algorithm that combines the intersection-union ratio (IU / R) of the target bounding boxes detected by the infrared camera and the target bounding boxes detected by the millimeter-wave radar is as follows:

[0023] When 0 < IoU < 1, the camera category information and radar position information are fused. If the camera target and the radar target are matched for the first time, the camera category information is added to the radar detection box. Otherwise, it means that the radar and camera have been continuously and stably detecting the same target, and the visual detection category information is output, which includes the target position and status information detected by the millimeter-wave radar.

[0024] When IoU = 0 and the confidence level of the camera detection box P > 0.6, the system compensates for radar misses and outputs visual category information. When IoU = 0 and the confidence level of the camera detection box P < 0.6, if the radar target already has category information, it means that the radar target with category information and the visual detection box have not matched successfully. The system compensates for the visual detection box misses and outputs the radar target with category information. If the radar target does not have category information, the target matching is performed again.

[0025] Beneficial effects: This invention improves some details of the existing YOLOv5 algorithm network structure, giving the network better detection performance and faster detection speed, and further ensuring the accuracy and speed of the fusion algorithm.

[0026] This invention preprocesses the initial radar data, reducing interference from invalid targets and improving the algorithm's running speed. It also employs an unscented Kalman filter algorithm to detect and track radar data, reducing the false alarm rate and improving the reliability and stability of radar data.

[0027] To address the problem that a single sensor cannot accurately acquire comprehensive status information of pedestrian targets at night, this invention fully leverages the performance advantages of infrared cameras and millimeter-wave radar, avoiding situations where a single sensor misses or falsely detects pedestrians. A fusion algorithm is used to quickly and accurately detect pedestrian targets in different nighttime scenarios. Attached Figure Description

[0028] Figure 1 This is a diagram of the Ghost convolutional structure.

[0029] Figure 2 Here is a diagram of the SimCSPSPPF module structure;

[0030] Figure 3 Here is a diagram of the SE module structure;

[0031] Figure 4 Here is a diagram of the YOLOv5-ours network structure;

[0032] Figure 5 To improve the comparison chart of module combination detection results;

[0033] Figure 6 A comparison chart of detection results from different algorithms;

[0034] Figure 7 The result of Kalman filtering;

[0035] Figure 8 A framework diagram of a decision-level fusion algorithm for infrared cameras and millimeter-wave radar;

[0036] Figure 9 This is a visual inspection result image;

[0037] Figure 10 This is a diagram showing the target detection results after aligning radar information with the image.

[0038] Figure 11 This is a diagram showing the detection results of the fusion algorithm. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described in detail below with reference to examples. It should be understood that the specific examples described herein are merely illustrative of the invention, and the specific implementation of the invention will be described in detail below with reference to specific examples.

[0040] The infrared thermal imager used in this example is a FLIR PathFindIR, with a sampling frequency of 30 frames / second (i.e., a sampling period of 33.33ms), a resolution of 320*240, and a focal length of 19mm. The millimeter-wave radar is a Continental ARS404, with a sampling frequency of 20 frames / second (a sampling period of 50ms).

[0041] Step 1: Improve the YOLOv5 algorithm to enhance the performance of object detection on the visual sensor. Specifically, add an end-side neural network (GhostNet) to the CBL (Conv+BN+Leakyrelu) convolutional block to enhance the feature map. The Ghost convolutional structure is as follows: Figure 1 As shown; replace the SPP module with the SimCSPSPPF module, which has a similar structure to CSP, as shown. Figure 2 As shown, the SPPF module consists of three 5×5 max-pooling layers sequentially. Compared to the SPP module, the SPPF module has a faster detection speed and higher efficiency, and can solve multi-scale problems to some extent. The SimCSPSPPF module enhances the network's representation capabilities. The SimCSPSPPF module structure is as follows: Figure 3 As shown; adding the SE attention mechanism (Squeeze-and-Excitation Networks) module can, to some extent, strengthen the feature information of important feature layers and weaken the feature information of non-important feature layers. Adding the SE module before the SimCSPSPPF structure enhances the network's detection capability based on the integration of input image channel features. The SE module structure is as follows. Figure 3 As shown. The overall network structure of the improved YOLOv5 algorithm is as follows. Figure 4 As shown.

[0042] Step two involves adding individual modules to the original YOLOv5 algorithm and training the model on the CVC09 infrared pedestrian dataset to obtain optimal model weights. Precision, recall, and mAP are used as performance metrics to evaluate network quality. The training results before and after the algorithm improvement are compared, as shown in Table 1. Furthermore, individual modules are combined and added to the YOLOv5 algorithm, improving the detection results of the combined modules. Figure 5 As shown.

[0043] Analyzing the training results in Table 1, improvements to the Ghost convolution, SPPF module, and the SE attention mechanism alone can affect detection accuracy to varying degrees. Specifically, improving the Ghost module resulted in a 1.82% decrease in accuracy but a 31% increase in detection speed. Improving the SE module improved accuracy by 2.05%, but the change in detection speed was not significant. Although improving the SE module alone did not significantly change the mAP value, combining it with other modules resulted in a significant improvement in mAP value, such as... Figure 5 As shown, the combination of the SPPF module, Ghost convolution, and SE module results in the lowest target loss and the highest precision. Although the recall rate is not the highest, it exhibits the least fluctuation and is relatively stable after 40 rounds. Experimental results demonstrate that improving the algorithm by combining these three modules is an effective approach.

[0044] Table 1 Comparison of detection results for different improved modules

[0045] Improvement methods P(％) mAP(%) Recall (%) FPS Before improvement 88.42 89.28 85.21 29 Ghost module 86.60 88.30 84.53 38 SPP module 89.06 90.14 85.90 32 SE module 90.47 89.59 88.36 28

[0046] Step three: Verify the detection performance of the improved algorithm proposed in this paper. Comparative experiments were conducted with several typical object detection models, including Faster R-CNN, YOLOv3, YOLOv5, and YOLOv5-ours. The test results are shown in Table 2. It is clear that the improved YOLOv5-ours algorithm has a 3.7% improvement in detection accuracy and a 44.8% improvement in detection speed compared to YOLOv5. Compared to the classic deep learning object detection algorithm YOLOv5-ours, it has better detection performance and speed in pedestrian feature extraction, making it suitable for the visual information processing part of the sensor fusion detection algorithm in this invention.

[0047] Table 2 Comparison of detection results of different algorithm models

[0048] Detection methods P(％) mAP(%) Recall (%) FPS Fast-RCNN 79.37 76.02 72.19 14 YOLOv3 85.60 76.34 80.30 25 YOLOv5 88.43 89.85 85.28 29 YOLOv5-ours 92.13 94.62 93.24 42

[0049] Step four involves simultaneously acquiring infrared pedestrian images using an infrared camera and acquiring pedestrian radar data using millimeter-wave radar.

[0050] Step 5: Use the optimal model weights obtained from step 2 to predict the infrared pedestrian images collected in step 4, and obtain visual detection bounding boxes including category label and confidence information.

[0051] To more clearly compare the detection performance of different algorithm models, target detection was performed in nighttime road scenes, nighttime road scenes with occlusion, nighttime environmental scenes with low contrast, and foggy weather scenes. The detection results are as follows: Figure 6 As shown in the results, the improved YOLOv5 algorithm used in this invention has the highest confidence level and the best detection effect. Especially in scenario 2, when pedestrians are partially obscured by obstacles on the road, the improved algorithm of this invention can also accurately detect pedestrian targets.

[0052] However, in scenario 3, the algorithm presented in this paper exhibits a weakness in detecting small targets when the environmental contrast is low. In scenario 4, the performance of the infrared camera in detecting images is reduced due to heavy fog, resulting in missed detections and false detections. Since radar information is less affected by weather and lighting conditions, it can be used to improve the robustness and anti-interference ability of target detection during the target detection process.

[0053] Step six involves preprocessing the millimeter-wave radar data acquired in step four. Empty and stationary targets can be eliminated by checking if values ​​such as relative angle, relative distance, and relative velocity are zero. For invalid targets, a target lifetime L can be set. t >4 and velocity V t Target points that are not equal to 0 are considered as real target points; otherwise, they are discarded as false target points.

[0054] Step seven involves employing an unscented Kalman filter algorithm to filter out non-valid targets, reducing bounce error and lowering the false alarm rate of radar detection. Valid targets are then continuously tracked to obtain more accurate pedestrian movement information. The specific method of the unscented Kalman filter algorithm is as follows:

[0055] (1) Use the basic principle of UT transformation to obtain a set of sampling points and corresponding weights.

[0056]

[0057] (2) Calculate the state prediction at time k+1 for 2n+1 Sigma point sets, j = 1, 2, ... 2n+1.

[0058] X (j) (k+1 / k)=f[k,X( j) (k / k)]

[0059] (3) Prediction of system state variables at time k+1.

[0060]

[0061]

[0062] (4) Based on the predicted value at time k+1, use UT transformation again to generate a new Sigma point set.

[0063]

[0064] (5) Substitute the Sigma point set into the observation equation, j = 1, 2, ... 2n+1.

[0065] Z (j) (k+1 / k)=h[X (j) (k+1 / k)]

[0066] (6) System prediction mean and covariance.

[0067]

[0068]

[0069] In the formula

[0070]

[0071] The cross-covariance between the measurement and state vectors is:

[0072]

[0073] (7) Calculate the Kalman gain.

[0074] K(k+1)=P XZ P ZZ -1

[0075] (8) State and covariance update.

[0076]

[0077] P(k+1 / k+1)=P(k+1 / k)-K(k+1)P ZZ K T (k+1)

[0078] To evaluate the effectiveness of Kalman filtering in detecting and tracking effective radar targets, data on target 1 moving at a constant velocity in a straight line and target 2 moving at a constant acceleration in a curved path were selected from the acquired radar data and preprocessed. The effect of Kalman filtering is shown in the figure below. Figure 7As shown in Figure (a), the blue dots represent radar-detected target points, and the red dots represent target points after Kalman filtering. Figure (a) shows a missing detection phenomenon for moving targets with variable speed curves, as indicated by the green circles. Simultaneously, the horizontal and vertical coordinate errors suddenly increase, as shown in Figures (b) and (c). This phenomenon may be caused by sudden changes in radar target direction. When target 1 and target 2 meet and there is obstruction, the target will also be briefly lost, as indicated by the black circles. Experiments show that the Kalman filtering algorithm can, to some extent, compensate for the temporary loss of radar target detection and accurately predict and track moving targets.

[0079] Step 8, Spatial Fusion: Based on the millimeter-wave radar's installation location, establish the relationship between the radar and the world coordinate system. Using the camera's internal and external parameters and installation location, obtain the transformation relationships between three sets of coordinate systems: pixel coordinate system and image coordinate system, image coordinate system and camera coordinate system, and camera coordinate system and world coordinate system. Finally, obtain the transformation relationship between the radar coordinate system and the pixel coordinate system.

[0080]

[0081] In the formula, the position of the radar-detected target is (r, θ), and its position in the world coordinate system is (X). W ,Y W Z W R is the rotation matrix between the camera coordinate system and the world coordinate system, T is the translation matrix between the camera coordinate system and the world coordinate system, and Z is the rotation matrix between the camera coordinate system and the world coordinate system. c Let f be the Z-axis coordinate of the radar observation point in the camera coordinate system, and d be the focal length of the camera. x and d y This represents the physical size of each pixel on the x and y axes, and (p0, q0) is the origin of the image coordinate system in the pixel coordinate system.

[0082] Step 9: The millimeter-wave radar sampling frequency is 20Hz (sampling period is 50ms), and the camera sampling frequency is 30Hz (sampling period is 33.33ms). The least common multiple of the sampling periods of the two sensors, 100ms, is selected as the sampling period for the fusion system. The millimeter-wave radar and camera complete time fusion through interval sampling.

[0083] Step 10: After completing the spatiotemporal fusion of the infrared camera and millimeter-wave radar, project the Kalman-filtered radar pedestrian target data onto the corresponding simultaneous infrared pedestrian image. Centered on the radar target projection point, establish a rectangle with a width of 0.5m and a height of 2m as the radar detection target bounding box.

[0084] Step 11: The Intersection over Union (IoU) of the visually detected target bounding boxes and the millimeter-wave radar detected target bounding boxes is used for decision-making regarding the sensor fusion detection results. The formula for calculating IoU is:

[0085]

[0086] Where S r For visual detection of the target bounding box area, S R The area of ​​the target frame for millimeter-wave radar detection and tracking.

[0087] Step 12, the decision-level fusion algorithm framework for infrared cameras and millimeter-wave radar is as follows: Figure 8 As shown, when 0 < IoU < 1, visual detection category information and radar position information are fused. If the camera target and the radar target are matched for the first time, visual detection category information is added to the radar detection box. Otherwise, it means that the radar and camera have been continuously and stably detecting the same target, and visual detection category information is output, which includes the target position and status information detected by the millimeter-wave radar. When IoU = 0 and the confidence level P of the camera detection box is > 0.6, radar missed detection is compensated, and visual category information is output. When IoU = 0 and the confidence level P of the camera detection box is < 0.6, if the radar target already has category information, it means that the radar target with category information and the visual detection box have not matched successfully. Visual detection box missed detection is compensated, and the radar target with category information is output. If the radar target does not have category information, target matching is performed again.

[0088] To verify the robustness and anti-interference ability of the fusion algorithm for target detection, this invention collects infrared images of pedestrians on urban roads and in residential areas to conduct experimental comparisons between single-sensor target detection algorithms and fusion target detection algorithms. Figure 9 For visual detection of target bounding boxes, Figure 10 This image projects radar-detected target data onto an infrared image, where red dots represent radar target projection points and blue boxes represent radar-detected target bounding boxes. Figure 11 The images show the target detection results of the fusion algorithm proposed in this paper. The red box represents the visual detection results, the blue box represents the radar target detection results including visual category information, and the green box represents the detection results jointly detected by radar and visual information.

[0089] Step 13: To compare the detection performance of single sensors and fusion algorithms, 500 sets of infrared pedestrian images were selected and detected by radar, vision, and fusion algorithms respectively. The experimental results are shown in Table 3. Compared with the single sensor detection algorithm, the fusion algorithm proposed in this invention combines the advantages of millimeter-wave radar and vision sensors, making up for the shortcomings of single sensor performance. The target detection accuracy reached 95.57%, while effectively reducing the false negative rate and achieving better detection results.

[0090] Table 3 Comparison of Sensor Detection Results

[0091] processing method Total number of pedestrians Number of positive checks Number of missed detections Accuracy (%) False negative rate (%) Radar detection 1852 1719 133 92.82 7.18 Visual inspection 1852 1695 157 91.52 8.48 Fusion detection 1852 1770 82 95.57 4.43

[0092] This invention utilizes an infrared camera to acquire infrared pedestrian images while simultaneously obtaining raw pedestrian radar data from a millimeter-wave radar. A deep learning algorithm is used to obtain visual detection bounding boxes from the infrared pedestrian images. Spatial fusion and temporal alignment target matching methods are employed to project the millimeter-wave radar pedestrian data onto the infrared pedestrian images, generating radar detection bounding boxes. Finally, a decision-level fusion algorithm is used to output the fused detection results. The fusion algorithm achieves a target detection accuracy of 95.57%. This method improves the accuracy and robustness of nighttime pedestrian target detection, demonstrating good detection performance and strong practicality.

[0093] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention should be covered within the protection scope of the present invention.

Claims

1. A method for detecting pedestrians at night, characterized in that, It includes the following steps: Step 1: Obtain an infrared pedestrian image dataset and perform noise reduction preprocessing on the infrared pedestrian images. Step 2: Improve the deep learning object detection model YOLOv5 to obtain the YOLOv5-ours algorithm. The improvements include adding the end-side neural network GhostNet in the CBL convolutional block to enhance the feature map; replacing the SPP module with a SimCSPSPPF module similar to the CSP structure. The SimCSPSPPF module divides the image feature layer into two parts. One part performs conventional convolution operations, and the other part undergoes SPPF processing after convolution, and finally the two parts are merged together; adding a SE attention mechanism module before the SimCSPSPPF structure to integrate the input image channel features and input them into the enhanced feature extraction network; using the YOLOv5-ours algorithm to train the dataset to obtain a weight file. Step 3: Use the weight file to predict the infrared pedestrian images to obtain visual detection target boxes including pedestrian category label and confidence Confidence information. Step 4: Obtain pedestrian radar data and perform preprocessing on the pedestrian radar data to obtain effective radar pedestrian targets. Step 5: Perform unscented Kalman filter detection and tracking on the effective radar pedestrian targets. Step 6: Use the infrared camera and millimeter-wave radar target matching strategy to project the unscented Kalman filter pedestrian targets onto the pedestrian images and generate radar detection target boxes. Step 7: Calculate the intersection over union (IoU) of the visual target detection box and the radar detection target box, and output the sensor fusion target detection result according to the sensor fusion algorithm. The specific method is as follows: When 0 < IoU < 1, fuse the camera category information and the radar position information. If the camera target and the radar target are matched for the first time, add the camera category information to the radar detection box, otherwise output the visual detection category information, including the target position and status information detected by the millimeter-wave radar. When IoU = 0 and the confidence P of the camera detection box > 0.6, compensate for the radar missed detection and output the visual category information. When IoU = 0 and the confidence P of the camera detection box < 0.6, if the radar target already has category information, compensate for the visual detection box missed detection and output the radar detection target with the existing category information; otherwise, perform target matching again.

2. The method for detecting pedestrians at night according to claim 1, characterized in that: The SPPF module in Step 2 consists of three 5×5 maximum pooling Maxpool layers in series.

3. The method for detecting pedestrians at night according to claim 1, characterized in that: The preprocessing of the millimeter-wave radar data in Step 4 specifically includes: using whether values such as relative angle, relative distance, and relative speed are 0 to eliminate empty and stationary targets detected by the radar; for invalid targets, set the target lifetime Lt > 4, and consider target points with a speed Vt not equal to 0 as real target points, otherwise eliminate them as false target points.

4. The method for detecting pedestrians at night according to claim 1, characterized in that: The unscented Kalman filter used in Step 5 detects and tracks the effective pedestrian targets collected by the radar. The unscented Kalman filter uses a deterministic form to approximate the posterior distribution of the state and generates more accurate estimates of the system state mean and covariance.

5. The method for detecting pedestrians at night according to claim 1, characterized in that: The target matching strategy using infrared cameras and millimeter-wave radar in step 6 includes spatial matching and temporal matching, specifically: For spatial fusion, the relationship between the radar and the world coordinate system can be established based on the radar's installation location. Using the camera's internal and external parameters and installation location, the transformation relationships between three sets of coordinate systems can be obtained: pixel coordinate system and image coordinate system, image coordinate system and camera coordinate system, and camera coordinate system and world coordinate system. Finally, the transformation relationship between the radar coordinate system and the pixel coordinate system can be obtained. In the formula, the position of the target detected by the radar is: Its position in the world coordinate system is ( R is the rotation matrix between the camera coordinate system and the world coordinate system, and T is the translation matrix between the camera coordinate system and the world coordinate system. For radar observation points in the camera coordinate system Axis coordinates For the camera's focal length, and express shaft and The physical size of each pixel on the axis. This represents the position of the origin of the image coordinate system in the pixel coordinate system. The millimeter-wave radar has a sampling frequency of 20Hz and a sampling period of 50ms, while the infrared camera has a sampling frequency of 30Hz and a sampling period of 33.33ms. To ensure sensor time synchronization, the least common multiple of the sampling periods of the two sensors, 100ms, is selected as the sampling period of the fusion system. The millimeter-wave radar and the camera complete time fusion by sampling at intervals.

Images