A pedestrian target detection method based on radar-video-infrared decision-level fusion

Abstract

The application discloses a pedestrian target detection method based on radar video infrared decision-level fusion, and relates to the technical field of automatic driving perception. The application aims to solve the problem that the existing pedestrian detection method uses a single mode of video sensor to perform a pedestrian target detection task under poor light conditions, and the intelligent driving vehicle has more false detection and missed detection conditions for pedestrian detection, so that the intelligent driving vehicle depending on ADAS cannot timely avoid pedestrians and makes an incorrect driving strategy and even a collision occurs. The process is as follows: a pedestrian detection platform is built; laser radar, visible light cameras and infrared cameras are time and space aligned; pedestrian targets in laser radar point clouds, pedestrian targets in visible light images and pedestrian targets in infrared images are detected; target detection results are fused based on a decision-level fusion algorithm to obtain final pedestrian target detection results.

Description

Technical Field

[0001] This invention provides a pedestrian target detection method based on radar video infrared decision-level fusion, which relates to the field of autonomous driving perception technology. Background Technology

[0002] In recent years, with the continuous development of deep learning and object detection technologies, the accuracy of video-based pedestrian detection methods can meet the needs of intelligent driving under good lighting conditions. However, since lighting conditions in the actual driving environment are not always ideal, relying on a single-modality visible light camera cannot meet the pedestrian detection needs of all-weather intelligent driving tasks under adverse lighting conditions such as night, backlight, and glare. Furthermore, when using LiDAR for detection, if the speed of detection is pursued, there will often be a lot of false alarms.

[0003] LiDAR has a wide detection range, strong light robustness, and can provide rich spatial information, while infrared cameras detect the heat emitted by objects and are not affected by changes in light. If multi-source and heterogeneous information fusion (MSHIF) technology can be used to complement the pedestrian targets detected by LiDAR and infrared cameras in different modal data and the pedestrian detection results of visible light cameras, the limitations of a single sensor in the perception process can be effectively avoided, thereby achieving a more comprehensive perception and recognition of pedestrian targets and improving the external perception capabilities of intelligent driving vehicles. Summary of the Invention

[0004] The purpose of this invention is to address the problem that existing pedestrian detection methods using single-modality video sensors in poor lighting conditions result in numerous false detections and missed detections by autonomous vehicles. This leads to autonomous vehicles relying on ADAS being unable to avoid pedestrians in time, making incorrect driving strategies or even causing collisions. Therefore, this invention proposes a pedestrian target detection method based on radar, video, and infrared decision-level fusion.

[0005] The specific process of a pedestrian target detection method based on radar video infrared decision-level fusion is as follows:

[0006] Step 1: Set up a pedestrian detection platform, which includes LiDAR, visible light cameras, and infrared cameras;

[0007] Step 2: Perform spatiotemporal alignment on the lidar, visible light camera, and infrared camera;

[0008] Step 3: Based on the spatiotemporally aligned LiDAR, visible light camera, and infrared camera, perform target detection on pedestrian targets in the LiDAR point cloud, pedestrian targets in the visible light image, and pedestrian targets in the infrared image.

[0009] Step 4: Based on the decision-level fusion algorithm, fuse the pedestrian targets detected by LiDAR, visible light camera and infrared camera to obtain the final pedestrian target detection result.

[0010] The beneficial effects of this invention are as follows:

[0011] To address the limitations and shortcomings of existing multi-source heterogeneous sensor fusion detection methods for pedestrian detection, this invention proposes a pedestrian target detection method based on radar, video, and infrared decision-level fusion; the multi-source heterogeneous sensors include lidar, visible light camera, and infrared camera.

[0012] First, a hardware and software platform for detection was designed based on practical requirements. Spatiotemporal consistency among different sensors was constructed through mathematical principles and software design. Complex-YOLOv4 was selected as the target detection method for LiDAR, and YOLOv7 was selected as the target detection method for visible light and infrared cameras. Based on this, training datasets were selected and training strategies were adjusted. Finally, a multi-source heterogeneous information fusion method was used to process pedestrian targets detected by different sensors. Based on the decision-level fusion algorithm CLOCs, decision-level fusion pedestrian detection was achieved. The method proposed in this invention can improve the accuracy of pedestrian fusion detection and has certain significance for the development of intelligent driving perception technology in the ADAS field. Attached Figure Description

[0013] Figure 1 This is a flowchart of the present invention;

[0014] Figure 2 This is a diagram showing the hardware connections of the platform.

[0015] Figure 3 This is a schematic diagram of the physical assembly of the platform;

[0016] Figure 4 The diagram illustrates the data release frequency before and after alignment. (a) shows the data release frequency before alignment, and (b) shows the data release frequency after alignment.

[0017] Figure 5 Image data diagrams required for camera intrinsic parameter calibration;

[0018] Figure 6 Images and point cloud data used for radar calibration;

[0019] Figure 7A diagram showing the location and matching relationship of feature points in the original image for a particular registration;

[0020] Figure 8 This is a projection of the lidar point cloud onto a visible light image.

[0021] Figure 9 This is a registration diagram of an infrared image and a visible light image;

[0022] Figure 10 The image shows the results of point cloud data inspection using Complex-YOLOv4;

[0023] Figure 11 This is a diagram showing the results of detection using Yolov7 on visible light image data.

[0024] Figure 12 The image shows the results of detecting infrared image data using Yolov7.

[0025] Figure 13 The images show a comparison of pedestrian detection results under good lighting conditions. (a) shows the result of fusion detection using the method of this invention. (b) shows the projection of the detection result using LiDAR alone onto the visible light image (the smallest bounding rectangle of the eight points of the detection bounding box in three-dimensional space projected onto the image plane). (c) shows the result of detection using a visible light camera alone. (d) shows the mapping of the detection result using an infrared camera alone onto the visible light image.

[0026] Figure 14 The images show a comparison of pedestrian detection results under poor lighting conditions. (a) shows the result of fusion detection using the method of this invention. (b) shows the projection of the detection result using LiDAR alone onto the visible light image (the smallest bounding rectangle of the eight points of the detection bounding box in three-dimensional space projected onto the image plane). (c) shows the result of detection using a visible light camera alone. (d) shows the mapping of the detection result using an infrared camera alone onto the visible light image. Detailed Implementation

[0027] Specific Implementation Method 1: The specific process of this implementation method for pedestrian target detection based on radar video infrared decision-level fusion is as follows:

[0028] Step 1: Set up a pedestrian detection platform, which includes LiDAR, visible light cameras, and infrared cameras;

[0029] Step 2: Perform spatiotemporal alignment on the lidar, visible light camera, and infrared camera;

[0030] Step 3: Based on the spatiotemporally aligned LiDAR, visible light camera, and infrared camera, perform target detection on pedestrian targets in the LiDAR point cloud, pedestrian targets in the visible light image, and pedestrian targets in the infrared image.

[0031] Step 4: Based on the decision-level fusion algorithm, fuse the pedestrian targets detected by LiDAR, visible light camera and infrared camera to obtain the final pedestrian target detection result.

[0032] Specific Implementation Method Two: This implementation method differs from Specific Implementation Method One in that step one involves building a pedestrian detection platform, which includes a lidar, a visible light camera, and an infrared camera; the specific process is as follows:

[0033] Based on actual testing requirements, this invention first designed a hardware and software platform for testing.

[0034] In terms of software, we selected the Ubuntu 18.04 operating system, Python programming language, PyCharm integrated development environment, and used the Torch deep learning framework, OpenCV open-source vision library, as well as the SDK and robot operating system ROS that are compatible with the Pandar40P LiDAR, LT-H8179 visible light camera and COIN417G2 infrared camera.

[0035] In terms of hardware, the Pandar40P LiDAR, LT-H8179 visible light camera, and COIN417G2 infrared camera were selected as data acquisition modules. The Pandar40P LiDAR, LT-H8179 visible light camera, and COIN417G2 infrared camera were used to acquire point cloud data, visible light images, and infrared images, respectively. At the same time, the Jetson Xavier development platform was used as the data processing module (processing steps two, three, and four) to ensure efficient data processing and analysis.

[0036] The main parameters of each sensor are shown in Tables 1 to 3. The platform hardware wiring diagram and the platform physical assembly diagram are shown in Tables 1 to 3. Figure 2 , Figure 3 As shown.

[0037] Table 1. Main parameters of Pandar40P LiDAR

[0038]

[0039]

[0040] Table 2 Main parameters of the LT-H8179 visible light camera

[0041] integrated circuit Sony IMX179 Pixel size 1.4μm Image sensor size 1 / 3.2 inch Resolution and frame rate 640×480@30Hz

[0042] Table 3 Main parameters of the COIN417G2 infrared camera

[0043] detector Uncooled vanadium oxide Pixel size 17μm spectrum 8~14μm Lens parameters 9.1mm Resolution and frame rate 384×288@25Hz

[0044] Figure 2 In the text, Pandar40P is the model number of the Hesai lidar system.

[0045] LT-H8179: Industrial camera model;

[0046] COIN417G2: Infrared camera model;

[0047] GBE usually stands for "Gigabit Ethernet". This is a network technology that provides high-speed data transmission and is suitable for both Local Area Networks (LANs) and Wide Area Networks (WANs).

[0048] MDI: Usually stands for "Medium Dependent Interface", which is a network technology used for the physical layer interface in Ethernet. In Ethernet, MDI is used to connect network devices (such as computers, switches, routers) to the transmission medium (such as twisted pair, fiber optic).

[0049] RJ-45: A commonly used network connector standard, often used to connect network devices such as computers, routers, switches, and network cables;

[0050] HDMI: an abbreviation for "High Definition Multimedia Interface," is a digital audio and video interface standard used to connect high-definition televisions, computer monitors, projectors, and other multimedia devices.

[0051] [E]DP / HDMI: This term usually refers to the switching or conversion function between DisplayPort and HDMI interfaces;

[0052] LDDR4x: Low Power DDR4x (LPDDR4x) is a type of low-power memory used in mobile devices;

[0053] QSPI NOR: A type of non-volatile flash memory based on a quad serial peripheral interface; it is a common type of memory used in embedded systems, embedded applications, and microcontrollers.

[0054] eMMC is an abbreviation for "Embedded Multi Media Card," which is a storage solution that integrates a storage controller and flash memory chips.

[0055] Power Subsystem: Power Management System;

[0056] PMIC is an abbreviation for "Power Management Integrated Circuit." It is a chip that integrates multiple power management functions to provide power and energy management for electronic devices.

[0057] USB is an abbreviation for "Universal Serial Bus".

[0058] Figure 2 This diagram is based on the official hardware schematics of the Nvidia Jetson Xavier and only illustrates the wiring diagrams for the hardware components of this invention.

[0059] The other steps and parameters are the same as in Specific Implementation Method 1.

[0060] Specific Implementation Method Three: This implementation method differs from Specific Implementation Method One or Two in that step two involves spatiotemporal alignment of the lidar, visible light camera, and infrared camera; the specific process is as follows:

[0061] Synchronization of timestamps and precise geometric relationships among multi-source heterogeneous sensors are crucial prerequisites and foundations for the implementation of fusion detection algorithms. These sensors include LiDAR, visible light cameras, and infrared cameras. These two factors directly affect the validity and consistency of sensor data, thus impacting the accuracy and reliability of the fusion detection algorithm. Time alignment of data acquired from multi-source heterogeneous sensors can be achieved through software design, and coordinate transformation relationships between the data can be determined through joint calibration.

[0062] Step 21: Time-align the data obtained from the lidar, visible light camera, and infrared camera;

[0063] Step 22: Spatial alignment of the data obtained from the lidar, visible light camera, and infrared camera.

[0064] Other steps and parameters are the same as in specific implementation method one or two.

[0065] Specific Implementation Method Four: This implementation method differs from Specific Implementation Methods One to Three in that, in step two-one, time alignment is performed on the data obtained by the lidar, visible light camera, and infrared camera; the specific process is as follows:

[0066] Time alignment refers to unifying the sampling times of different sensors to a reference standard time, thereby synchronizing the starting point of the sampling time and the frequency of the data information. This invention employs a soft time synchronization method, using a synchronization node program written in ROS to implement the time alignment process.

[0067] Let the Pandar40P lidar have an F... lidar Point cloud data is published at a frequency of 10Hz. The dimension is (n l ,4), where n l This indicates the number of points in each frame of LiDAR point cloud data; 4 represents... It contains the dimensions of point cloud data, including location information (x, y, z) and reflection intensity; the LT-H8179 visible light camera uses F... camera Visible light image data is released at a frequency of 30Hz. The dimension is (w c ,h c ,3), where w c and h c Represents the length and width of a visible light image, 3 indicates It is a color image containing three RGB channels; the COIN417G2 infrared camera uses an F... ir Infrared image data is released at a frequency of 30Hz. The dimension is (w ir ,h ir ,1), where w ir and h ir These represent the length and width of the infrared image, respectively, with 1 indicating... It is a grayscale image that contains only one channel of grayscale value;

[0068] Meanwhile, at any time t, the synchronization node continuously receives point cloud data released by the Pandar40P lidar. Visible light image data released by the LT-H8179 visible light camera Infrared image data released by the COIN417G2 infrared camera During synchronization, first set up a receiver. and The three types of data are stored in separate storage spaces, and the data stored in each storage space is updated in real time with the data obtained by the lidar, visible light camera and infrared camera. The cache will only send out synchronization data when all three storage areas have obtained the data. The synchronization period T for the synchronization node to send synchronization data is selected according to formula (1).

[0069]

[0070] In the ROS system, the topics and posting cycles published by the data acquisition nodes of each sensor before synchronization are as follows: Figure 4 As shown in (a), after time alignment via soft synchronization, the topics and publication cycles published by the synchronized nodes are as follows: Figure 4 As shown in (b):

[0071] Figure 4 In (a), the camera_info and image_raw nodes represent the camera intrinsic parameter matrix and visible light image data published by the visible light camera data acquisition node. infrared_image_raw represents the infrared image data released by the infrared camera data acquisition node. hesai_points represents the point cloud data published by the LiDAR data acquisition node. Figure 4 In (b), sync / camera_info, sync / image_raw, sync / infrared_image_raw, and sync / hesai_points represent the synchronization data published by the synchronization node. It can be seen that after time alignment is achieved through soft synchronization, the frequency at which the synchronization node publishes synchronization data is fixed at 10Hz.

[0072] The other steps and parameters are the same as those in one of the specific implementation methods one to three.

[0073] Specific Implementation Method Five: This implementation method differs from Specific Implementation Methods One to Four in that, in step two-two, spatial alignment is performed on the data obtained by the lidar, visible light camera, and infrared camera; the specific process is as follows:

[0074] In addition to achieving precise time-series synchronization during data acquisition, multi-source heterogeneous sensors also require synchronization of the spatial relationships between different sensors. This invention involves two main parts: joint calibration of a lidar and a visible light camera, and joint calibration of an infrared camera and a visible light camera.

[0075] Step 221: Joint calibration of LiDAR and visible light camera:

[0076] The joint calibration of lidar and visible light camera is essentially about solving the spatial positional relationship between these heterogeneous sensors. By using the transformation matrix between the pixel set in the two-dimensional image and the corresponding points on the surface of the three-dimensional object, the image and point cloud are represented in a unified coordinate system.

[0077] Joint calibration includes the calibration of the intrinsic parameters of the visible light camera and the calibration of the extrinsic parameters between the visible light camera and the lidar;

[0078] Step 222: Joint calibration between the infrared camera and the visible light camera.

[0079] The other steps and parameters are the same as those in one of the specific implementation methods one to four.

[0080] Specific Implementation Method Six: This implementation method differs from Specific Implementation Methods One to Five in that the joint calibration of the lidar and the visible light camera in step 221 includes the calibration of the external parameters between the visible light camera and the lidar, as well as the calibration of the internal parameters of the visible light camera.

[0081] The specific process is as follows:

[0082] Step 2211: Calibrate the intrinsic parameters of the visible light camera;

[0083] The intrinsic parameters of a visible light camera are calibrated using the camera-calibration tool in the ROS toolbox. The specific steps are as follows:

[0084] a) Acquire a set of calibration board image data for calibrating the camera. The calibration board images cover different angles, distances, and postures. Multiple acquisitions of the calibration board images can fill most of the camera's field of view, and the corner points of the calibration board are clearly visible in the images. The acquired calibration board image data is as follows: Figure 5 As shown.

[0085] Obtain calibration board information for calibrating the camera, including the number of rows and columns of corner points on the calibration board and the actual dimensions between the corner points;

[0086] b) Run the camera-calibration tool and obtain the camera's intrinsic parameters based on the visible light camera imaging model in the camera-calibration tool;

[0087] The intrinsic parameters of a camera include its eigenvalue matrix, radial distortion coefficient, and tangential distortion coefficient.

[0088] The radial distortion coefficients are taken from the first two in the array; since the distortion of industrial cameras is relatively small, only the first two are taken.

[0089] The main internal parameters of the camera are shown in Table 4:

[0090] Table 4 shows the internal parameters of the visible light camera obtained from calibration.

[0091]

[0092]

[0093] Step 2212: Calibrate the extrinsic parameters between the visible light camera and the lidar; the specific process is as follows:

[0094] The extrinsic parameter calibration between the visible light camera and the lidar is determined by the rotation matrix R. 3×3 Translation matrix T 3×1 composition;

[0095] The process of extrinsic parameter calibration involves the image pixel coordinate system (u,v), the image physical coordinate system (x,y), and the camera coordinate system (X). C ,Y C Z C ) and lidar coordinate system (X L ,Y L Z L The conversion relationship between )

[0096] The process of transforming from the image pixel coordinate system (u,v) to the image physical coordinate system (x,y) is represented by formula (2):

[0097]

[0098] In the formula, u0 and v0 represent the center pixel coordinates of the image, and dx and dy represent the size of the camera's image sensor.

[0099] From the image physical coordinate system (x,y) to the visible light camera coordinate system (X... C ,Y C Z C The conversion process is represented by formula (3):

[0100]

[0101] In the formula, f represents the focal length of the camera;

[0102] Visible light camera coordinate system (X c ,Y c Z c ) and the lidar coordinate system (X) L ,Y L Z L The following relationship exists:

[0103]

[0104] In the formula, R 3×3 and T 3×1 These represent the rotational and translational relationships between the camera and the lidar, respectively.

[0105] Rotational torque R 3×3 From the lidar coordinate system (X) L ,Y L Z LThe matrix is ​​composed of the pitch angle roll around the x-axis, the roll angle pitch around the y-axis, and the yaw angle yaw around the z-axis of the next point.

[0106] Translation torque T 3×1 From the lidar coordinate system (X) L ,Y L Z L The translation distances Δx, Δy, and Δz relative to the camera coordinate system constitute (X) L +Δx=X C Y L +Δy=Y C Z L +Δz=Z C );

[0107] Due to the refraction of light by different parts of the lens and minute offsets or inaccurate assembly between lens elements, the scale ratio of the image center and surrounding areas changes. Points in the lens's physical coordinate system experience radial and tangential distortion. Therefore, a point (x, y) in the physical coordinate system will be affected by distortion and change to (x...). distorted ,y distorted ), x distorted Equation (5) represents: y distorted This can be expressed by formula (6):

[0108]

[0109] In the formula, r represents the distance from the optical center. 2 =x 2 +y 2 ;

[0110] k1, k2, k3, k4, k5, and k6 represent radial distortion coefficients. Since the distortion of industrial cameras is relatively small, the values ​​of k3, k4, k5, and k6 are approximately 0. In this invention, only k1 and k2 are selected.

[0111] d1 and d2 represent the tangential distortion coefficients;

[0112] The radial and tangential distortion coefficients were obtained through the intrinsic parameters calibration of the visible light camera.

[0113] Substitute formulas (5) and (6) into formula (2) ((x) distorted ,y distorted Replace (x,y) to get the pixel coordinates (u) in the image pixel coordinate system. distorted ,v distorted );

[0114] Let f x =f / dx, f y=f / dy, pixel coordinates in the image pixel coordinate system (u distorted ,v distorted ) and the lidar coordinate system (X) L ,Y L Z L The conversion process (from formula 2 to formula 6) is represented by formula (7):

[0115]

[0116] In the formula, M1 represents the intrinsic parameters of the visible light camera;

[0117] M2 represents the rotation and translation relationship between the lidar and the visible light camera, i.e., the extrinsic parameter matrix. (unknown);

[0118] f x f y Indicates intermediate variables;

[0119] Use the cam-lidar-calibration tool in the ROS toolbox to perform extrinsic parameter matrix analysis between the visible light camera and the lidar. The calibration (to obtain M2) is performed using the following steps:

[0120] a) Simultaneously sample the calibration board using a lidar and a visible light camera, with the calibration board images covering different angles, distances, and orientations;

[0121] During sampling, the 3D pose of the calibration board needs to be continuously changed, and the sampled images and point cloud data are as follows: Figure 7 As shown;

[0122] b) Regarding the sampled calibration plate images:

[0123] The `cv::find Chessboard Corners` method from the OpenCV open-source vision library was used to extract the corner points on the calibration board image, and the pixel coordinates of the calibration board center in the calibration board image were calculated. and normal vector

[0124] Regarding the sampled calibration board point cloud data:

[0125] The sampled calibration board point cloud data is separated from a complete frame of data acquired by the LiDAR using a preset spatial threshold; then, 3D RANSAC is used to fit the boundary of the calibration board, and the center three-dimensional coordinates of the calibration board are calculated. and normal vector i = 1, 2, ..., N represents the sequence number of the collected data;

[0126] c) Repeat steps a) and b) M times to obtain M sets.

[0127] Group M The intrinsic parameters of the visible light camera obtained from calibration are substituted into the transformation relationship between the lidar coordinate system and the camera coordinate system (4), and then optimized by a genetic algorithm to obtain the transformation matrix R representing the positional relationship between the lidar coordinate system and the camera coordinate system. 3×3 and T 3×1 ;

[0128] The specific data is shown in Table 5.

[0129] Table 5 shows the external parameter matrix parameters obtained from calibration.

[0130]

[0131] The other steps and parameters are the same as those in one of the specific implementation methods one to five.

[0132] Specific Implementation Method Seven: This implementation method differs from Specific Implementation Methods One through Six in that the joint calibration between the infrared camera and the visible light camera in step two-two-two is as follows:

[0133] Considering that the information obtained by the infrared camera and the visible light camera are both two-dimensional images, and that the installation positions and orientations of the two cameras are very close during the platform construction process, this invention believes that the two images can be registered by calculating the homography matrix.

[0134] The specific steps are as follows:

[0135] Step 2221: Preprocess the visible light and infrared images; the specific process is as follows:

[0136] For infrared images:

[0137] The infrared image is inverted, and the inverted infrared image is then denoised using a Gaussian filter with a kernel size of 3.

[0138] For visible light images:

[0139] Denoising of visible light images is achieved by median filtering with a kernel size of 3.

[0140] Step 2: Use the Canny operator to perform edge detection on the preprocessed visible light image and infrared image to obtain the edges of the visible light image and infrared image;

[0141] This step can reduce the differences between images of different modalities and improve the similarity of boundaries in images of different modalities so that the extracted feature points can be matched later.

[0142] Step 2223: Use the ORB algorithm to extract and describe the feature points of the edges in the visible light and infrared images (the relationship between the feature points and the surrounding pixels);

[0143] The feature points at the edges of the visible light image and the infrared image are matched using a brute-force matching method to obtain the matched feature points.

[0144] The RANSAC algorithm is used to filter the matched feature points to obtain matching pairs; thus, more accurate feature point matching pairs are obtained.

[0145] The homography matrix between feature points is calculated by matching pairs;

[0146] Step 2224: Repeat steps 2221 to 2223 M' times, and calculate the average value of the homography matrix as the conversion relationship between the visible light image and the infrared image;

[0147] The value is 5≤M′≤10.

[0148] When using the above relationships for registration between an infrared camera and a visible light camera, the extracted feature points' positions and matching relationships on the original image are as follows: Figure 8 As shown.

[0149] The average value of the homography matrix obtained from 5 experiments can be expressed by formula (8):

[0150]

[0151] The projection of the lidar point cloud onto the visible light image is obtained through the above two alignment processes, as shown below. Figure 9 As shown, the fusion effect of infrared images in visible light images is as follows: Figure 10 As shown.

[0152] The other steps and parameters are the same as those in one of the specific implementation methods one to six.

[0153] Specific Implementation Method Eight: This implementation method differs from Specific Implementation Methods One to Seven in that, in step three, based on the spatiotemporally aligned LiDAR, visible light camera, and infrared camera, target detection is performed on pedestrian targets in the LiDAR point cloud, pedestrian targets in the visible light image, and pedestrian targets in the infrared image; the specific process is as follows:

[0154] After completing the construction of the data acquisition platform and the spatiotemporal alignment of the data, it is necessary to comprehensively consider the accuracy and speed of pedestrian detection by intelligent driving vehicles, select pedestrian target detection methods based on the data acquired by different sensors, and introduce the selection of detection networks for each sensor, training of detection models, and schematic diagrams of detection results in this invention.

[0155] Step 31: Obtain the trained Complex-YOLOv4 detection network based on deep learning; the specific process is as follows:

[0156] The KITTI dataset was selected as the training set.

[0157] The KITTI dataset's lidar point cloud data is used as the input to the deep learning-based Complex-YOLOv4 detection network, and the pedestrian targets in the KITTI dataset's lidar point cloud are used as the output of the deep learning-based Complex-YOLOv4 detection network. The deep learning-based Complex-YOLOv4 detection network is trained until it converges, and the trained deep learning-based Complex-YOLOv4 detection network is obtained.

[0158] Step 3.2: Obtain the trained Yolov7 detection network for visible light images; the specific process is as follows:

[0159] The visible light images in the COCO dataset are used as the input to the YOLOv7 detection network, and the pedestrian targets in the visible light images in the COCO dataset are used as the output of the YOLOv7 detection network. The YOLOv7 detection network is pre-trained until it converges to obtain the pre-trained YOLOv7 detection network.

[0160] The visible light images in the KITTI dataset are used as the input of the pre-trained Yolov7 detection network, and the pedestrian targets in the visible light images in the KITTI dataset are used as the output of the pre-trained Yolov7 detection network. The pre-trained Yolov7 detection network is trained until it converges, and a trained Yolov7 detection network for visible light images is obtained.

[0161] Step 3: Obtain the trained Yolov7 detection network for infrared images; the specific process is as follows:

[0162] Infrared images from the SCUT dataset are used as input to the YOLOv7 detection network, and pedestrian targets in the infrared images from the SCUT dataset are used as output to the YOLOv7 detection network. The YOLOv7 detection network is pre-trained until it converges to obtain a pre-trained YOLOv7 detection network.

[0163] The acquired infrared image is used as the input of the pre-trained YOLOv7 detection network, and the pedestrian target in the acquired infrared image is used as the output of the pre-trained YOLOv7 detection network. The pre-trained YOLOv7 detection network is trained until convergence, and a trained YOLOv7 detection network for infrared images is obtained.

[0164] Considering that the data acquired by the infrared camera is a single-channel grayscale image and that the infrared camera itself has a low resolution, the following modifications were made during the training process:

[0165] (1) Cancel color transformation and scale transformation (in the hyperparameter settings under the yolov7 project file, you can set the flag of some data augmentation methods to zero) to prevent training samples that do not conform to the imaging rules of infrared images and targets that are too small.

[0166] (2) The loss function of the Yolov7 detection network is:

[0167] Loss = α × loss box +β×loss obj +γ×loss cls

[0168] Where Loss is the overall loss function value of YOLOv7;

[0169] The lossbox is the loss for regressing the target location coordinates;

[0170] lossobj is the loss of the target confidence;

[0171] losscls is the loss based on the confidence level of the target class;

[0172] α, β, and γ are the corresponding weighting coefficients; α = 0.1; γ = 0; β = 0.2;

[0173] In this invention, since only pedestrians are considered as a target, γ = 0;

[0174] Furthermore, in this invention, the loss caused by the loss of target confidence is appropriately reduced by adjusting β from 0.3 to 0.2, thereby reducing the impact on the training results caused by the scarcity of pedestrian targets in the test images in the training data.

[0175] The loss caused by the presence or absence of the target in the loss function was appropriately reduced (the weight in the loss function was changed from 0.3 to 0.2). The weight of the loss was adjusted to reduce the impact of the lack of pedestrian targets in the test images in the training data on the training results.

[0176] (3) Copy the single-channel infrared grayscale image three times and fuse it into a three-channel infrared image for training to ensure the consistency of the input data format of the Yolov7 network.

[0177] For visible light cameras and infrared cameras, since the data acquired by both are two-dimensional digital images, it is possible to use data from different modalities to train two detection models with the same structure. This invention selects the YOLOv7 detection network to detect pedestrian targets in visible light images and infrared images. For the YOLOv7 detection network for visible light image data, the COCO dataset is used for training and the KITTI dataset is used for fine-tuning. For the YOLOv7 detection network for infrared image data, the SCUT dataset is used for initial training and then further training is performed using infrared image annotation data collected by the platform.

[0178] Steps 3 and 4: Based on the trained deep learning-based Complex-YOLOv4 detection network, pedestrian targets in the LiDAR point cloud under test are detected. The expression is:

[0179]

[0180] Among them, B l,i =[x i ,y i ,z i ,w i ,l i ,h i ,θ i The figure represents the 3D bounding box of a pedestrian target in the lidar point cloud data, including the center coordinates x of the bounding box. i y i z i The length, width, and height of the bounding box (w) i l i h i And the heading angle θ of the bounding box in the lidar coordinate system. i C l,i This represents the confidence level of the pedestrian target, where n1 represents the number of pedestrians detected in a given frame of point cloud data. This represents the lidar point cloud data collected at time t, and Complex_Yolo(·) represents the trained deep learning-based Complex-YOLOv4 detection network for the point cloud data.

[0181] Complex-YOLOv4 first projects point cloud data onto the XOY plane and encodes it into a two-dimensional image. Then, it identifies pedestrian targets in the encoded two-dimensional image and reconstructs the detected pedestrians back into three-dimensional space using joint calibration relationships between sensors. The results of using Complex-YOLOv4 to detect point cloud data are shown below. Figure 11 As shown.

[0182] Step 3.5: Based on the Yolov7 detection network trained for visible light images, detect pedestrian targets in the visible light image to be tested. The expression is:

[0183]

[0184] Among them, B c,j = [x1, y1, x2, y2] represents the 2D bounding box of a pedestrian target in visible light image data, including the coordinates x1, y1 of the upper left corner and x2, y2 of the lower right corner; C c,j n represents the confidence level of the pedestrian target, and n2 represents the number of pedestrians detected in a certain frame of image data. This represents the visible light image data acquired at time t, YOLO. c (·) represents a well-trained Yolov7 detection network for visible light images;

[0185] The results of detection using Yolov7 on visible light image data are as follows: Figure 12 As shown.

[0186] Step 36: Based on the Yolov7 detection network trained for infrared images, detect pedestrian targets in the infrared image to be tested. The expression is:

[0187]

[0188] Among them, B ir,k = [x1,y1,x2,y2] represents the 2D bounding box of a pedestrian target in infrared image data, including the coordinates x1, y1 of the upper left corner and x2, y2 of the lower right corner; C ir,k n represents the confidence level of the pedestrian target; n3 represents the number of pedestrians detected in a certain frame of image data. Represents the infrared image data acquired at time t, YOLOv3 ir (·) represents a trained Yolov7 detection network for infrared images.

[0189] The results of using Yolov7 to detect infrared image data are as follows: Figure 13 As shown.

[0190] The other steps and parameters are the same as those in any of the specific implementation methods one to seven.

[0191] Specific Implementation Method Nine: This implementation method differs from Specific Implementation Methods One through Eight in that step four involves fusing pedestrian targets detected by LiDAR, visible light cameras, and infrared cameras using a decision-level fusion algorithm to obtain the final pedestrian target detection result; the specific process is as follows:

[0192] After obtaining the detection results of pedestrian targets from different sensors, considering the requirements for speed and accuracy in pedestrian detection tasks, this invention uses the decision-level fusion algorithm CLOCs (Camera-LiDAR Object CandidatesFusion for 3D Object Detection) to fuse pedestrian targets detected by different sensors. The CLOCs algorithm provides a low-complexity multimodal fusion framework. This method is data-driven and establishes the connection between targets detected by different modal sensors through training with a large number of samples. Its time and computational costs are much lower than those of data-level fusion and feature-level fusion methods. It also allows different modal detection algorithms to be trained using separate data, which significantly improves the performance of target detection algorithms.

[0193] Step 4.1. Let P be the set of candidate detection boxes for the visible light image and the infrared image without using non-maximum suppression. 2D This can be expressed by formula (12):

[0194]

[0195] in This represents the i-th candidate detection box in the visible light image and the infrared image. This represents the position of the i-th candidate detection box in the visible light image and the infrared image. represents the confidence level of the i-th candidate detection box in the visible light image and the infrared image; k represents the total number of candidate detection boxes in the visible light image and the infrared image;

[0196] Step 4.2. Let P be the set of candidate detection boxes for the lidar point cloud without using non-maximum suppression. 3D This can be expressed by formula (13):

[0197]

[0198] in This represents the j-th candidate detection box in the LiDAR point cloud. This represents the position of the j-th candidate detection box in the LiDAR point cloud. The confidence score of the j-th candidate detection box in the LiDAR point cloud is represented by ; n represents the total number of candidate detection boxes in the LiDAR point cloud.

[0199] Step 43, using P 2D and P 3D Construct a set of joint consistency detection candidate tensors T with dimensions k×n×4;

[0200] Input the tensor T into the trained decision-level fusion model, and the trained decision-level fusion model outputs the pedestrian targets detected by different sensors;

[0201] The candidate detection box set P of the visible light image and infrared image to be tested 2D And the candidate detection box set P of the LiDAR point cloud. 3D Input a pre-trained decision-level fusion model, and output a set of 3D bounding boxes P. res ;

[0202] P res By using nonmaximum suppression and then filtering with a preset target confidence threshold, the final pedestrian target is obtained.

[0203] The other steps and parameters are the same as those in any of the specific implementation methods one to seven.

[0204] Specific Implementation Method Ten: This implementation method differs from Specific Implementation Methods One to Nine in that, in step four-three, P is utilized. 2D and P 3D Construct a set of joint consistency detection candidate tensors T with dimensions k×n×4; the specific process is as follows:

[0205] Each element of tensor T is represented by formula (14):

[0206]

[0207] Where d j Represents the lidar coordinate system (X) L ,Y L Z L The j-th candidate detection box of the lidar point cloud in the plane and the lidar coordinate system (X) L ,Y L Z L Normalized distance between the origin;

[0208] k represents the total number of candidate detection boxes in the visible light image and the infrared image;

[0209] n represents the total number of candidate detection boxes in the LiDAR point cloud;

[0210] 4 indicates that each tensor contains 4 elements;

[0211] IOU i,j This represents the j-th candidate detection box in the lidar point cloud. After being projected onto the 2D image, it is compared with the i-th candidate detection box in the visible light image and the infrared image. The crossover and union ratio between them;

[0212] This represents the confidence level of the i-th candidate detection box in the visible light image and the infrared image;

[0213] This represents the confidence level of the j-th candidate detection box in the LiDAR point cloud;

[0214] In step four, tensor T is input into the trained decision-level fusion model, and the trained decision-level fusion model outputs a fused pedestrian target detected by different sensors; the specific process is as follows:

[0215] C res =CLOCs(T) (15)

[0216] in, Let CLOCs(·) represent the confidence scores of the fused n candidate detection boxes, and let CLOCs(·) denote the trained decision-level fusion model.

[0217]

[0218] Among them, P res This represents the set of n candidate detection boxes after fusion. This represents the j-th candidate detection box in the lidar point cloud after decision-level fusion adjustment. This indicates the position of the j-th candidate detection box in the LiDAR point cloud after decision-level fusion adjustment. Let represent the confidence level of the j-th candidate detection box in the LiDAR point cloud after decision-level fusion adjustment;

[0219] P res By using non-maximum suppression and then filtering with a preset target confidence threshold, the final pedestrian target is obtained.

[0220] The process of obtaining the trained decision-level fusion model CLOCs(·) is as follows:

[0221] The decision-level fusion model CLOCs(·) is trained by using the target detection boxes in the labeled 2D images and 3D point clouds until the model converges, thus obtaining the trained decision-level fusion model CLOCs(·); and the decision-level fusion of detection results of different modalities is completed.

[0222] The other steps and parameters are the same as those in any of the specific implementation methods one to nine.

[0223] In this invention, the fusion target considered is a target that can be detected by all three sensors. Targets that exist only within the detection range of one or two sensors, such as those with a slightly wider field of view than an infrared camera or a sensory domain that a lidar cannot perceive (targets on the side or behind the vehicle), are not considered in this invention. In actual detection, considering that using only the detection results from the visible light camera as input, single-modal image detection often cannot effectively reflect the presence of pedestrians in the driving environment of intelligent vehicles due to adverse lighting conditions such as backlight, glare, and low light at night, this invention uses the union of the visible light image candidate detection box set (without non-maximum suppression) and the infrared image candidate detection box set (without non-maximum suppression) from step three as the CLOCs algorithm candidate detection box set P. 2D Although this will introduce more pedestrian target candidates, more accurate pedestrian detection results can still be obtained by complementing the detection results of different modal data such as infrared images, visible light images and lidar point clouds.

[0224] The final pedestrian detection results obtained by the method of this invention are as follows: Figure 14 As shown, Figure 14 (a) to (d) show the projections of the detection results obtained using the method of this invention, the detection results obtained using LiDAR alone, and the detection results obtained using a single infrared camera into the visible light image (the smallest bounding rectangle of the eight points of the detection bounding box in three-dimensional space projected onto the image plane), the detection results obtained using a single visible light camera, and the detection results obtained using a single infrared camera into the visible light image. It can be seen that, through the method of this invention, false targets detected by LiDAR are effectively suppressed after decision-level fusion, and the positions and sizes of other valid pedestrian targets remain consistent with those detected by LiDAR, while the confidence level of the targets is significantly improved.

[0225] Pedestrian detection results under poor lighting conditions, such as Figure 14 As shown.

[0226] As can be seen, compared with using a single-modality sensor for detection, the method of the present invention can comprehensively utilize the information provided by different modal sensors, thereby making up for the limitations of a single modality in a specific environment or condition and improving the accuracy of target detection. At the same time, single-modality pedestrian target detection may be affected by factors such as sensor detection characteristics and changes in illumination, while the method of the present invention can fuse information from multiple modalities to enhance the robustness of the detection system to illumination, enabling it to operate stably in various complex illumination scenarios.

[0227] This invention may have other embodiments. Without departing from the spirit and essence of this invention, those skilled in the art can make various corresponding changes and modifications according to this invention, but these corresponding changes and modifications should all fall within the protection scope of the appended claims.

Claims

1. A pedestrian target detection method based on radar video infrared decision-level fusion, characterized in that: The specific process of the method is as follows: Step 1: Set up a pedestrian detection platform, which includes LiDAR, visible light cameras, and infrared cameras; Step 2: Perform spatiotemporal alignment on the lidar, visible light camera, and infrared camera; Step 3: Based on the spatiotemporally aligned LiDAR, visible light camera, and infrared camera, perform target detection on pedestrian targets in the LiDAR point cloud, pedestrian targets in the visible light image, and pedestrian targets in the infrared image. Step 4: Based on the decision-level fusion algorithm, fuse the pedestrian targets detected by LiDAR, visible light camera and infrared camera to obtain the final pedestrian target detection result; In step four, the pedestrian targets detected by lidar, visible light camera and infrared camera are fused based on the decision-level fusion algorithm to obtain the final pedestrian target detection result; The specific process is as follows: Step 4.

1. Define the candidate detection box sets for the visible light and infrared images without using non-maximum suppression. This can be expressed by formula (12): (12) in The first part represents the visible light image and the infrared image. One candidate detection box, The first part represents the visible light image and the infrared image. The position of each candidate detection box. The first part represents the visible light image and the infrared image. Confidence of each candidate detection box; This represents the total number of candidate detection boxes in the visible light image and the infrared image; Step 4.2: Define the set of candidate detection boxes for the LiDAR point cloud without using nonmaximum suppression. This can be expressed by formula (13): (13) in The first point cloud of the lidar is represented by the... One candidate detection box, The first point cloud of the lidar is represented by the... The position of each candidate detection box. The first point cloud of the lidar is represented by the... Confidence of each candidate detection box; This represents the total number of candidate detection boxes in the lidar point cloud; Step 43, Utilize and Construct a set of dimensions Consistency Joint Detection Candidate Tensors ; tensor Input the trained decision-level fusion model, and the trained decision-level fusion model outputs the pedestrian targets detected by different sensors; Set up candidate detection boxes for the visible light image and infrared image to be tested. and the candidate detection box set of LiDAR point cloud. Input a pre-trained decision-level fusion model, and the pre-trained decision-level fusion model outputs a set of 3D detection boxes. ; Will By using non-maximum suppression and then filtering with a preset target confidence threshold, the final pedestrian target is obtained. In step four three, the use and Construct a set of dimensions Consistency Joint Detection Candidate Tensors ; The specific process is as follows: tensor Each element is represented by formula (14): (14) in Representing the lidar coordinate system The first point cloud of lidar in the plane Candidate detection boxes and LiDAR coordinate system Normalized distance between origins; This represents the total number of candidate detection boxes in the visible light image and the infrared image; This represents the total number of candidate detection boxes in the lidar point cloud; 4 indicates that each tensor contains 4 elements; The first point cloud of the lidar is represented by the... Candidate detection boxes After being projected into a two-dimensional image, it is compared with the visible light image and the infrared image. Candidate detection boxes The crossover ratio between them; The first part represents the visible light image and the infrared image. Confidence of each candidate detection box; The first point cloud of the lidar is represented by the... The confidence level of each candidate detection box.

2. The pedestrian target detection method based on radar video infrared decision-level fusion according to claim 1, characterized in that: Step one involves building a pedestrian detection platform, which includes a lidar unit, a visible light camera, and an infrared camera; the specific process is as follows: The Pandar40P LiDAR, LT-H8179 visible light camera, and COIN417G2 infrared camera were selected as data acquisition modules. The Pandar40P LiDAR, LT-H8179 visible light camera, and COIN417G2 infrared camera were used to acquire point cloud data, visible light images, and infrared images, respectively.

3. The pedestrian target detection method based on radar video infrared decision-level fusion according to claim 2, characterized in that: In step two, the LiDAR, visible light camera, and infrared camera are spatiotemporally aligned. The specific process is as follows: Step 21: Time-align the data obtained from the lidar, visible light camera, and infrared camera; Step 22: Spatial alignment of the data obtained from the lidar, visible light camera, and infrared camera.

4. The pedestrian target detection method based on radar video infrared decision-level fusion according to claim 3, characterized in that: In step two, the data obtained by the lidar, visible light camera and infrared camera are time-aligned. The specific process is as follows: Let any At any time, the Pandar40P lidar... Frequent release of point cloud data , The dimension is ,in This indicates the number of points in each frame of LiDAR point cloud data; 4 represents... It contains the dimensions of point cloud data, including location information (x, y, z) and reflection intensity; the LT-H8179 visible light camera uses... Frequent release of visible light image data , The dimension is ,in and Represents the length and width of a visible light image, 3 indicates It is a color image containing three RGB channels; the COIN417G2 infrared camera... The frequency of releasing infrared image data , The dimension is ,in and These represent the length and width of the infrared image, respectively, with 1 indicating... It is a grayscale image that contains only one channel of grayscale value; At the same time, in any At all times, the synchronization node continuously receives point cloud data from the Pandar40P LiDAR. Visible light image data released by the LT-H8179 visible light camera Infrared image data released by the COIN417G2 infrared camera During synchronization, first set up a receiver. , and The system uses three separate caches for different types of data, each stored in an independent storage space. The data in each storage space is updated in real-time based on data acquired by the LiDAR, visible light camera, and infrared camera. A cache only sends synchronization data out when all three storage areas have acquired the data. The synchronization period for each synchronization node to send synchronization data is specified. Select according to formula (1); (1)。 5. The pedestrian target detection method based on radar video infrared decision-level fusion according to claim 4, characterized in that: In step two, spatial alignment is performed on the data obtained from the lidar, visible light camera, and infrared camera; the specific process is as follows: Step 221: Joint calibration of LiDAR and visible light camera: Joint calibration includes the calibration of the intrinsic parameters of the visible light camera and the calibration of the extrinsic parameters between the visible light camera and the lidar; Step 222: Joint calibration between the infrared camera and the visible light camera.

6. The pedestrian target detection method based on radar video infrared decision-level fusion according to claim 5, characterized in that: The joint calibration of the lidar and visible light camera in step 221 includes the calibration of extrinsic parameters between the visible light camera and the lidar, as well as the calibration of intrinsic parameters of the visible light camera. The specific process is as follows: Step 2211: Calibrate the intrinsic parameters of the visible light camera; The specific steps are as follows: a) Acquire a set of calibration board image data for calibrating the camera. The calibration board images cover calibration board images at different angles, distances, and orientations. Obtain calibration board information for calibrating the camera, including the number of rows and columns of corner points on the calibration board and the actual dimensions between the corner points; b) Run the camera-calibration tool and obtain the camera's intrinsic parameters based on the visible light camera imaging model in the camera-calibration tool; The intrinsic parameters of a camera include its eigenvalue matrix, radial distortion coefficient, and tangential distortion coefficient. The radial distortion coefficients are taken from the first two in the array; Step 2212: Calibrate the extrinsic parameters between the visible light camera and the lidar; the specific process is as follows: Image pixel coordinate system To the physical coordinate system of the image The conversion process is represented by formula (2): (2) In the formula, , Represents the center pixel coordinates of the image. , Indicates the size of the camera's image sensor; From the physical coordinate system of the image To the visible light camera coordinate system The conversion process is represented by formula (3): (3) In the formula, Represents the camera's focal length; Visible light camera coordinate system With lidar coordinate system The following relationship exists: (4) In the formula, and These represent the rotational and translational relationships between the camera and the lidar, respectively. Due to the refraction of light by different parts of the lens and the offset between lens elements, points in the lens's physical coordinate system exhibit radial and tangential distortion. Therefore, a point in the physical coordinate system... It will be affected by distortion and change as , This can be expressed by formula (5): This can be expressed by formula (6): (5) (6) In the formula, Indicates the distance from the optical center. ; , , , , , Represents the radial distortion coefficient. , , , The value is 0; , Indicates the tangential distortion coefficient; Substituting equations (5) and (6) into equation (2) yields the pixel coordinates in the image pixel coordinate system. ; set up , Pixel coordinates in the image pixel coordinate system With lidar coordinate system The conversion process is represented by formula (7): (7) In the formula, This indicates the internal parameters of the visible light camera; This represents the rotation and translation relationship between the lidar and the visible light camera, i.e., the extrinsic parameter matrix. ; , Indicates intermediate variables; Use the cam-lidar-calibration tool in the ROS toolbox to perform extrinsic parameter matrix analysis between the visible light camera and the lidar. The calibration process is as follows: a) Simultaneously sample the calibration board using a lidar and a visible light camera, with the calibration board images covering different angles, distances, and orientations; b) Regarding the sampled calibration plate images: The `cv::find Chessboard Corners` method from the OpenCV open-source vision library was used to extract the corner points on the calibration board image, and the pixel coordinates of the calibration board center in the calibration board image were calculated. and normal vector ; Regarding the sampled calibration board point cloud data: The sampled calibration board point cloud data is separated from a complete frame of data acquired by the LiDAR using a preset spatial threshold; then, 3D RANSAC is used to fit the boundary of the calibration board, and the center three-dimensional coordinates of the calibration board are calculated. and normal vector , Indicates the sequence number of the collected data; c) Repeat steps a) and b) M times to obtain M sets. , , , ; Group M , , , The intrinsic parameters of the calibrated visible light camera are substituted into the transformation relationship between the lidar coordinate system and the camera coordinate system (4), and then optimized by a genetic algorithm to obtain the final result. and .

7. A pedestrian target detection method based on radar video infrared decision-level fusion according to claim 5, characterized in that: The joint calibration between the infrared camera and the visible light camera in step two is as follows: Step 2221: Preprocess the visible light and infrared images; the specific process is as follows: For infrared images: The infrared image is inverted, and the inverted infrared image is then denoised using a Gaussian filter with a kernel size of 3. For visible light images: Denoising of visible light images is achieved by median filtering with a kernel size of 3. Step 2: Use the Canny operator to perform edge detection on the preprocessed visible light image and infrared image to obtain the edges of the visible light image and infrared image; Step 2223: Use the ORB algorithm to extract and describe the feature points of the edges in the visible light and infrared images; The feature points at the edges of the visible light image and the infrared image are matched using a brute-force matching method to obtain the matched feature points. The RANSAC algorithm is used to filter the matched feature points to obtain matching pairs; The homography matrix between feature points is calculated by matching pairs; Step 2224: Repeat steps 2221 to 2223. Then, the average value of the homography matrix is ​​calculated and used as the conversion relationship between the visible light image and the infrared image; 。 8. The pedestrian target detection method based on radar video infrared decision-level fusion according to claim 7, characterized in that: In step three, based on the spatiotemporally aligned LiDAR, visible light camera, and infrared camera, target detection is performed on pedestrian targets in the LiDAR point cloud, pedestrian targets in the visible light image, and pedestrian targets in the infrared image. The specific process is as follows: Step 31: Obtain the trained Complex-YOLOv4 detection network based on deep learning; the specific process is as follows: The KITTI dataset was selected as the training set. The KITTI dataset's lidar point cloud data is used as the input to the deep learning-based Complex-YOLOv4 detection network, and the pedestrian targets in the KITTI dataset's lidar point cloud are used as the output of the deep learning-based Complex-YOLOv4 detection network. The deep learning-based Complex-YOLOv4 detection network is trained until it converges, and the trained deep learning-based Complex-YOLOv4 detection network is obtained. Step 3.2: Obtain the trained Yolov7 detection network for visible light images; the specific process is as follows: The visible light images in the COCO dataset are used as the input to the YOLOv7 detection network, and the pedestrian targets in the visible light images in the COCO dataset are used as the output of the YOLOv7 detection network. The YOLOv7 detection network is pre-trained until it converges to obtain the pre-trained YOLOv7 detection network. The visible light images in the KITTI dataset are used as the input of the pre-trained Yolov7 detection network, and the pedestrian targets in the visible light images in the KITTI dataset are used as the output of the pre-trained Yolov7 detection network. The pre-trained Yolov7 detection network is trained until it converges, and a trained Yolov7 detection network for visible light images is obtained. Step 3: Obtain the trained Yolov7 detection network for infrared images; the specific process is as follows: Infrared images from the SCUT dataset are used as input to the YOLOv7 detection network, and pedestrian targets in the infrared images from the SCUT dataset are used as output to the YOLOv7 detection network. The YOLOv7 detection network is pre-trained until it converges to obtain a pre-trained YOLOv7 detection network. The acquired infrared image is used as the input of the pre-trained YOLOv7 detection network, and the pedestrian target in the acquired infrared image is used as the output of the pre-trained YOLOv7 detection network. The pre-trained YOLOv7 detection network is trained until convergence, and a trained YOLOv7 detection network for infrared images is obtained. The following modifications were made during the training process: (1) Cancel color transformations and scale transformations; (2) The loss function of the Yolov7 detection network is: in This is the overall loss function value of YOLOv7; It is the loss from the target location coordinate regression; It is the loss of target confidence; It is the loss of confidence in the target category; , , These are the corresponding weighting coefficients; ; ; ; (3) Copy the single-channel infrared grayscale image three times and fuse it into a three-channel infrared image for training; Steps 3 and 4: Based on the trained deep learning-based Complex-YOLOv4 detection network, pedestrian targets in the LiDAR point cloud under test are detected. The expression is: (9) in, This represents the 3D bounding box of a pedestrian target in the lidar point cloud data, including the center coordinates of the bounding box. , , The length, width, and height of the bounding frame , , And the heading angle of the bounding box in the lidar coordinate system ; Indicates the confidence level of the pedestrian target. This represents the number of pedestrians detected in a given frame of point cloud data. express The point cloud data collected by LiDAR in real time, This represents a trained deep learning-based Complex-YOLOv4 detection network for point cloud data. Step 3.5: Based on the Yolov7 detection network trained for visible light images, detect pedestrian targets in the visible light image to be tested. The expression is: (10) in, This represents the 2D bounding box of a pedestrian target in visible light image data, including the coordinates of the top-left corner of the bounding box. , and the coordinates of the bottom right corner , ; Indicates the confidence level of the pedestrian target. This indicates the number of pedestrians detected in a given frame of image data. express Visible light image data acquired in real time, This represents a trained Yolov7 detection network for visible light images; Step 36: Based on the Yolov7 detection network trained for infrared images, detect pedestrian targets in the infrared image to be tested. The expression is: (11) in, This represents the 2D bounding box of a pedestrian target in infrared image data, including the coordinates of the top-left corner of the bounding box. , and the coordinates of the bottom right corner , ; Indicates the confidence level of the pedestrian target; This indicates the number of pedestrians detected in a given frame of image data. express Infrared image data collected in real time, This represents a Yolov7 detection network trained for infrared images.

9. A pedestrian target detection method based on radar video infrared decision-level fusion according to claim 8, characterized in that: In step four-three, the tensor The trained decision-level fusion model is input, and the output is a fusion of pedestrian targets detected by different sensors; the specific process is as follows: (15) in, For the fusion The confidence level of each candidate detection box. This represents the trained decision-level fusion model; (16) in, Indicates after fusion A set of candidate detection boxes, This represents the j-th candidate detection box in the lidar point cloud after decision-level fusion adjustment. This indicates the position of the j-th candidate detection box in the LiDAR point cloud after decision-level fusion adjustment. Let represent the confidence level of the j-th candidate detection box in the LiDAR point cloud after decision-level fusion adjustment; Will By using non-maximum suppression and then filtering with a preset target confidence threshold, the final pedestrian target is obtained. The trained decision-level fusion model The acquisition process is as follows: The decision-level fusion model is evaluated using labeled 2D images and object detection boxes in 3D point clouds. Train the model until it converges to obtain a well-trained decision-level fusion model. .

Images