Vehicle recognition method based on multi-sensor data fusion

Abstract

The application discloses a kind of vehicle identification methods based on multi-sensor data fusion, comprising the following steps: S1. synchronously collecting the video information of vehicle and radar point cloud information;S2. extract vehicle image feature information in video information, and 3D detection is carried out to radar point cloud information, and vehicle prediction frame information is obtained;S3. vehicle image feature information and vehicle prediction frame information are fused, and vehicle fusion result is obtained;S4. whether the maximum outer rectangle overlap of vehicle image contour projection and vehicle prediction frame contour in vehicle fusion result is greater than set value, if yes, vehicle fusion result is effective, and the outline of vehicle is determined;If no, vehicle fusion result is invalid.The application solves the problem that the recognition rate of traditional video monitoring scheme is not high and the monitoring range is small in low visibility environment, and improves the robustness of vehicle identification.

Description

Technical Field

[0001] This invention relates to the field of vehicle recognition, and more specifically to a vehicle recognition method based on multi-sensor data fusion. Background Technology

[0002] In recent years, highways have gradually achieved full coverage of traffic sensing and monitoring equipment based on roadside gantries. Although video-based vehicle detection algorithms are mature and low-cost, the 2D data collected by video has large errors in measuring distance, speed and location of distant objects. It is also more sensitive to severe weather conditions such as heavy fog, rainstorms and nighttime, meaning it cannot effectively identify the outline of vehicles.

[0003] While vehicle detection based solely on 3D LiDAR point clouds can cover multiple lanes and achieve accurate measurement of vehicle 3D contours and vehicle type classification, its data acquisition frequency is low, resulting in limited data throughput over short periods. Therefore, an effective vehicle recognition method is urgently needed to address these issues. Summary of the Invention

[0004] In view of this, the purpose of this invention is to overcome the deficiencies in the prior art and provide a vehicle recognition method based on multi-sensor data fusion, which can improve the robustness of vehicle recognition in harsh external environments such as dust, haze and lighting conditions, and provide richer spatial information of target objects for vehicle type recognition in the entire traffic flow.

[0005] The vehicle recognition method based on multi-sensor data fusion of the present invention includes the following steps:

[0006] S1. Synchronously collect vehicle video information and radar point cloud information;

[0007] S2. Extract vehicle image feature information from video information and perform 3D detection on radar point cloud information to obtain vehicle prediction box information;

[0008] S3. Perform feature fusion on vehicle image feature information and vehicle prediction box information to obtain vehicle fusion result;

[0009] S4. Determine whether the maximum overlap between the vehicle image contour projection and the vehicle prediction box contour in the vehicle fusion result is greater than the set value. If yes, the vehicle fusion result is valid and the vehicle's external contour is determined; otherwise, the vehicle fusion result is invalid.

[0010] Furthermore, the synchronization includes time synchronization and spatial synchronization;

[0011] The time synchronization specifically includes: simultaneously acquiring and receiving vehicle video information while acquiring vehicle radar point cloud information and receiving radar signals;

[0012] The spatial synchronization specifically includes: calibrating the coordinate positions of the radar and camera in the world coordinate system where the vehicle is located, and obtaining the relative positions of the radar and camera; wherein the radar is used to collect radar point cloud information, and the camera is used to collect video information.

[0013] Furthermore, vehicle image feature information is extracted from the video information, specifically including:

[0014] Constructing an improved YOLOv5 network: Before inputting the algorithm block, the input low-level feature map is divided into a first part and a second part; the first part is connected through a weighted direct connection mapping, and the second part is inferred through the algorithm block; the processing results of the first part and the calculation results of the second part are concatenated and merged through a cross-level hierarchical structure to obtain the network output.

[0015] The vehicle images from the video information are input into the improved YOLOv5 network, which outputs multi-scale target detection results for the vehicle images.

[0016] Furthermore, 3D detection is performed on the radar point cloud information to obtain vehicle prediction bounding box information, specifically including:

[0017] The foreground and background of the radar point cloud information are segmented, and then all foreground vehicle target point clouds are processed by bottom-up Bin operation to obtain 3D prediction boxes.

[0018] The 3D prediction bounding box is refined using standard coordinates to obtain refined prediction bounding box information, which is then used as the vehicle prediction bounding box information.

[0019] Furthermore, a foreground point cloud segmenter is used to segment the foreground and background of the radar point cloud information, specifically including:

[0020] Interpolation is performed based on the inverse distance weighted average of k-nearest neighbors. The interpolated features are then stacked through cross-layer connections. The stacked features are then input into Unit PointNet for further extraction.

[0021] Repeat the above steps, and finally obtain the segmentation prediction result through n*n convolution operation.

[0022] Furthermore, feature fusion is performed on vehicle image feature information and vehicle prediction box information. Specifically, this includes calculating and projecting the vehicle outline and position in the vehicle prediction box information onto a point corresponding to the vehicle outline result in the vehicle image feature information, based on the calibrated parameters.

[0023] Furthermore, step S4 also includes: determining the detection accuracy of the hazardous chemical vehicle outline.

[0024] The outlines of hazardous chemical vehicles are normalized, and after being scaled down proportionally, they are compared with standard vehicle outlines to obtain standardized hazardous chemical vehicle outlines.

[0025] Calculate the degree of overlap between the standardized outline of a hazardous chemical vehicle and the actual vehicle's appearance outline, and determine the detection accuracy of the hazardous chemical vehicle outline based on the degree of overlap.

[0026] The beneficial effects of this invention are as follows: This invention discloses a vehicle recognition method based on multi-sensor data fusion. Based on high-definition images, radar, and other multi-source data fusion technologies and deep neural network algorithms, it adaptively aligns, synchronizes, and associates heterogeneous multi-source data, achieving intelligent pipeline normalization of the data stream. Furthermore, it trains a special transport vehicle recognition enhancement model, improving the robustness of recognition and solving the problems of low recognition rate and small monitoring range in traditional video monitoring schemes under low visibility conditions. This invention can improve the robustness of vehicle recognition in harsh external environments such as dust, fog, and lighting conditions, and can provide richer spatial information of target objects for vehicle type recognition throughout the traffic flow. Attached Figure Description

[0027] The present invention will be further described below with reference to the accompanying drawings and embodiments:

[0028] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0029] Figure 2 This is a schematic diagram of data fusion and registration according to the present invention;

[0030] Figure 3 This is a diagram of the cross-stage network structure of the present invention;

[0031] Figure 4 This is a network structure diagram of the image detection algorithm of the present invention;

[0032] Figure 5 This is a schematic diagram illustrating the principle of the foreground cloud segmenter of the present invention. Detailed Implementation

[0033] The present invention will be further described below with reference to the accompanying drawings, as shown in the figures:

[0034] The vehicle recognition method based on multi-sensor data fusion of the present invention includes the following steps:

[0035] S1. Synchronously collect vehicle video information and radar point cloud information;

[0036] S2. Extract vehicle image feature information from video information and perform 3D detection on radar point cloud information to obtain vehicle prediction box information;

[0037] S3. Perform feature fusion on vehicle image feature information and vehicle prediction box information to obtain vehicle fusion result;

[0038] S4. Determine whether the maximum overlap between the vehicle image contour projection and the vehicle predicted bounding box contour in the vehicle fusion result is greater than a set value. If yes, the vehicle fusion result is valid, and the vehicle's outline is determined; otherwise, the vehicle fusion result is invalid. The set value can be set according to actual working conditions. If the vehicle fusion result is valid, it is only necessary to compare the vehicle fusion result with the outline data information in the existing vehicle outline database to determine the target vehicle's outline. If the vehicle fusion result is invalid, the vehicle outline cannot be identified, and steps S1-S4 can be repeated.

[0039] In this embodiment, before synchronously acquiring video information and radar point cloud information of the vehicle, the acquisition equipment needs to be calibrated. The acquisition equipment includes radar and camera. The radar is a lidar and the camera is an RGB-D camera. The calibration of the acquisition equipment specifically includes: quickly calibrating the camera intrinsic parameter matrix K using the Zhang Zhengyou calibration method, using the intrinsic parameters as known quantities, and using ArUco flags to calibrate the installation parameters of the RGB-D camera and LiDAR (Light Detection and Ranging) system.

[0040] The synchronization mentioned in the synchronous acquisition of vehicle video information and radar point cloud information includes time synchronization and spatial synchronization;

[0041] Because the LiDAR and camera have different sampling frequencies, they take different amounts of time to extract effective targets. The data sent to the fusion system center may be information from different times, which can lead to significant deviations in the later fusion results. Therefore, it is necessary to perform time synchronization processing.

[0042] The time synchronization specifically includes: acquiring and receiving vehicle video information while collecting radar point cloud information and receiving radar signals; that is, using radar acquisition of effective target sequences and camera acquisition of effective target sequences as two sub-threads, and using the subsequent fusion process as the main thread, triggering the camera data frame receiving thread while receiving radar signals, thereby ensuring the time synchronization of the sensors.

[0043] To achieve better synchronous data acquisition, spatial synchronization processing is also required in addition to ensuring time synchronization.

[0044] The spatial synchronization specifically includes: calibrating the coordinate positions of the radar and camera in the vehicle's world coordinate system to obtain their relative positions; wherein the radar is used to collect radar point cloud information, and the camera is used to collect video information. After spatial registration is achieved, the fusion relationship between the radar point cloud information and the video information can be further derived based on the relative positional relationship between the radar and the camera, such as... Figure 2 As shown.

[0045] In this embodiment, by improving the existing YOLOv5 network, the improved YOLOv5 network can output multi-scale target detection results more accurately and reduce the false detection rate.

[0046] Extracting vehicle image feature information from video information, specifically including:

[0047] An improved YOLOv5 network is constructed by modifying the backbone of the existing YOLOv5 network: before the input algorithm block, the input low-level feature map is divided into a first part and a second part; the first part is connected through a weighted direct connection mapping, and the second part is used for inference computation through the algorithm block; the processing results of the first part and the computation results of the second part are concatenated through a cross-level hierarchical structure to obtain the network output; that is, the first-order detection network using the above idea consists of some algorithm blocks and some connection layers. Figure 3 As shown, the feature map obtained through the basic network is divided into two parts, which can be represented by equation (1):

[0048] x0=[x'0,x”0] (1)

[0049] Here, x”0 is obtained through reasoning in the second part of the algorithm block. T x'0 is directly related to the output result x of the second part of the algorithm block. T The network output x is obtained by performing connection fusion. U Its network computational reasoning can be represented by equations (2)(3)(4):

[0050]

[0051]

[0052]

[0053] Among them, X K This represents the computational inference within the detection algorithm block, and W represents the weight values ​​of each network layer.

[0054] Through the above steps, an improved YOLOv5 network is obtained, such as... Figure 4As shown, vehicle images from video information are input into an improved YOLOv5 network, which outputs multi-scale target detection results for the vehicle images.

[0055] In this embodiment, the Point-RCNN algorithm is used to perform 3D detection on radar point cloud information to obtain vehicle prediction bounding boxes. The Point-RCNN algorithm effectively controls network complexity while maintaining a relatively accurate recognition rate, meeting the established requirements and operating robustly.

[0056] The Point-RCNN algorithm employs a 3D detection network model that includes network a, which generates 3D prediction boxes from the original point cloud in a bottom-up manner, and network b, which refines the 3D prediction results using standard coordinates.

[0057] The radar point cloud information is input into the above three-dimensional detection network model. In network a, the foreground and background are first segmented using the foreground point cloud segmenter. Then, all foreground vehicle target point clouds are processed by bottom-up Bin operation to obtain 3D prediction boxes. The 3D prediction boxes can be used for later refinement of 3D prediction boxes and fusion.

[0058] The 3D prediction bounding box is input into network b for standard coordinate refinement to obtain refined prediction bounding box information, which is then used as the vehicle prediction bounding box information.

[0059] In this embodiment, a foreground point cloud segmenter is used to segment the foreground and background of the radar point cloud information. The segmentation principle of the foreground point cloud segmenter is as follows: Figure 5 As shown, the foreground point cloud segmenter consists of a sampling layer, a grouping layer, and a single PointNet to pre-define the search region, thereby enabling the region segmentation task of the point cloud; the specific segmentation process is as follows:

[0060] Interpolation is performed based on the inverse distance weighted average of k-nearest neighbors. The interpolated features are then stacked through cross-layer connections. The stacked features are then input into Unit PointNet for further extraction.

[0061] Repeat the above steps, and finally obtain the segmentation prediction result through a 1*1 convolution operation.

[0062] In this embodiment, feature fusion is performed on vehicle image feature information and vehicle prediction box information. Specifically, this includes: calculating and projecting the vehicle outline and position in the vehicle prediction box information onto a point corresponding to the vehicle outline result in the vehicle image feature information, based on calibrated parameters. The calibrated parameters are determined according to the calibration settings implemented by the ArUco flags mentioned above. Through the above operations, the vehicle outline and position detected by the radar can correspond one-to-one with the vehicle outline detected by the camera, thereby recognizing the vehicle's external shape outline.

[0063] In this embodiment, in step S4, since the pixel size occupied by the outline of the hazardous chemical vehicle differs greatly between the far and near scenes, the vehicle outline recognition detection accuracy of the hazardous chemical vehicle will decrease. Therefore, the detection accuracy of the hazardous chemical vehicle outline can be determined by the following method:

[0064] The outlines of hazardous chemical vehicles are normalized, and after being scaled down proportionally, they are compared with standard vehicle outlines to obtain standardized hazardous chemical vehicle outlines.

[0065] The degree of overlap between the standardized outline of a hazardous chemical transport vehicle and the actual vehicle's exterior outline is calculated, and the detection accuracy of the hazardous chemical transport vehicle outline is determined based on the degree of overlap. By determining the detection accuracy of the hazardous chemical transport vehicle outline, the accuracy of vehicle shape outline recognition can be evaluated based on the determination results, thereby improving the reliability of vehicle recognition.

[0066] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A vehicle recognition method based on multi-sensor data fusion, characterized in that: Includes the following steps: S1. Synchronously collect vehicle video information and radar point cloud information; S2. Extract vehicle image feature information from video information and perform 3D detection on radar point cloud information to obtain vehicle prediction box information; Extracting vehicle image feature information from video information, specifically including: Constructing an improved YOLOv5 network: Before inputting the algorithm block, the input low-level feature map is divided into a first part and a second part; the first part is connected through a weighted direct connection mapping, and the second part is inferred through the algorithm block; the processing results of the first part and the calculation results of the second part are concatenated and merged through a cross-level hierarchical structure to obtain the network output. The vehicle images from the video information are input into the improved YOLOv5 network, and the multi-scale target detection results of the vehicle images are output. 3D detection is performed on radar point cloud information to obtain vehicle prediction bounding box information, specifically including: The foreground and background of the radar point cloud information are segmented, and then all foreground vehicle target point clouds are processed by bottom-up Bin operation to obtain 3D prediction boxes. The 3D prediction bounding box is refined using standard coordinates to obtain refined prediction bounding box information, which is then used as the vehicle prediction bounding box information. Specifically, the foreground and background of radar point cloud information are segmented using a foreground point cloud segmenter, including: Interpolation is performed based on the inverse distance weighted average of k-nearest neighbors. The interpolated features are then stacked through cross-layer connections. The stacked features are then input into Unit PointNet for further extraction. Repeat the above steps, and finally obtain the segmentation prediction result through n*n convolution operation; S3. Perform feature fusion on the vehicle image feature information and the vehicle prediction box information to obtain the vehicle fusion result; S4. Determine whether the maximum overlap between the vehicle image contour projection and the vehicle prediction box contour in the vehicle fusion result is greater than the set value. If yes, the vehicle fusion result is valid and the vehicle's external contour is determined; otherwise, the vehicle fusion result is invalid.

2. The vehicle recognition method based on multi-sensor data fusion according to claim 1, characterized in that: The synchronization includes time synchronization and spatial synchronization; The time synchronization specifically includes: simultaneously acquiring and receiving vehicle video information while acquiring vehicle radar point cloud information and receiving radar signals; The spatial synchronization specifically includes: calibrating the coordinate positions of the radar and camera in the world coordinate system where the vehicle is located, and obtaining the relative positions of the radar and camera; wherein the radar is used to collect radar point cloud information, and the camera is used to collect video information.

3. The vehicle recognition method based on multi-sensor data fusion according to claim 1, characterized in that: Feature fusion is performed on vehicle image feature information and vehicle prediction bounding box information. Specifically, the vehicle outline and position in the vehicle prediction bounding box information are projected onto a point corresponding to the vehicle outline result in the vehicle image feature information according to the calibrated parameters.

4. The vehicle recognition method based on multi-sensor data fusion according to claim 1, characterized in that: S4 further includes: the detection accuracy for determining the outline of hazardous chemical vehicles. The outlines of hazardous chemical vehicles are normalized, and after being scaled down proportionally, they are compared with standard vehicle outlines to obtain standardized hazardous chemical vehicle outlines. Calculate the degree of overlap between the standardized outline of a hazardous chemical vehicle and the actual vehicle's appearance outline, and determine the detection accuracy of the hazardous chemical vehicle outline based on the degree of overlap.

Images