A Pedestrian Detection Algorithm for Vision Blind Spots Based on Vehicle-Road Collaboration

Abstract

The present invention discloses a method for detecting pedestrians in a vision blind area based on vehicle-road collaboration. Cameras and lidar are provided on both the roadside and the vehicle side. The vehicle-side devices include a vehicle-mounted lidar and a vehicle-mounted camera that are relatively fixed, and the roadside devices include a roadside lidar and a roadside camera that are relatively fixed. It includes coordinate system one, a pedestrian detection method, and vehicle-road collaboration fusion. The present invention not only combines lidar and cameras for pedestrian detection, so that the point cloud data supplements the depth information and blind areas lacking in the image data, while the image data supplements the blind areas near the point cloud and the defect of sparse data in the distance, forming a complementarity in perception, but also the present invention sets lidar and cameras on both the roadside and the vehicle side and registers and converts the data obtained by the devices on both sides, so as to achieve over-the-horizon perception of vehicle-road collaboration and supplement the vision blind area perception data of the vehicle.

Description

Technical Field

[0001] The present invention relates to the technical field of pedestrian detection in the vehicle blind spot, and more specifically, to a pedestrian detection algorithm for the blind spot based on vehicle-road cooperation. Background Art

[0002] Currently, vehicles in the fields of autonomous driving and assisted driving have problems such as limited perception range, the existence of blind spots, and limited recognition of long-distance objects. By using vehicle-road cooperation technology to fuse multi-source heterogeneous raw sensor data at the data layer and decision layer, the vehicle's pre-perception and detection capabilities for pedestrians in the blind spot can be enhanced, which can greatly improve traffic safety.

[0003] There is a method for solving the pedestrian pose for a vehicle, including: multiple network-connected vehicles use their surround-view cameras to perform real-time perception of pedestrians within the field of view, and obtain the three-dimensional point cloud of pedestrians in the single-vehicle field of view; according to the driving speed return value of the vehicle-mounted system, filter out invalid vehicles to form a local vehicle network that can share valid data; based on the local vehicle network, perform synthesis and stitching of the three-dimensional point cloud of pedestrians in the multi-vehicle field of view to obtain a global three-dimensional point cloud; use the positional relationship between points in the point cloud to obtain the global three-dimensional pedestrian pose; share the global three-dimensional pedestrian pose to all vehicles within the local vehicle network; based on the single-vehicle safety range, analyze the global three-dimensional pedestrian pose to obtain the pedestrian warning information within the single-vehicle safety range. By using the present invention, in scenarios such as parking lots and intersections, the vehicle's perception range can be extended, and at the same time, vehicles without perception capabilities can perceive pedestrian information within the safety range.

[0004] However, the above solution only obtains images through a surround-view monocular camera, and then estimates the three-dimensional point cloud of the single-vehicle field of view based on key point estimation. It is difficult to ensure the accuracy of the estimated three-dimensional point cloud. And in terms of unifying the coordinate system, only the RANSAC algorithm is used to solve the transformation matrix and perform stitching, which requires high computing power and long computing time, and it is difficult to achieve real-time point cloud registration and transformation. At the same time, the above solution realizes cooperative perception only through vehicles, without using roadside nodes. And the object of cooperation is the surrounding stationary vehicles, and the applicable scenarios of this method are limited. For example, if there are no stationary vehicles around, or the stationary vehicles are not started, it is impossible to perform perception and communication. And the detection of pedestrians is only based on images to obtain pedestrian bounding boxes, and the pedestrian detection accuracy is low. Summary of the Invention

[0005] The object of the present invention is to overcome the deficiencies of the prior art pedestrian pose calculation method for vehicles that only obtains images through cameras and realizes collaborative perception only between vehicles, resulting in limited usage scenarios. At the same time, the detection of pedestrians is only based on the image to obtain the pedestrian bounding box, with low detection accuracy. The present invention provides a blind spot pedestrian detection algorithm based on vehicle-road collaboration. The present invention not only combines lidar and cameras for pedestrian detection, enabling the point cloud data to supplement the depth information and blind spots lacking in the image data, while the image data supplements the blind spots near the point cloud and the defect of sparse data in the distance, forming a complementarity in perception. Moreover, the present invention also sets lidar and cameras on both the roadside and the vehicle side and registers and converts the data obtained by the two-side devices, thereby realizing the ultra-long-range perception of vehicle-road collaboration and supplementing the blind spot perception data of the vehicle.

[0006] The object of the present invention can be achieved by the following technical solutions:

[0007] A blind spot pedestrian detection method based on vehicle-road collaboration, with cameras and lidar installed on both the roadside and the vehicle side. The vehicle-side devices include a vehicle-mounted lidar and a vehicle-mounted camera that are relatively fixed, and the roadside devices include a roadside lidar and a roadside camera that are relatively fixed. It includes coordinate system one, pedestrian detection method, and vehicle-road collaboration fusion. Among them, coordinate system one includes the following steps:

[0008] S1.1: Use a calibration tool to jointly calibrate the camera and lidar on the same side to obtain the internal reference matrix K of the camera and the external reference matrix between the camera and the lidar

[0009] S1.2: The lidar collects point cloud P, where the point cloud collected by the vehicle-mounted lidar is P1, and the point cloud collected by the roadside lidar is P2. Preprocess the point clouds P1 and P2 respectively. The preprocessing includes point cloud denoising, point cloud downsampling, point cloud ground segmentation, and region of interest extraction, so as to obtain the point clouds P1 roi and P2 roi .

[0010] S1.3: After the vehicle enters the communication range of the roadside device, use the vehicle GPS and IMU to determine the vehicle pose information, communicate to obtain the roadside device pose information, and calculate the initial value T of the transformation matrix GPS , taking T GPS as the initial value, perform initial registration on the preprocessed point clouds P1 roi and P2 roi to obtain the transformation matrix T SAC , taking T SAC as the initial value, perform precise registration to obtain the first coordinate system transformation matrix T ini t;

[0011] S1.4: Estimate the movement of the vehicle relative to the first registration position based on the first coordinate system transformation matrix T init ; then continuously output the transformation matrix according to the coordinate transformation relationship

[0012] Pedestrian detection includes the following steps:

[0013] S2.1: The in-vehicle camera and the roadside camera respectively collect images, perform pedestrian detection on the images obtained by the cameras to obtain 2D plane detection frames, and project the point cloud P into the image coordinate system according to the internal reference matrix K and the external reference matrix and count the point cloud set P that falls within the 2D plane detection frame in the image coordinate system detect ;

[0014] S2.2: Cluster the points in the point cloud set P aetect select the category with the largest number of points as the pedestrian point cloud P person and generate a sequence of point cloud bounding boxes B1; person

[0015] S2.3: Perform 3D pedestrian detection on the point cloud P and generate a sequence of pedestrian point cloud bounding boxes B2;

[0016] S2.4: Match the sequences of pedestrian point cloud bounding boxes B1 and B2 formed by the above two methods and obtain the final sequence of pedestrian bounding boxes B using the weighted average method fusion ;

[0017] S2.5: Process the point cloud P with the point clouds P1 and P2 as inputs respectively through steps S2.1 to S2.4 to obtain the sequence of pedestrian bounding boxes B1 fusion formed by the in-vehicle device processing and the sequence of pedestrian bounding boxes B2 fusion formed by the roadside device processing;

[0018] Vehicle-road collaborative fusion includes the following steps:

[0019] S3: Transform the pedestrian point cloud bounding box B2fusion formed by the roadside device processing into the vehicle coordinate system according to the transformation matrix then the sequence of pedestrian bounding boxes B1 fusion and the pedestrian point cloud bounding box B2 transformed by the transformation matrix fusion are the pedestrian detection results from the vehicle's perspective.

[0020] In the present invention, detection devices are provided on both the roadside and the in-vehicle side, and the detection device on each side includes a camera and a lidar.

[0021] In the camera and lidar on the same side, by using a calibration tool to calibrate the image collected by the camera and the point cloud P collected by the lidar on the same side, the conversion relationship between the camera and the lidar on the same side can be obtained, that is, the internal reference matrix K and the external reference matrix Thus, the point cloud set P that falls within the plane detection frame of the image in the point cloud P is extracted aetect , thereby obtaining the pedestrian point cloud bounding box sequence B1 that combines the camera and the lidar. At the same time, only using the P collected by the lidar to form the pedestrian point cloud bounding box sequence B2, and matching and weighted averaging the pedestrian point cloud bounding box sequences B1 and B2 to obtain the final pedestrian bounding box sequence B fusion . The above steps are performed on the data collected by the detection devices on both sides, that is, steps S1.1 and steps S2.1 - S2.5 are performed, and the vehicle-side device obtains the pedestrian bounding box sequence B1 fusion , and the roadside device obtains the pedestrian bounding box sequence B2 fusion , then, the pedestrian bounding box sequence B2 on the roadside is required fusion to be converted into the vehicle perspective. The disadvantage of the camera is that it cannot obtain accurate depth information, while the disadvantage of the lidar is that the perception data for objects at a relatively long distance is too sparse and difficult to detect. A single sensor has limitations in the perception and detection of complex environments. Therefore, through the present invention, pedestrian detection is performed by combining image and point cloud data. The point cloud data supplements the depth information and blind spots lacking in the image data, while the image data supplements the blind spots near the point cloud and the defect of sparse data in the distance, forming a complement in perception

[0022] When converting the pedestrian bounding box sequence B2 on the roadside fusion into the vehicle perspective, the transformation matrix is obtained by registering the point clouds P1 and P2 collected by the lidars on the vehicle side and the roadside That is, by performing steps S1.2 - S1.4, the pedestrian bounding box sequence B2 can be fusion converted into the vehicle perspective. In this way, the pedestrian bounding box sequence B2 obtained by the roadside device fusion can supplement the pedestrian bounding box sequence B1 obtained by the vehicle-side device fusion to achieve ultra-long-range perception of vehicle-road collaboration and supplement the perception data of the vehicle's vision blind spot

[0023] Among them, when registering the point clouds P1 and P2 collected by the lidars on both sides, the essence of lidar point cloud registration is to perform a spatial transformation on the point cloud according to the overlapping area. Point cloud registration has a good effect on point clouds with rigid transformation. However, due to the different number of laser lines, angular distribution, and sampling resolution of the roadside lidar and the vehicle-mounted lidar, the densities of the two parts of the point clouds obtained are inconsistent, and these two parts of the point clouds are heterogeneous. In order to improve the registration accuracy, the present invention preprocesses the point clouds P1 and P2 to reduce noise and remove unnecessary areas such as the area above the vehicle roof, and then performs registration. Moreover, during the registration process, initial registration and precise registration are performed twice, greatly improving the registration accuracy. And, in the context of fast vehicle movement and high radar frame rate, the existing point cloud registration methods need to perform frame-by-frame registration, which has a large amount of calculation and cannot meet the real-time requirement in the scenario of vehicle-road cooperation. In order to achieve real-time registration and reduce the amount of calculation, estimate the movement of the vehicle relative to the first registration position According to the movement Output the transformation matrix in real time In this way, when the vehicle enters a set of roadside devices, that is, when it enters the communication azimuth of the roadside lidar and the roadside camera located at the same place, only one preprocessing and registration process of step S1.2 and step S1.3 needs to be performed to obtain the first coordinate system transformation matrix T init , and then through the first coordinate system transformation matrix T init and the movement of the vehicle relative to the first registration position the transformation matrix can be output in real time until it enters the communication range of the next set of roadside devices.

[0024] The calibration tool can use the Calibration Tool Kit in the Autoware framework, Sensor Calibration in the Apollo 2.0 framework, but_calibration_camera_velodyne, etc.

[0025] Furthermore, when performing joint calibration in step S1.1, when the camera and the lidar on the same side simultaneously observe point M, the projection coordinates of point M in the image coordinate system are U = (u, v, 1) T , and the coordinates in the lidar coordinate system are M l (x l , y l , z l );

[0026] Suppose the conversion relationship between the lidar and the camera is Then

[0027]

[0028] Among them is the intrinsic parameter matrix of the camera. The conversion matrix can be solved by collecting at least four sets of data from the lidar and camera on the same side.

[0029] In this scheme, the scanning point (x l ,y l , z l ) in the image pixel coordinate system can be obtained by Calculate and then transform to get Easy to know, There are 12 parameters that need to be solved, so more than 4 sets of lidar corresponding points are needed to get the result. Therefore, multiple sets of data can be used in calibration, and the least squares method can be used to solve the external reference matrix.

[0030] Furthermore, in the step S1.2, when preprocessing the point cloud P, the following steps are included:

[0031] S1.2.1: Point cloud denoising: Assume that the input point cloud is P input =(p1, p2, ..., p n ), the filter radius is set to R filter , the threshold of the number of neighboring points is num th , use the radius filter algorithm to denoise and get the denoised point cloud set P filtered ;

[0032] When using the radius filter algorithm for denoising, for each point cloud set P input Point p in i , if p i is the center of the sphere, R filter The number of neighboring points in the sphere with radius num ngb <num th , then p i Remove, otherwise p i Keep, and finally get a denoised point cloud set P filtered .

[0033] S1.2.2: Point cloud downsampling: For the denoised point cloud set P filtered Perform voxel downsampling of the same specification respectively, divide the three-dimensional space into voxels with side lengths of (δx, δy, δz), and let the point in the voxel be P voxel =(p1, p2, ..., p n ), then the centroid of all laser reflection points in the voxel is p c =(x c ,y c , zc ), where the coordinate values of the centroid are

[0034]

[0035]

[0036]

[0037] Use this centroid p c to replace the points within the voxel, and traverse each voxel in sequence, then the downsampled point cloud P ds is formed;

[0038] In the fusion of point cloud data from multi-source lidars, different lidars often have different sampling points for the same object due to the distribution of laser beams, field of view angles, and resolutions, resulting in different characteristics of the point cloud data obtained. This situation poses a great challenge to point cloud registration. Considering the above reasons, for the two point cloud sets used for registration, voxel downsampling of the same specification can be performed first to improve the registration accuracy.

[0039] S1.2.3: Point cloud ground segmentation: For the downsampled point cloud P ds remove the point cloud of the ground part in it to obtain the point cloud set P ng without the ground part;

[0040] The reflected point cloud formed by the mechanical lidar emitting laser to the ground is generally a concentric circular arc centered on the lidar. When the positions of two lidars are different, even for the same piece of ground, the point characteristics on these circular arcs are completely different, which may interfere with the registration of the point cloud. Therefore, it is necessary to segment the point cloud of the ground. For the segmentation of the point cloud of the ground part, an algorithm based on Random Sample Consensus (RANSAC) can be used to obtain the point cloud set P ng

[0041] S1.2.4: According to the field of view angle of the roadside lidar, the distances to surrounding buildings and obstacles, and the area that needs to be concerned about during vehicle driving, remove the point cloud in the point cloud set P ng with a height exceeding a certain range, and only leave the point cloud data P roi of the region of interest.

[0042] Generally speaking, due to the different vertical field of view angles of different lidars, different positions, and different distances from objects, the acquired point cloud data is not consistent in height. It is very likely that some point clouds in a point cloud set are completely higher than those in another point cloud set. That is to say, there are no corresponding points in another point cloud set for these point clouds with too high height, which is not helpful for point cloud registration. If the vertical field of view angle of the vehicle-mounted lidar is larger and some data at higher heights is collected, such as the top of a tree, the wall of a higher floor, etc., there are no corresponding points for this part of the data in the point cloud of the roadside lidar. It not only fails to promote point cloud registration but may also lead to an increase in registration time or an increase in registration error. On the other hand, the area of concern during vehicle driving is also limited. Usually, only the scene within a certain height range above the ground is concerned. Therefore, the point cloud data above a certain height can also be discarded. Reducing redundant data is also beneficial for data transmission and processing and reduces the interference to the valid data in the area of interest.

[0043] S1.2.5: Respectively use the point cloud data P1 and P2 as the input point cloud sets P input Perform the processing of the above steps S1.2.1 to S1.2.4 to obtain the denoised point cloud sets P1 roi and P2 roi .

[0044] Furthermore, in the step S1.2.3, the random sample consensus algorithm is used to remove the point cloud of the ground part in the downsampled point cloud P ds

[0045] Furthermore, in the step S1.3, on the vehicle side, the coordinate systems of the vehicle-mounted lidar, GPS, and IMU with relatively fixed pre-calibrated positions and the vehicle are calibrated in advance. The transformation matrix between the vehicle-mounted lidar coordinate system and the Earth-centered Earth-fixed coordinate system is obtained through the vehicle position and orientation information given by GPS and IMU, so as to determine the position of the vehicle-mounted lidar in the Earth-centered Earth-fixed coordinate system;

[0046] On the roadside, the coordinate system of the roadside lidar fixed on the roadside is pre-calibrated with the Earth-centered Earth-fixed coordinate system, so as to determine the coordinate position of the roadside lidar in the Earth-centered Earth-fixed coordinate system and broadcast the coordinate position;

[0047] According to the positions of the lidars on both sides in the Earth-centered Earth-fixed coordinate system, calculate the initial value T of the transformation matrix GPS ; Using T GPS as the initial value, through the sampling consensus initial registration algorithm SAC-IA, the pre-processed point cloud P1 roi and P2 roi are initially registered to obtain the transformation matrix T SAC ; Using TSAc is the initial value. The point clouds P1 and P2 are accurately registered through the Iterative Closest Point (ICP) method to obtain the final first coordinate system transformation matrix T roi and P2 roi and obtain the final first coordinate system transformation matrix T init .

[0048] Since the vehicle-mounted lidar, GPS, and IMU are fixedly installed on the vehicle, the vehicle and the vehicle-mounted lidar can be calibrated in advance, and the coordinate systems of GPS, IMU, and the vehicle-mounted lidar are all transformed into the Earth-Centered Earth-Fixed (ECEF) coordinate system. The direction information of the vehicle is obtained using the information provided by GPS and IMU to determine the transformation matrix between the lidar coordinate system and the ECEF coordinate system. Since the present invention also uses IMU for initial pose estimation, compared with relying only on GPS position information, the fusion accuracy and stability of the present invention are higher. At the same time, the initial accuracy obtained through the information provided by GPS and IMU is higher than that obtained only through GPS, thus obtaining a transformation matrix T with high initial accuracy SAc , and after accurately registering the point clouds P1 roi and P2 roi through the Iterative Closest Point (ICP) method, the obtained first coordinate system transformation matrix T init has higher fusion accuracy.

[0049] The conversion of the WGS-84 coordinate system of GPS to the Earth-Centered Earth-Fixed (ECEF) coordinate system is as follows:

[0050]

[0051] where alt is the elevation, lat is the latitude, lon is the longitude, the semi-major axis a of the reference ellipsoid of the WGS-84 coordinate system is 6378137.0 m, and the polar flattening of the reference ellipsoid

[0052] Furthermore, in S1.4, the transformation matrix of the vehicle obtained by the output of the LeGO-LOAM algorithm at the moment of obtaining the i-th frame of point cloud relative to the first registration position is then the transformation matrix at the current moment is

[0053] Furthermore, in step S2.1, according to the calibrated internal parameters K and external parameters of the camera and lidar using the formula

[0054]

[0055] project the point cloud P into the image coordinate system.

[0056] Furthermore, in step S2.2, for the point cloud P detectWhen clustering the points within, according to the performance of the lidar, two parameters (ε, MinPts) are selected. Among them, ε describes the neighborhood distance threshold of a certain data point, and MinPts describes the minimum number of data points in the neighborhood with a radius of ε for a data point. If there are at least MinPts points within the ε neighborhood corresponding to a point p, then point p is a core object. The DB-SCAN algorithm is used to cluster the point cloud P detect Cluster the points within, and arbitrarily select a core object p without a category i As a seed, find all the points that this core object p i Can density-reach a clustering cluster C of MinPts k , until all points are traversed completely, and select the point cloud cluster C with the largest number of points k As the pedestrian point cloud P person , According to P person Form an axis-aligned bounding box (AABB) sequence

[0057] At the beginning of the DB-SCAN algorithm, we arbitrarily select a core object p1 without a category as a seed, and then find all the sample sets C1 that this core object can density-reach, that is, a clustering cluster. Then continue to select another core object without a category to find the density-reachable sample set C2,..., C n , until all points are traversed completely. Select the point cloud cluster C with the largest number of points k As the pedestrian point cloud P person , According to P person Form an axis-aligned bounding box (AABB) sequence

[0058] When clustering the points within the point cloud P detect , other clustering methods such as k-means, OPTICS, Agglomerative, and Divisive can also be used.

[0059] Furthermore, in the step S2.3, the Point-Pillar detection method is used for 3D pedestrian detection. First, the point cloud P is encoded using the Pillar encoding method to obtain a pseudo 2D map, then the encoded pseudo 2D map is processed using 2D convolution, and finally the detection head of SSD is used to detect the target to obtain a bounding box sequence

[0060] Point Pillar detects the point cloud and will first complete all the point cloud sets P detectConversion to a pseudo-image, cutting is performed on the x and y axes of the three-dimensional space to generate individual Pillars. Each single point cloud is represented by a 9-dimensional augmented vector, and two methods, truncation and padding, are used to convert the sparse point cloud into dense data. Finally, the point cloud P detect will be aggregated onto a dense Tensor of size (D * N * P). After applying a simplified version of PointNet and a linear layer to each point cloud, learning the features of the point cloud to generate a Tensor of size (C * P * N), the point cloud is moved back to its original position according to the index to generate a pseudo-image. After the progressive downsampling process of the backbone, the corresponding features are upsampled to a unified size and concatenated; finally, SSD detection is used to form a sequence

[0061] In addition to Point-Pillar, other point cloud pedestrian detection models such as SECOND, VoxelNet, MVF, LaserNet, PointRCNN, PIXOR, etc. are also available.

[0062] Furthermore, in step S2.4, taking the center position of the bounding box as the reference, the distances between each pair of the bounding box sequence B1 and the bounding box sequence B2 can be obtained. Using the minimum distance sum as the optimization target, the Hungarian algorithm is used for pairwise matching to obtain the corresponding relationship between the bounding box sequence B1 and the bounding box sequence B2

[0063] Among them, and are the i-th and j-th bounding boxes in the bounding box sequence B1 and the bounding box sequence B2 respectively.

[0064] Compared with the prior art, the beneficial effects of the present invention are:

[0065] (1) Combining lidar and camera for pedestrian detection, so that the point cloud data supplements the depth information and blind spots lacking in the image data, while the image data supplements the blind spots near the point cloud and the defect of sparse data in the distance, forming a complementarity in perception.

[0066] (2) Setting lidar and camera on both the roadside and the vehicle side and registering and converting the data obtained by the two sides of the device, so as to realize the ultra-long-range perception of vehicle-road cooperation and supplement the perception data of the vehicle's visual blind area.

[0067] (3) Preprocessing the point cloud P to remove unnecessary parts, greatly improving the registration accuracy on both sides of the vehicle and the road.

[0068] (4) In the case of the vehicle running dynamically, the data coordinate systems on the vehicle side and the roadside can be unified. Brief Description of the Drawings

[0069] Figure 1 This is a flowchart of the method of the present invention. Detailed Description of the Embodiments

[0070] The present invention will be further described below in conjunction with the detailed embodiments. Among them, the drawings are only for illustrative purposes, showing only schematic diagrams, not physical diagrams, and should not be construed as a limitation of this patent; in order to better illustrate the embodiments of the present invention, some components in the drawings will be omitted, enlarged or reduced, which does not represent the size of the actual product; for those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0071] In the drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components; in the description of the present invention, it should be understood that if there are terms such as "upper", "lower", "left", "right", etc. indicating the orientation or positional relationship, they are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, the terms describing the positional relationship in the drawings are only for illustrative purposes and should not be construed as a limitation of this patent. For those of ordinary skill in the art, the specific meanings of the above terms can be understood according to specific circumstances.

[0072] Embodiment 1

[0073] As Figure 1 shown, a method for detecting pedestrians in the blind spot of vision based on vehicle-road cooperation. Cameras and lidar are installed on both the roadside and the vehicle side. Among them, the vehicle-side equipment includes a relatively fixed vehicle-mounted lidar and a vehicle-mounted camera, and the roadside equipment includes a relatively fixed roadside lidar and a roadside camera. It includes Coordinate System 1, a pedestrian detection method, and vehicle-road cooperation fusion. Among them, Coordinate System 1 includes the following steps:

[0074] S1.1: Use a calibration tool to jointly calibrate the camera and lidar on the same side to obtain the internal reference matrix K of the camera and the external reference matrix between the camera and the lidar

[0075] S1.2: The lidar collects point cloud P. Among them, the point cloud collected by the vehicle-mounted lidar is P1, and the point cloud collected by the roadside lidar is P2. Preprocess the point clouds P1 and P2 respectively. The preprocessing includes point cloud denoising, point cloud downsampling, point cloud ground segmentation, and region of interest extraction, so as to obtain the point clouds P1 roi and P2 roi .

[0076] S1.3: After the vehicle enters the communication range of the roadside device, use the vehicle GPS and IMU to determine the vehicle pose information, communicate to obtain the roadside device pose information, and calculate the initial value T of the transformation matrix GPS , with T GPS as the initial value, perform initial registration on the preprocessed point clouds P1 roi and P2 roi to obtain the transformation matrix T SAC , with T SAC as the initial value, perform precise registration to obtain the first coordinate system transformation matrix T init ;

[0077] S1.4: Based on the first coordinate system transformation matrix T init , estimate the movement of the vehicle relative to the first registration position Then continuously output the transformation matrix according to the coordinate transformation relationship

[0078] Pedestrian detection includes the following steps:

[0079] S2.1: The in-vehicle camera and the roadside camera respectively collect images, perform pedestrian detection on the images obtained by the cameras to obtain 2D plane detection frames, and project the point cloud P into the image coordinate system according to the internal reference matrix K and the external reference matrix , and count the point cloud set P that falls within the 2D plane detection frame in the image coordinate system detect ;

[0080] S2.2: Cluster the points in the point cloud set P aetect , select the category with the largest number of points as the pedestrian point cloud P person , and generate a sequence of point cloud bounding boxes B1;

[0081] S2.3: Perform 3D pedestrian detection on the point cloud P and generate a sequence of pedestrian point cloud bounding boxes B2;

[0082] S2.4: Match the sequences of pedestrian point cloud bounding boxes B1 and B2 formed by the above two methods, and use the weighted average method to obtain the final sequence of pedestrian bounding boxes B fusion ;

[0083] S2.5: Process the point cloud P with point clouds P1 and P2 as inputs respectively through steps S2.1 to S2.4 to obtain the sequence of pedestrian bounding boxes B1 fusion formed by the vehicle-side device processing and the sequence of pedestrian bounding boxes B2 fusion formed by the roadside device processing;

[0084] Vehicle-road collaborative fusion includes the following steps:

[0085] S3: According to the transformation matrix Convert the pedestrian point cloud bounding box B2fusion formed by the roadside device processing to the vehicle coordinate system, then the pedestrian bounding box sequence B1 fusion and the transformation matrix The converted pedestrian point cloud bounding box B2 fusion is the pedestrian detection result from the vehicle's perspective.

[0086] The present invention sets detection devices on both the roadside and the vehicle side, and each side's detection device includes a camera and a lidar.

[0087] In the camera and lidar on the same side, use a calibration tool to calibrate the image collected by the camera and the point cloud P collected by the lidar on the same side, and then the conversion relationship between the camera and the lidar on the same side can be obtained, that is, the internal reference matrix K and the external reference matrix Thus, the point cloud set P that falls within the plane detection frame of the image in the point cloud P is extracted detect , and then the pedestrian point cloud bounding box sequence B1 that combines the camera and the lidar is obtained. At the same time, only use the P collected by the lidar to form the pedestrian point cloud bounding box sequence B2, match and weighted average the pedestrian point cloud bounding box sequences B1 and B2 to obtain the final pedestrian bounding box sequence B fusion . Perform the above steps on the data collected by the detection devices on both sides, that is, perform step S1.1 and steps S2.1 - S2.5. The vehicle-side device obtains the pedestrian bounding box sequence B1 fusion , and the roadside device obtains the pedestrian bounding box sequence B2 fusion , then, the pedestrian bounding box sequence B2 on the roadside is required fusion to be converted to the vehicle's perspective. The disadvantage of the camera is that it cannot obtain accurate depth information, while the disadvantage of the lidar is that the perception data for objects at a relatively far distance is too sparse to be detected. A single sensor has limitations in the perception and detection of complex environments. Therefore, through the present invention, pedestrian detection is performed by combining image and point cloud data. The point cloud data supplements the depth information and blind spots lacking in the image data, while the image data supplements the blind spots near the point cloud and the defect of sparse data in the distance, forming a complement in perception.

[0088] When converting the pedestrian bounding box sequence B2 on the roadside fusion to the vehicle's perspective, the transformation matrix is obtained by registering the point clouds P1 and P2 collected by the lidars on the vehicle side and the roadside That is, perform steps S1.2 - S1.4, and then the pedestrian bounding box sequence B2 can be fusion converted to the vehicle's perspective. In this way, the pedestrian bounding box sequence B2 obtained by the roadside device fusion can be used to fusionSupplemented to achieve ultra-long-range perception of vehicle-road collaboration, and supplemented the perception data of the vehicle's vision blind area.

[0089] Among them, when registering the point clouds P1 and P2 collected by the lidars on both sides, the essence of lidar point cloud registration is to perform a spatial transformation on the point cloud according to the overlapping area. Point cloud registration has a good effect on the point cloud of rigid transformation. However, due to the different number of laser lines, angular distribution, and sampling resolution of the roadside lidar and vehicle-mounted lidar, the density of the two parts of the point cloud obtained is inconsistent, and these two parts of the point cloud are heterogeneous. In order to improve the registration accuracy of the present invention, the point clouds P1 and P2 are preprocessed to reduce noise and remove unnecessary areas such as the area above the vehicle roof, and then registered. And during the registration process, two registrations, namely initial registration and precise registration, are performed, which greatly improves the registration accuracy. Moreover, in the context of fast vehicle movement and high radar frame rate, the existing point cloud registration methods require frame-by-frame registration, with a large amount of calculation, and cannot meet the real-time requirement in the vehicle-road collaboration scenario. In order to achieve real-time registration and reduce the amount of calculation, estimate the movement of the vehicle relative to the first registration position According to the movement Output the transformation matrix in real time In this way, when the vehicle enters a set of roadside devices, that is, enters the communication range of the roadside lidar and roadside camera located at the same place, only the preprocessing and registration processes of step S1.2 and step S1.3 need to be performed once to obtain the first coordinate system transformation matrix T init , and then through the first coordinate system transformation matrix T init and the movement of the vehicle relative to the first registration position The transformation matrix can be output in real time until entering the communication range of the next set of roadside devices.

[0090] In this embodiment, the calibration tool uses the Calibration Tool Kit in the Autoware framework.

[0091] Further, when performing joint calibration in step S1.1, when the camera and lidar on the same side simultaneously observe point M, the projection coordinates of point M in the image coordinate system are U=(u, v, 1) T , and the coordinates of point M in the lidar coordinate system are M l (x l , y l , z l );

[0092] Let the transformation relationship between the lidar and the camera be Then

[0093]

[0094] Among them is the internal parameter matrix of the camera. The transformation matrix can be solved by collecting at least four groups of data through the lidar and the camera on the same side.

[0095] In this solution, the coordinates (u, v) of the scanning point (x l , y l , z l ) in the image pixel coordinate system of the lidar point cloud can be calculated by and then obtained through transformation It is easy to know that a total of 12 parameters need to be solved, so more than 4 groups of corresponding lidar points are required to obtain the result. Therefore, multiple groups of data can be used in the calibration, and the least squares method is used to solve the external reference matrix

[0096] Furthermore, in step S1.2, when preprocessing the point cloud P, the following steps are included:

[0097] S1.2.1: Point cloud denoising: Assume the input point cloud is P input =(p1, p2,..., p n ), the set filtering radius is R filter , and the adjacent point number threshold is num th . The radius filtering algorithm is used for denoising to obtain the denoised point cloud set P filtered ;

[0098] When using the radius filtering algorithm for denoising, for each point p input in the point cloud set P i , if the number of adjacent points num i filter R within the sphere with p i ngb as the center and R th <num i , then p i is removed, otherwise p filtered is retained, and finally a denoised point cloud set P

[0099] S1.2.2: Point cloud downsampling: For the denoised point cloud set P filtered , the same-sized voxel downsampling is performed respectively. The three-dimensional space is divided into voxels with side lengths (δx, δy, δz). Assume the points within the voxel are P voroi =(p1, p2,..., p n ), then the centroid of all laser reflection points within this voxel is p c =(x c , y c , zc ), where the coordinate values of the centroid are

[0100]

[0101]

[0102]

[0103] Use this centroid p c to replace the points within the voxel, and traverse each voxel in turn, then the downsampled point cloud P ds ;

[0104] In the point cloud data fusion of multi-source lidars, different lidars often have different sampling points for the same object due to the distribution of laser beams, field of view angles, and resolutions, so the characteristics of the obtained point cloud data are different. This situation poses a great challenge to point cloud registration. Considering the above reasons, for the two point cloud sets used for registration, voxel downsampling of the same specification can be performed first to improve the registration accuracy.

[0105] S1.2.3: Point cloud ground segmentation: For the downsampled point cloud P ds remove the point cloud of the ground part to obtain the point cloud set P ng without the ground part;

[0106] The reflected point cloud formed by the mechanical lidar emitting laser to the ground is generally a concentric arc centered on the lidar. When the positions of two lidars are different, even for the same piece of ground, the point characteristics on these arcs are completely different, which may interfere with the registration of the point cloud. Therefore, it is necessary to segment the point cloud of the ground. For the segmentation of the point cloud of the ground part, an algorithm based on Random Sample Consensus (RANSAC) can be used to obtain the point cloud set P ng

[0107] S1.2.4: According to the field of view angle of the roadside lidar, the distances to surrounding buildings and obstacles, and the area that needs to be concerned about during vehicle driving, remove the point cloud in the point cloud set P ng with a height exceeding a certain range, and only leave the point cloud data P roi of the region of interest.

[0108] Generally speaking, due to the different vertical field of view angles of different lidars, different positions, and different distances from objects, the acquired point cloud data is not consistent in height. It is very likely that some point clouds in a point cloud set are completely higher than those in another point cloud set. That is to say, there are no corresponding points in another point cloud set for these point clouds with too high height, which is not helpful for point cloud registration. If the vertical field of view angle of the vehicle-mounted lidar is larger and some data at higher heights is collected, such as the top of a tree, the wall of a higher floor, etc., there are no corresponding points for this part of the data in the point cloud of the roadside lidar. It not only cannot promote point cloud registration, but may also lead to an increase in registration time or an increase in registration error. On the other hand, the area of concern during vehicle driving is also limited, and usually only the scene within a certain height range above the ground is concerned. Therefore, the point cloud data above a certain height can also be discarded. Reducing redundant data is also beneficial for data transmission and processing, and also reduces the interference to the effective data in the area of interest.

[0109] S1.2.5: Respectively use the point cloud data P1 and P2 as the input point cloud sets P input Perform the processing of the above steps S1.2.1 to S1.2.4 to obtain the denoised point cloud sets P1 roi and P2 roi .

[0110] Furthermore, in the step S1.2.3, the point cloud of the ground part in the downsampled point cloud P ds is removed using the random sample consensus algorithm

[0111] Furthermore, in the step S1.3, on the vehicle side, the coordinate systems of the vehicle-mounted lidar, GPS, and IMU with relatively fixed pre-calibrated positions are calibrated in advance, and the vehicle is calibrated. The transformation matrix between the vehicle-mounted lidar coordinate system and the Earth-centered Earth-fixed coordinate system is obtained through the vehicle position and orientation information given by GPS and IMU, so as to determine the position of the vehicle-mounted lidar in the Earth-centered Earth-fixed coordinate system;

[0112] On the roadside, the coordinate system of the roadside lidar fixed on the roadside is pre-calibrated with the Earth-centered Earth-fixed coordinate system, so as to determine the coordinate position of the roadside lidar in the Earth-centered Earth-fixed coordinate system and broadcast the coordinate position;

[0113] According to the positions of the lidars on both sides in the Earth-centered Earth-fixed coordinate system, the initial value T of the transformation matrix is calculated GPS ; Using T GPS as the initial value, through the sampling consensus initial registration algorithm SAC-IA, the preprocessed point cloud P1 roi and P2 roi are initially registered to obtain the transformation matrix T SAC ; Using TSAC is the initial value. The point clouds P1 and P2 are precisely registered through the Iterative Closest Point (ICP) method to obtain the final first coordinate system transformation matrix T roi and P2 roi for the first time init .

[0114] Since the vehicle-mounted lidar, GPS, and IMU are fixedly installed on the vehicle, the vehicle and the vehicle-mounted lidar can be calibrated in advance, and the coordinate systems of GPS, IMU, and the vehicle-mounted lidar are all transformed into the Earth-Centered Earth-Fixed (ECEF) coordinate system. The direction information of the vehicle is obtained using the information provided by GPS and IMU to determine the transformation matrix between the lidar coordinate system and the ECEF coordinate system. Since the present invention also uses IMU for initial pose estimation, compared with relying only on GPS position information, the fusion accuracy and stability of the present invention are higher. At the same time, the initial accuracy obtained through the information provided by GPS and IMU is higher than that obtained only through GPS, thereby obtaining a transformation matrix T with high initial accuracy SAC , and after precisely registering the point clouds P1 roi and P2 roi through the Iterative Closest Point (ICP) method for the first time, the obtained first coordinate system transformation matrix T init has higher fusion accuracy

[0115] The conversion of the GPS WGS-84 coordinate system to the Earth-Centered Earth-Fixed (ECEF) coordinate system is as follows:

[0116]

[0117] where alt is the elevation, lat is the latitude, lon is the longitude, the semi-major axis a of the reference ellipsoid of the WGS-84 coordinate system is 6378137.0 m, and the polar flattening of the reference ellipsoid

[0118] Further, in S1.4, the transformation matrix of the vehicle relative to the first registration position at the moment of obtaining the i-th frame of point cloud output by the LeGO-LOAM algorithm is Then the transformation matrix at the current moment is

[0119] Further, in step S2.1, according to the calibrated internal parameters K and external parameters of the camera and lidar using the formula

[0120]

[0121] the point cloud P is projected into the image coordinate system

[0122] Further, in step S2.2, for the point cloud P detectWhen clustering the points within, according to the performance of the lidar, two parameters (ε, MinPts) are selected. Among them, ε describes the neighborhood distance threshold of a certain data point, and MinPts describes the minimum number of data points in the neighborhood with a radius of ε for a data point. If there are at least MinPts points within the ε neighborhood corresponding to a point p, then point p is a core object. Use the DB-SCAN algorithm to cluster the point cloud P detect Cluster the points within, and arbitrarily select a core object p without a category i As a seed, find all the samples of this core object p i A cluster C that can be density-reachable to MinPts k , until all points are traversed completely, and select the point cloud cluster C with the largest number of points k As the pedestrian point cloud P person , according to P person Form an axis-aligned bounding box (AABB) sequence

[0123] At the beginning of the DB-SCAN algorithm, we arbitrarily select a core object p1 without a category as a seed, and then find all the sample sets C1 that this core object can be density-reachable to, that is, a cluster. Then continue to select another core object without a category to find the density-reachable sample set C2,..., C n , until all points are traversed completely. Select the point cloud cluster C with the largest number of points k As the pedestrian point cloud P person , according to P person Form an axis-aligned bounding box (AABB) sequence

[0124] Furthermore, in the step S2.3, the Point-Pillar detection method is used for 3D pedestrian detection. First, the point cloud P is encoded using the Pillar encoding method to obtain a pseudo-2D image, then the encoded pseudo-2D image is processed using 2D convolution, and finally the detection head of SSD is used to detect the target to obtain a bounding box sequence

[0125] The Point Pillar detects the point cloud and first completes the conversion of all point cloud sets P detect To the conversion of the pseudo-image, cut on the x and y axes in the three-dimensional space to generate individual Pillars. Each single point cloud is represented by a 9-dimensional augmented vector, and the sparse point cloud is converted into dense data using two methods: truncation and padding. Finally, the point cloud P detectThey will all gather on a dense Tensor of size (D*N*P). After applying a simplified version of PointNet and a linear layer to each point cloud, the features of the point cloud are learned to generate a Tensor of size (C*P*N). Then, the point cloud is moved back to its original position according to the index to generate a pseudo-image. After the progressive downsampling process of the backbone, the corresponding features are upsampled to a unified size and concatenated. Finally, SSD detection is used to form a sequence

[0126] Further, in step S2.4, taking the center position of the bounding box as a reference, the distances between the bounding boxes in bounding box sequence B1 and bounding box sequence B2 can be obtained. Taking the minimum distance sum as the optimization objective, the Hungarian algorithm is used for pairwise matching to obtain the corresponding relationship between bounding box sequence B1 and bounding box sequence B2

[0127] Among them, and are the i-th and j-th bounding boxes in bounding box sequence B1 and bounding box sequence B2 respectively.

[0128] Embodiment 2

[0129] This embodiment is similar to Embodiment 1. The difference is that in this embodiment, the calibration tool used is Sensor Calibration in the Apollo 2.0 framework.

[0130] Embodiment 3

[0131] This embodiment is similar to Embodiment 1. The difference is that in this embodiment, when clustering the points in point cloud P detect the k-means clustering method is used. The point cloud pedestrian detection model is SECOND.

[0132] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, rather than limitations on the implementation manners of the present invention. For those of ordinary skill in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to enumerate all the implementation manners here. Any modifications, equivalent replacements, and improvements made within the spirit and principle of the present invention shall be included within the protection scope of the claims of the present invention.

Claims

1. A method for detecting pedestrians in a blind spot based on vehicle-road cooperation, characterized in that Cameras and lidar are installed on both the roadside and the vehicle side. The vehicle-side devices include a relatively fixed vehicle-mounted lidar and a vehicle-mounted camera, and the roadside devices include a relatively fixed roadside lidar and a roadside camera, including a coordinate system one, a pedestrian detection method, and vehicle-road collaborative fusion. Among them, the coordinate system one includes the following steps: S1.1: Use a calibration tool to jointly calibrate the camera and lidar on the same side to obtain the internal reference matrix of the camera and the external reference matrix between the camera and the lidar ; S1.2: The lidar collects point cloud P, where the point cloud collected by the vehicle-mounted lidar is P1 and the point cloud collected by the roadside lidar is P2. Preprocess the point clouds P1 and P2 respectively. The preprocessing includes point cloud denoising, point cloud downsampling, point cloud ground segmentation, and region of interest extraction, so as to obtain the point cloud that can be used for registration and ; S1.3: After the vehicle enters the communication range of the roadside device, use the vehicle GPS and IMU to determine the vehicle pose information, communicate to obtain the roadside device pose information, and calculate the initial value of the transformation matrix , with as the initial value, preprocess the point cloud and for initial registration to obtain the transformation matrix , with as the initial value, perform precise registration to obtain the first coordinate system transformation matrix ; S1.4: Based on the first coordinate system transformation matrix , estimate the motion of the vehicle relative to the first registration position , and then continuously output the transformation matrix according to the coordinate transformation relationship; The pedestrian detection includes the following steps: S2.1: The in-vehicle camera and the roadside camera respectively collect images, perform pedestrian detection on the images obtained by the cameras to obtain 2D plane detection frames, and project the point cloud P into the image coordinate system according to the internal reference matrix and the external reference matrix , and count the point cloud set that falls within the range of the 2D plane detection frame in the image coordinate system ; S2.2: Cluster the points in the point cloud set and select the cluster with the largest number of points as the pedestrian point cloud and generate a sequence of point cloud bounding boxes ; S2.3: Perform 3D pedestrian detection on the point cloud P and generate a sequence of pedestrian point cloud bounding boxes ; S2.4: Match the pedestrian point cloud bounding box sequence and and use the weighted average method to obtain the final pedestrian bounding box sequence ; S2.5: Process the point cloud P with the point clouds P1 and P2 as inputs respectively through steps S2.1 to S2.4, and respectively obtain the pedestrian bounding box sequences formed by the vehicle-side devices and the pedestrian bounding box sequences formed by the roadside devices ; The vehicle-road collaborative fusion includes the following steps: S3: According to the transformation matrix , transform the pedestrian point cloud bounding box formed by the roadside device processing into the vehicle coordinate system, then the pedestrian bounding box sequence and the pedestrian point cloud bounding box after transformation by the transformation matrix are the pedestrian detection results from the vehicle's perspective.​ 2. The method for detecting pedestrians in a vision blind area based on vehicle-road cooperation according to claim 1, wherein, When performing joint calibration in the step S1.1, when the camera and the lidar on the same side simultaneously observe the point M, the projected coordinates of the point M in the image coordinate system are , and the coordinates in the lidar coordinate system are ; Assume the conversion relationship between the lidar and the camera is , then Among them is the internal parameter matrix of the camera. The transformation matrix can be solved by collecting at least four groups of data respectively through the lidar and the camera on the same side .

3. The method for detecting pedestrians in a blind spot based on vehicle-road cooperation according to claim 1, wherein, In the step S1.2, when preprocessing the point cloud P, the following steps are included: S1.2.1: Point cloud denoising: Assume the input point cloud is , the set filtering radius is , the threshold of the number of neighboring points is , use the radius filtering algorithm for denoising to obtain the denoised point cloud set ; S1.2.2: Point cloud downsampling: For the denoised point cloud set perform voxel downsampling of the same specification respectively, and divide the three-dimensional space into voxels with side length . Let the points in the voxel be , then the centroid of all laser reflection points in this voxel is , where the coordinate value of the centroid point is Using this center of gravity to replace the points within the voxel and traversing each voxel in sequence forms the downsampled point cloud ; S1.2.3: Point cloud ground segmentation: For the downsampled point cloud remove the point cloud of the ground part to obtain a set of point clouds with the ground part removed ; S1.2.4: Remove the point clouds with heights exceeding a certain range from the point cloud set according to the field of view angle of the roadside lidar, the surrounding buildings, the distances of obstacles, and the areas that need to be concerned about during vehicle driving, and only leave the point cloud data of the area of interest ; ; S1.2.5: Respectively take the point cloud data P1 and P2 as the input point cloud sets Perform the processing of the above steps S1.2.1 to S1.2.4 to obtain the denoised point cloud sets and .

4. The method for detecting pedestrians in a blind spot based on vehicle-road cooperation according to claim 3, wherein, In the step S1.2.3, the point cloud of the ground part in the downsampled point cloud is removed by using the random sample consensus algorithm. ​ 5. The vision blind area pedestrian detection method based on vehicle-road cooperation according to claim 3, characterized in that In the step S1.3, on the vehicle side, the vehicle-mounted lidar coordinate system, the GPS and IMU coordinate systems with relatively fixed pre-calibrated positions, and the vehicle are calibrated. The transformation matrix between the vehicle-mounted lidar coordinate system and the Earth-centered Earth-fixed coordinate system is obtained through the vehicle position and orientation information given by the GPS and IMU, so as to determine the position of the vehicle-mounted lidar in the Earth-centered Earth-fixed coordinate system; On the roadside, the coordinate system of the roadside lidar fixed on the roadside is pre-calibrated with the Earth-centered Earth-fixed coordinate system, so as to determine the coordinate position of the roadside lidar in the Earth-centered Earth-fixed coordinate system and broadcast this coordinate position; Calculate the initial value of the transformation matrix based on the positions of the lidars on both sides in the Earth-centered Earth-fixed coordinate system ; With as the initial value, through the Sampling Consensus Initial Alignment algorithm SAC-IA, perform initial alignment on the preprocessed point clouds and to obtain the transformation matrix ; With as the initial value, through the Iterative Closest Point method ICP, perform precise alignment on the point clouds and to obtain the final first coordinate system transformation matrix .

6. The method for detecting pedestrians in a vision blind area based on vehicle-road collaboration according to claim 1, wherein, In S1.4, the transformation matrix of the vehicle output by the LeGO-LOAM algorithm at the moment of obtaining the th frame of point cloud relative to the first registration position is , then the transformation matrix at the current moment is .

7. The method for detecting pedestrians in a blind spot based on vehicle-road cooperation according to claim 2, wherein In the step S2.1, according to the calibrated internal parameters of the camera and the lidar and the external parameters , use Equation Project the point cloud P into the image coordinate system.

8. The method for detecting pedestrians in a blind spot based on vehicle-road cooperation according to claim 7, wherein, In the step S2.2, when clustering the points in the point cloud , according to the performance of the lidar, two parameters ( ) are selected. Among them, describes the neighborhood distance threshold of a certain data point, and MinPts describes the minimum number of data points in the neighborhood with a radius of for a data point. If there are at least MinPts points in the corresponding neighborhood of a point, then the point is a core object. The DB-SCAN algorithm is used to cluster the points in the point cloud . Arbitrarily select a core object without a category as a seed, and find all the clustering clusters that can reach MinPts in density for this core object , until all points are traversed completely. Select the point cloud cluster with the largest number of points as the pedestrian point cloud , and form an axis-aligned bounding box (AABB) sequence according to .

9. The method for detecting pedestrians in a vision blind area based on vehicle-road cooperation according to claim 1, wherein In the step S2.3, the Point-Pillar detection method is used for 3D pedestrian detection. First, the point cloud P is encoded by the Pillar encoding method to obtain a pseudo 2D map. Then, the encoded pseudo 2D map is processed using 2D convolution. Finally, the detection head of SSD is used to detect the target, and a bounding box sequence is obtained. .

10. The method for detecting pedestrians in a blind spot based on vehicle-road cooperation according to claim 1, wherein, In the step S2.4, taking the center position of the bounding box as a reference, a bounding box sequence can be obtained and the distance between each pair of the bounding box sequences is calculated. With the minimum distance sum as the optimization goal, the Hungarian algorithm is used for pairwise matching to obtain the correspondence between the bounding box sequence and the bounding box sequence ( , ).

Images