Highly automated driving inspection method and system based on multi-sensor fusion deployment

Abstract

The present application relates to the field of deep learning technology, and specifically to a highly automated driving inspection method and system based on multi-sensor fusion deployment. The method includes: collecting the position of the target vehicle and its corresponding three-dimensional point cloud data, two-dimensional image, and inspection target position through multiple sensors; constructing three-dimensional space into voxels from the three-dimensional point cloud data, and obtaining the downsampling degree of the voxels based on the distance between the points in the voxels and the target vehicle and the local density of the points; performing downsampling based on this to obtain the downsampled three-dimensional point cloud data; training the three-dimensional point cloud data and two-dimensional image, and then fusing them with the position of the target vehicle to obtain a target positioning map within the perception range of the target vehicle; combining the target positioning map with the inspection target position to obtain path planning, and complete autonomous inspection of the target vehicle. The present application improves the accuracy of obtaining the target positioning map, making automatic inspection more accurate.

Description

Technical Field

[0001] The present application relates to the field of deep learning technology, and specifically to a highly automated driving inspection method and system based on multi-sensor fusion deployment. Background Art

[0002] With the development of autonomous driving technology, autonomous driving vehicles have been applied to many fields, such as highway road inspections, substation inspections, factory safety inspections, etc.

[0003] Highly automated driving is level 4 in the level of automated driving. Under the designed operating conditions, vehicles of this level can complete all driving tasks and can also take over driving tasks. During the inspection process, highly automated driving vehicles use multiple sensors such as lidars, cameras, and Beidou positioning modules to achieve environmental perception and positioning, thereby achieving autonomous inspections in complex road environments. At present, the perception and positioning data of multiple sensors are integrated to improve the perception and positioning accuracy, so as to provide a better navigation route for automated vehicles during automatic inspections. Deep learning algorithms are usually used to detect obstacles such as pedestrians and vehicles on the three-dimensional point cloud images of lidar and the RGB images of cameras, and the target detection results are integrated with Beidou positioning signals to achieve perception and positioning of the surrounding environment. Since there are many redundant data in the three-dimensional point cloud, direct fusion will result in large computational complexity and redundant parts, so downsampling is usually performed. However, the traditional downsampling method is to downsample the points in all areas of the three-dimensional point cloud data, which will result in the loss of some important features, missed detection, and blurred perception of the surrounding environment, making it impossible to complete the automated driving inspection well. Summary of the invention

[0004] In order to solve the technical problem of blurred perception of the surrounding environment, this application provides a highly automated driving inspection method and system based on multi-sensor fusion deployment. The technical solutions adopted are as follows:

[0005] In the first aspect, the present application proposes a highly automated driving inspection method based on multi-sensor fusion deployment, the method comprising the following steps:

[0006] Collect the position of the target vehicle and its corresponding 3D point cloud data and 2D image through multiple sensors; obtain the inspection target position;

[0007] Construct a three-dimensional space based on the coordinates of the points in the three-dimensional point cloud data to obtain the distance between each point and the target vehicle; divide the three-dimensional space into voxels of the same size, and obtain the distance coefficient of each voxel based on the distance between all points in each voxel and the target vehicle; obtain the local density of each point in the three-dimensional space through a density clustering algorithm, and obtain the downsampling degree of the voxel based on the distance coefficient of each voxel, the local density of all points in the voxel, and the number of points in the voxel;

[0008] Down-sample the voxels according to the down-sampling degree of the voxels to obtain down-sampled three-dimensional point cloud data;

[0009] Perform annotation training on the downsampled 3D point cloud data to obtain the 3D bounding box, category and distance of each target to be detected; perform annotation training on the 2D image to obtain the 3D bounding box, category and position of each target to be detected; match the 3D point cloud data with the target to be detected in the 2D image, and fuse the position of the target to be detected after matching with the position of the target vehicle to obtain a target positioning map within the perception range of the target vehicle;

[0010] The target positioning map and inspection target position are used as inputs of the search algorithm to obtain the path planning of the target vehicle at the current moment and complete the autonomous inspection of the target vehicle.

[0011] In the above scheme, the downsampling degree is constructed based on the distance characteristics and density characteristics of the target to be detected, which is used to characterize how a voxel needs to be downsampled; the side length of the voxel in the voxel grid downsampling method is reset according to the size of the downsampling degree, and the features of the target to be detected are retained as much as possible, thereby increasing the calculation speed of the deep learning model and reducing the missed detection rate of the model; a deep learning algorithm is used to perform target detection on the three-dimensional point cloud image and the two-dimensional RGB image at the current moment, respectively, and the point cloud detection results, image detection results, and Beidou positioning results are fused to obtain accurate target positioning results within the perception range at the current moment.

[0012] In one embodiment, three-dimensional point cloud data around the target vehicle is collected by laser radar, two-dimensional images are collected by visual sensors, and the position of the target vehicle is obtained by Beidou positioning signals.

[0013] In one embodiment, the method of constructing a three-dimensional space according to the coordinates of the points in the three-dimensional point cloud data and obtaining the distance between each point and the target vehicle is:

[0014] Obtain the coordinates of all points in the three-dimensional point cloud data, take the coordinate system of the laser radar as the reference, obtain the coordinates on the x, y, and z axes of the point cloud data, obtain the maximum and minimum values ​​on the x, y, and z axes of the three-dimensional point cloud data, and record the space formed by the maximum and minimum values ​​on the three axes as the three-dimensional space;

[0015] The coordinate of each point on the z-axis is the distance between each point and the target vehicle.

[0016] In one embodiment, the method for obtaining the distance coefficient of each voxel according to the distance between all points in each voxel and the target vehicle is:

[0017] For each voxel, the mean of the distances between all points and the target vehicle is calculated and recorded as the first mean, and the minimum distance is selected from the distances between all points and the target vehicle and recorded as the minimum distance;

[0018] The distance coefficient of the voxel is positively correlated with the first mean and the minimum distance.

[0019] In one embodiment, the method for obtaining the downsampling degree of a voxel according to the distance coefficient of each voxel, the local density of all points in the voxel, and the number of points in the voxel is:

[0020] The mean of the local density of all points within the voxel is calculated as the average local density of the voxel;

[0021] The degree of downsampling of each voxel is negatively correlated with the average local density of the voxel and the number of points within the voxel, and is positively correlated with the distance coefficient of the voxel.

[0022] In one embodiment, the method of downsampling voxels according to the downsampling degree of the voxels to obtain the downsampled three-dimensional point cloud data is:

[0023] If M i ∈[0,0.25), then m=3c; if M i ∈[0.25,0.5), then m=2c; if M i ∈[0.5,0.75), then m=1c; if M i ∈[0.75,1], then m=0; where w i ′ represents the side length of the small voxel on the x-axis after the i-th voxel is split into small voxels, h i ′ represents the side length of the small voxel on the y-axis after the i-th voxel is split into small voxels, l i ′ represents the side length of the small voxel on the z-axis after the i-th voxel is split into small voxels, c is a preset positive integer, M i represents the downsampling degree of the i-th voxel, w represents the side length of the voxel on the x-axis, h represents the side length of the voxel on the y-axis, and l represents the side length of the voxel on the z-axis.

[0024] In one embodiment, the method for labeling training of three-dimensional point cloud data is:

[0025] Pedestrians, vehicles, traffic lights and lane lines are classified and labeled. For the same category, each target is labeled with a labeling box. The labeled data is used as the neural network input, and the cross entropy loss function is used as the classification loss function. The CIOU loss function is used as the bounding box regression loss to obtain the 3D point cloud target detection model. The 3D point cloud target detection model is used to detect each target to be detected to obtain the 3D bounding box, category and distance of the target to be detected.

[0026] In one embodiment, the method for training annotation of a two-dimensional image is:

[0027] Pedestrians, vehicles, traffic lights and lane lines are classified and labeled, and each target of the same category is labeled with a labeling box; the camera's intrinsic parameter matrix is ​​obtained through the camera calibration method; the labeled two-dimensional image and intrinsic parameter matrix are used as neural network input, the cross entropy loss function is used as the classification loss function, and the angle regression loss function and size regression loss function are used as the bounding box regression loss to obtain the 3D bounding box, category and position of the target to be detected.

[0028] In one embodiment, the method of fusing the positions of the target to be detected and the target vehicle after matching to obtain a target positioning map within the sensing range of the target vehicle is:

[0029] Obtain the 3D point cloud data, 2D image and the position of the target vehicle collected at each moment; calculate the intersection-and-union ratio between the 3D bounding box of the 3D point cloud image and the 3D bounding box of the 2D image at the same moment; if the value of the intersection-and-union ratio is greater than the preset value, the match is successful;

[0030] For the successfully matched 3D bounding box, the position of the corresponding target to be measured and the distance to the target vehicle can be obtained. The position of the target vehicle can be obtained through satellite positioning, and the target to be measured is reflected in the satellite positioning map to obtain the target positioning map within the perception range of the target vehicle.

[0031] In a second aspect, an embodiment of the present application also provides a highly automated driving inspection system based on multi-sensor fusion deployment, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein when the processor executes the computer program, the steps of any one of the above-mentioned highly automated driving inspection methods based on multi-sensor fusion deployment are implemented.

[0032] The beneficial effects of this application are:

[0033] Traditionally, downsampling of three-dimensional point cloud data is global downsampling, so a lot of useful information will be lost. The present application constructs a downsampling degree based on the distance characteristics and density characteristics of the target to be detected, which is used to characterize how a voxel needs to be downsampled; the side length of the voxel in the voxel grid downsampling method is reset according to the size of the downsampling degree, and the features of the target to be detected are retained as much as possible, which increases the calculation speed of the deep learning model while reducing the missed detection rate of the model; a deep learning algorithm is used to perform target detection on the three-dimensional point cloud image and the two-dimensional RGB image at the current moment, and the point cloud detection results, image detection results, and Beidou positioning results are integrated to obtain an accurate target positioning result within the perception range at the current moment. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0035] Figure 1 A flow chart of a highly automated driving inspection method based on multi-sensor fusion deployment provided by one embodiment of the present application;

[0036] Figure 2 Sensor layout for highly automated vehicles. DETAILED DESCRIPTION

[0037] In order to further explain the technical means and effects adopted by the present application to achieve the predetermined invention purpose, the following, in combination with the accompanying drawings and preferred embodiments, describes in detail the specific implementation methods, structures, features and effects of the highly automated driving inspection method and system based on multi-sensor fusion deployment proposed in the present application. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.

[0038] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0039] An embodiment of a highly automated driving inspection method based on multi-sensor fusion deployment:

[0040] The specific scheme of the highly automated driving inspection method based on multi-sensor fusion deployment provided by the present application is described in detail below with reference to the accompanying drawings.

[0041] See also Figure 1, which shows a flow chart of a highly automated driving inspection method based on multi-sensor fusion deployment provided by an embodiment of the present application, the method comprising the following steps:

[0042] Step S001, collecting the position of the target vehicle and its corresponding three-dimensional point cloud data, two-dimensional image, and inspection target position through multiple sensors.

[0043] For highly automated driving vehicles, a BeiDou RTK positioning module is installed on the top of the robot to receive BeiDou positioning signals. Two rotating laser radars are placed on the left and right front for sensing in front of and on both sides of the robot. A solid-state laser is deployed at the rear to supplement the vehicle's rear sensing. Three visual sensors are deployed in the middle and left and right of the top of the robot. In this way, reliable, accurate and stable positioning and environmental perception can be obtained based on the perceived environment around it. The specific layout is as follows: Figure 2 shown.

[0044] For highly automated driving vehicles according to sensors, they are recorded as target vehicles, and sensor data of the target vehicles are collected, including point cloud data in LAS format of the lidar, continuous frame images of the visual sensor, and Beidou positioning signals, to obtain a multimodal data set. There are four types of targets in the data set, including traffic lights, pedestrians, vehicles, and lane lines. The acquired point cloud data are point cloud data of these four types of targets. The collected data are pre-processed, and bilateral filtering technology is used to remove noise in three-dimensional point cloud data and two-dimensional images to improve data quality. Bilateral filtering technology is a well-known technology, and the specific process will not be repeated.

[0045] In this application, inspections are to be carried out by vehicles. Therefore, this embodiment takes the inspection of substations as an example. During the inspection of substations, different transformers of the substations need to be inspected. The transformers are taken as inspection targets, and the positions of all transformers in the substation are obtained as inspection target positions.

[0046] At this point, the three-dimensional point cloud data, image, inspection target position and position of the target vehicle are obtained.

[0047] Step S002, construct a three-dimensional space according to the coordinates of the points in the three-dimensional point cloud data, and obtain the distance between each point and the target vehicle; divide the point cloud space into voxels of the same size, and obtain the distance coefficient of each voxel according to the distance between all points in each voxel and the target vehicle; obtain the local density of each point in the three-dimensional space through a density clustering algorithm, and obtain the downsampling degree of the voxel according to the distance coefficient of each voxel, the local density of all points in the voxel and the number of points in the voxel.

[0048] The key technologies of highly automated driving vehicles are divided into four parts: environmental perception, autonomous positioning, path planning, and decision control. Among them, the environmental perception part is completed through lidar sensors and camera sensors. Lidar can provide high-precision three-dimensional point cloud data and have very accurate measurements of the distance and position of surrounding objects; the camera can provide visual information such as traffic lights, pedestrians, vehicles, and lane lines in the surrounding environment. The Beidou positioning module obtains satellite positioning results and obtains the location information of the target vehicle.

[0049] By using data fusion technology to fuse the three-dimensional point cloud data of the lidar, the two-dimensional image data of the camera, and the satellite positioning results, it is possible to achieve comprehensive perception and positioning of the environment, allowing autonomous vehicles to drive safely and reliably in various environments. Sensor fusion methods are divided into front fusion and back fusion. This embodiment uses the back fusion method for data fusion. First, use the deep learning method to perform target detection on the three-dimensional point cloud and the two-dimensional image respectively, fuse the obtained detection results with the satellite positioning results, and finally obtain the perception and positioning results of the environment.

[0050] For the 3D point cloud data collected by LiDAR, due to the excessive number of point clouds and the inclusion of a lot of redundant data, the amount of computation required by the deep learning algorithm to process the data will increase, resulting in the inability to quickly calculate features. Usually, the point cloud data needs to be downsampled, but the traditional downsampling method is to downsample the points in all areas of the 3D point cloud data, which will cause some important features to be lost and miss detections to occur. Different point cloud data require different accuracies, so the degree of downsampling is also different.

[0051] The point cloud data file in LAS format is read through Open3D, where the coordinates of the point cloud data are based on the coordinate system of the laser radar. The coordinates of the point cloud data on the x, y, and z axes are obtained, and the maximum and minimum values ​​of all point cloud data on the x, y, and z axes are obtained. A space formed by the maximum and minimum values ​​is recorded as the point cloud space.

[0052] The point cloud space is evenly divided into k cubic grids, each of which has a side length of w×h×l, and each grid is a voxel. Each point is assigned to the corresponding voxel according to its coordinates. Since the laser radar coordinate system is used, the z value of each point in the point cloud data represents the distance between each point and the target vehicle, which is recorded as the distance of each point; in this embodiment, the side length of the three-dimensional grid is the ratio of the maximum side length of the point cloud space to 50.

[0053] For pedestrians and vehicles in the 3D point cloud, the closer they are to the autonomous vehicle, the more they will affect the normal driving of the autonomous vehicle. Therefore, the features of close objects should be retained as much as possible, while the features of distant objects can be downsampled to a greater extent. Since the lane lines and the road surface in the 3D point cloud are in the same plane, but due to the different reflectivity of the lane lines and the road surface to the lidar, the lane lines will eventually appear more densely in the 3D point cloud, while the road surface will appear relatively sparsely in the 3D point cloud. In addition, for traffic lights, their corresponding points will also appear relatively densely in the 3D point cloud.

[0054] Based on the above features, the downsampling degree of each voxel is constructed. First, the distance of each point in each voxel is obtained, the mean of the distances of all points is calculated, and the distance coefficient of each voxel is obtained according to the mean of the distances of all points in each voxel and the minimum value of the distances of the points.

[0055] The distance coefficient is positively correlated with the mean of the distances of all points and the minimum of the distances of the points.

[0056] It should be noted that positive correlation means that when one variable increases, the other variable also increases, and the two variables change in the same direction. When one variable changes from large to small or from small to large, the other variable also changes from large to small or from small to large. This application does not impose any special restrictions.

[0057] Preferably, in this embodiment, the distance coefficient is represents the mean distance of all points in the i-th voxel, min(d i ) represents the minimum distance of the points within the i-th voxel, H i Represents the distance coefficient of the i-th voxel.

[0058] Preferably, in another embodiment of the present application, the distance coefficient is exp() represents an exponential function with a natural constant as the base.

[0059] The smaller the mean of all distances is, the smaller the overall distance of the voxels is, and the smaller the minimum distance of the voxel midpoints is, the smaller the lower limit distance of the voxels is. Based on this, the distance coefficient of each voxel is obtained.

[0060] The coordinates of all points in the three-dimensional point cloud in three-dimensional space are used as the input of the DCP density clustering algorithm. The cutoff distance is 1% of the number of all points in three-dimensional space. The local density of each point in three-dimensional space is obtained. The mean of the local density of all points in each voxel is calculated as the average local density of the voxel. The downsampling degree of each voxel is obtained according to the average local density of the voxel and the number of points in the voxel. The downsampling degree of each voxel is negatively correlated with the average local density of the voxel and the number of points in the voxel, and is positively correlated with the distance coefficient of the voxel.

[0061] It should be noted that negative correlation means that when one variable increases, the other variable decreases accordingly, and the two variables change in opposite directions. When one variable changes from large to small or from small to large, the other variable also changes from small to large or from large to small. This application does not impose any special restrictions.

[0062] Preferably, in this embodiment, the downsampling degree of the voxels is represents the average local density of the ith voxel, n i represents the number of points inside the i-th voxel, norm() represents the linear normalization function, M i Indicates the downsampling degree of the i-th voxel.

[0063] Preferably, in another embodiment of the present application, the downsampling degree of the voxel is norm() represents the maximum and minimum normalization function.

[0064] H i The larger the value is, the farther the point in the voxel is from the autonomous driving vehicle, which means that the voxel has less impact on the normal driving of the autonomous driving vehicle, so the voxel should be downsampled. The larger the values ​​of and n are, the denser the points in the ith voxel are, and the more likely they are lane lines and traffic lights, so a smaller degree of downsampling should be performed. o The larger the value of , the more the i-th voxel should be downsampled.

[0065] At this point, the downsampling degree of each voxel is obtained.

[0066] Step S003 , down-sample the voxels according to the down-sampling degree of the voxels to obtain down-sampled three-dimensional point cloud data.

[0067] When downsampling is performed using the voxel grid downsampling method, the larger the voxel volume, the greater the degree of downsampling and the fewer features are retained; the smaller the voxel volume, the smaller the degree of downsampling and the more features are retained.

[0068] Each voxel is split according to the downsampling degree of each voxel, and is divided into multiple small voxels. The size of a voxel is determined according to the downsampling degree to preserve more features. The smaller the downsampling coefficient, the more features should be preserved, and the voxel should be split into smaller voxels; the larger the downsampling coefficient, the less features should be preserved, and the voxel should be split into larger voxels.

[0069] Based on the above analysis, the side length of the voxel after splitting is calculated. The specific process is as follows:

[0070] If M o ∈[0,0.25), then m=3c; if M i ∈[0.25,0.5), then m=2c; if M i ∈[0.5,0.75), then m=1c; if M i ∈[0.75,1], then m=0. Among them w i ′ represents the side length of the small voxel on the x-axis after the i-th voxel is split into small voxels, h i ′ represents the side length of the small voxel on the y-axis after the i-th voxel is split into small voxels, l i ′ represents the side length of the small voxel on the z-axis after the i-th voxel is split into small voxels, w represents the side length of the voxel on the x-axis, h represents the side length of the voxel on the y-axis, l represents the side length of the voxel on the z-axis, c is a positive integer used to adjust the splitting degree of the voxel, and in this embodiment, c is 2;

[0071] Thus, the voxel is divided into several small voxels according to the degree of downsampling of each voxel. The smaller the side length of the small voxels into which the voxel is split, the more features are retained after downsampling. The voxel grid downsampling method is used to downsample the small voxels, and all the points in the voxel are replaced by an average position of all the points in the voxel. Voxel grid downsampling is a well-known technology and will not be described in detail here.

[0072] After downsampling all voxels, downsampled three-dimensional point cloud data is obtained.

[0073] At this point, the downsampled three-dimensional point cloud data is obtained.

[0074] Step S004, perform annotation training on the downsampled three-dimensional point cloud data to obtain the 3D bounding box, category and distance of each target to be detected; perform annotation training on the two-dimensional image to obtain the 3D bounding box, category and position of each target to be detected; match the three-dimensional point cloud data with the target to be detected in the two-dimensional image, and fuse the position of the target to be detected after matching with the position of the target vehicle to obtain a target positioning map within the perception range of the target vehicle.

[0075] The point cloud annotation tool LabelCloud is used to annotate the objects in the downsampled 3D point cloud data with a 3D rectangular box, and pedestrians, vehicles, traffic lights and lane lines are annotated. The annotation method is to annotate the points belonging to pedestrians as 1, the points belonging to vehicles as 2, the points belonging to traffic lights as 3, and the points belonging to lane lines as 4, and annotate individual pedestrians, individual vehicles, and individual traffic lights respectively through the annotation box. The annotated 3D point cloud data is used as the input of the FVNet network, the Adam optimizer is used as the optimization algorithm of the model, the cross entropy loss function is used as the classification loss function of the model, and the CIOU loss function is used as the border regression loss of the model. After 300 rounds of iterative training, a 3D point cloud target detection model is obtained. Neural network training is a well-known technology, and the specific process will not be repeated.

[0076] Each annotated annotation box is recorded as a target to be detected. The 3D point cloud target detection model is used to detect the 3D point cloud data. Each type of annotated 3D point cloud is recorded as a target to be detected, and finally the 3D bounding box, category and distance of the target to be detected are obtained.

[0077] The camera's intrinsic parameter matrix is ​​obtained by the camera calibration method. Camera calibration is a well-known technology, and the specific process will not be repeated. The collected two-dimensional image is annotated using the labeling software LabelMe, and pedestrians, vehicles, traffic lights and lane lines are annotated. The labeling method is to mark the points belonging to pedestrians as 1, the points of vehicles as 2, the points of traffic lights as 3, and the points of lane lines as 4, and use the labeling box to mark a single pedestrian, a single vehicle, and a single traffic light. The annotated two-dimensional image and the camera's intrinsic parameter matrix are used as the input of the Deep3DBox network, and the Adam optimizer is used as the optimization algorithm of the model, the cross entropy loss function is used as the classification loss function of the model, and the angle regression loss function and the size regression loss function are used as the frame regression loss of the model. After 300 rounds of iterative training, the image target detection model is obtained. Neural network training is a well-known technology, and the specific process is not repeated. By using the image target detection model to detect the two-dimensional image captured by the camera, the 3D bounding box, category and position of the target to be detected can be obtained. In this embodiment, the Deep3Dbox network can detect the 3D bounding box and position in the two-dimensional image.

[0078] The Beidou vehicle positioning system receives signals from Beidou navigation satellites and uses the principle of triangulation to calculate the exact location of the vehicle. These signals contain information such as the satellite number, timestamp, and navigation data. By communicating with the vehicle terminal, this information can be obtained and analyzed in real time to determine the location of the target vehicle.

[0079] The three-dimensional point cloud data, two-dimensional image and the position of the target vehicle collected at each moment are obtained. The intersection over union (IOU) between the 3D bounding box of the three-dimensional point cloud image and the 3D bounding box of the two-dimensional image at the same moment is calculated, and the matching degree of the two targets is determined according to the size of the IOU. The calculation of the IOU is a well-known technology and the specific process is not repeated here.

[0080] In this embodiment, if the value of the intersection-and-union ratio is greater than 0.8, it means that the match is successful. The successfully matched 3D bounding box will be used as the final detection result at that moment. Then for the same target, the distance and category of the target to be detected can be obtained based on the detection results of the three-dimensional point cloud; the category and position of the target to be detected can be obtained based on the detection results of the two-dimensional image, and the position of the target vehicle at the current moment can be determined based on the results of satellite positioning. The point cloud detection results, image detection results and satellite positioning results obtained at the same time are fused, the position of the target vehicle is marked on the satellite image, and the positions of all targets to be detected around the target vehicle and the position of the target vehicle are obtained through the 3D point cloud and the two-dimensional image, and the target positioning map within the perception range of the target vehicle at the current moment is obtained.

[0081] Step S005, using the target positioning map and the inspection target position as inputs to the search algorithm, obtaining the path planning of the target vehicle at the current moment, and completing the autonomous inspection of the target vehicle.

[0082] In this application, to complete the vehicle inspection, the target positioning map and the inspection target position are used as the input of the A* search algorithm to obtain the path planning of the autonomous driving vehicle at the current moment. The A* search algorithm is a well-known technology and the specific process is not repeated here. The vehicle is driven according to the path planning results to complete the autonomous inspection of the autonomous driving vehicle during the inspection process.

[0083] Based on the same inventive concept as the above method, an embodiment of the present invention also provides a highly automated driving inspection system based on multi-sensor fusion deployment, including a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, it implements the steps of any one of the above-mentioned highly automated driving inspection methods based on multi-sensor fusion deployment.

[0084] It should be noted that the above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

[0085] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

Claims

1. A highly automated driving inspection method based on multi-sensor fusion deployment, characterized in that: The method comprises the following steps: Collect the position of the target vehicle and its corresponding 3D point cloud data and 2D image through multiple sensors; obtain the inspection target position; Construct a three-dimensional space based on the coordinates of the points in the three-dimensional point cloud data to obtain the distance between each point and the target vehicle; divide the three-dimensional space into voxels of the same size, and obtain the distance coefficient of each voxel based on the distance between all points in each voxel and the target vehicle; obtain the local density of each point in the three-dimensional space through a density clustering algorithm, and obtain the downsampling degree of the voxel based on the distance coefficient of each voxel, the local density of all points in the voxel, and the number of points in the voxel; Down-sample the voxels according to the down-sampling degree of the voxels to obtain down-sampled three-dimensional point cloud data; Perform annotation training on the downsampled 3D point cloud data to obtain the 3D bounding box, category and distance of each target to be detected; perform annotation training on the 2D image to obtain the 3D bounding box, category and position of each target to be detected; match the 3D point cloud data with the target to be detected in the 2D image, and fuse the position of the target to be detected after matching with the position of the target vehicle to obtain a target positioning map within the perception range of the target vehicle; The target positioning map and inspection target position are used as inputs of the search algorithm to obtain the path planning of the target vehicle at the current moment and complete the autonomous inspection of the target vehicle.

2. The highly automated driving inspection method based on multi-sensor fusion deployment as claimed in claim 1, characterized in that: The three-dimensional point cloud data around the target vehicle is collected through the laser radar, the two-dimensional image is collected through the visual sensor, and the position of the target vehicle is obtained through the Beidou positioning signal.

3. The highly automated driving inspection method based on multi-sensor fusion deployment as claimed in claim 2, characterized in that: The method of constructing a three-dimensional space according to the coordinates of the points in the three-dimensional point cloud data and obtaining the distance between each point and the target vehicle is: Obtain the coordinates of all points in the three-dimensional point cloud data, take the coordinate system of the laser radar as the reference, obtain the coordinates on the x, y, and z axes of the point cloud data, obtain the maximum and minimum values ​​on the x, y, and z axes of the three-dimensional point cloud data respectively, and record the space formed by the maximum and minimum values ​​on the three axes as the three-dimensional space; The coordinate of each point on the z-axis is the distance between each point and the target vehicle.

4. The highly automated driving inspection method based on multi-sensor fusion deployment as claimed in claim 1, characterized in that: The method for obtaining the distance coefficient of each voxel according to the distance between all points in each voxel and the target vehicle is: For each voxel, the mean of the distances between all points and the target vehicle is calculated and recorded as the first mean, and the minimum distance is selected from the distances between all points and the target vehicle and recorded as the minimum distance; The distance coefficient of the voxel is positively correlated with the first mean and the minimum distance, respectively.

5. The highly automated driving inspection method based on multi-sensor fusion deployment as claimed in claim 1, characterized in that: The method for obtaining the downsampling degree of a voxel according to the distance coefficient of each voxel, the local density of all points in the voxel, and the number of points in the voxel is: The mean of the local density of all points within the voxel is calculated as the average local density of the voxel; The degree of downsampling of each voxel is negatively correlated with the average local density of the voxel and the number of points within the voxel, and positively correlated with the distance coefficient of the voxel.

6. The highly automated driving inspection method based on multi-sensor fusion deployment as claimed in claim 1, characterized in that: The method of downsampling voxels according to the downsampling degree of the voxels to obtain the downsampled three-dimensional point cloud data is: If M i ∈[0,0.25), then m=3c; if M i ∈[0.25,0.5), then m=2c; if M i ∈[0.5,0.75), then m=1c; if M i ∈[0.75,1], then m=0; where w i ′ represents the side length of the small voxel on the x-axis after the i-th voxel is split into small voxels, h i ′ represents the side length of the small voxel on the y-axis after the i-th voxel is split into small voxels, l i ′ represents the side length of the small voxel on the z-axis after the i-th voxel is split into small voxels, c is a preset positive integer, M i represents the downsampling degree of the i-th voxel, w represents the side length of the voxel on the x-axis, h represents the side length of the voxel on the y-axis, and l represents the side length of the voxel on the z-axis.

7. The highly automated driving inspection method based on multi-sensor fusion deployment as claimed in claim 1, characterized in that: The method for labeling training of 3D point cloud data is: Pedestrians, vehicles, traffic lights and lane lines are classified and labeled. For the same category, each target is labeled with a labeling box. The labeled data is used as the neural network input, and the cross entropy loss function is used as the classification loss function. The CIOU loss function is used as the bounding box regression loss to obtain the 3D point cloud target detection model. The 3D point cloud target detection model is used to detect each target to be detected to obtain the 3D bounding box, category and distance of the target to be detected.

8. The highly automated driving inspection method based on multi-sensor fusion deployment as claimed in claim 1, characterized in that: The method for labeling training for two-dimensional images is: Pedestrians, vehicles, traffic lights and lane lines are classified and labeled, and each target of the same category is labeled with a labeling box; the camera's intrinsic parameter matrix is ​​obtained through the camera calibration method; the labeled two-dimensional image and intrinsic parameter matrix are used as neural network input, the cross entropy loss function is used as the classification loss function, and the angle regression loss function and size regression loss function are used as the bounding box regression loss to obtain the 3D bounding box, category and position of the target to be detected.

9. The highly automated driving inspection method based on multi-sensor fusion deployment as claimed in claim 1, characterized in that: The method of fusing the position of the target to be detected after matching with the position of the target vehicle to obtain a target positioning map within the perception range of the target vehicle is: Obtain the 3D point cloud data, 2D image and the position of the target vehicle collected at each moment; calculate the intersection-and-union ratio between the 3D bounding box of the 3D point cloud image and the 3D bounding box of the 2D image at the same moment; if the value of the intersection-and-union ratio is greater than the preset value, the match is successful; For the successfully matched 3D bounding box, the position of the corresponding target to be measured and the distance to the target vehicle can be obtained. The position of the target vehicle can be obtained through satellite positioning, and the target to be measured is reflected in the satellite positioning map to obtain the target positioning map within the perception range of the target vehicle.

10. A highly automated driving inspection system based on multi-sensor fusion deployment, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the steps of the highly automated driving inspection method based on multi-sensor fusion deployment as described in any one of claims 1 to 9 are implemented.

Images