A human-computer consistent human-computer co-driving environment perception method

Abstract

The application discloses a kind of based on man-machine consistent man-machine co-pilot environment perception method, including to the original point cloud data obtained based on laser radar is carried out ground segmentation and clustering processing, obtain the target detection result of laser radar;Image data is detected using deep learning algorithm, and the target detection result of camera is obtained;Joint calibration is carried out using camera and laser radar, and the target detection result obtained is obtained using graph matching algorithm to obtain coarse perception fusion result;The attention direction of safety officer is detected using eyeball tracking algorithm, and the original point cloud data in the region of the attention direction is finely detected to obtain secondary perception result;The obtained result is fused, and finally the environment perception result based on man-machine consistency is obtained.The application introduces safety officer attention direction detection mechanism, strengthens the perception accuracy, obtains environment perception result, to improve the short-term, close-range road condition perception accuracy, ensure driving safety.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of unmanned vehicles, in particular to an environment perception method based on human-machine consistency for human-machine co-driving. BACKGROUND

[0002] As a key module in the unmanned technology, the environment perception system is a direct means for detecting external traffic participants. Higher-precision perception results have been a research hotspot at home and abroad. Target detection, as one of the main research directions, has always been in the spotlight. With the continuous development of deep learning technology, target detection algorithms based on laser radar, cameras and multi-sensor fusion have been significantly improved, but these methods are difficult to achieve 100% detection accuracy and are difficult to land. Most of the current unmanned vehicles are equipped with safety officers to ensure that they can be switched to manual processing in case of an accident. Therefore, before the fully autonomous driving technology matures, the environment perception technology based on human-machine consistency will become the mainstream technology.

[0003] The existing technology has the problem that under the premise that the current unmanned technology is equipped with safety officers, most of the current environment perception algorithms only rely on sensors (radar, cameras and multi-sensor fusion) for target detection, without introducing safety officers into the perception process of unmanned vehicles. SUMMARY

[0004] The purpose of the present application is to overcome the shortcomings of the prior art. In order to achieve the above purpose, an environment perception method based on human-machine consistency for human-machine co-driving is used to solve the problems raised in the background technology.

[0005] An environment perception method based on human-machine consistency for human-machine co-driving, comprising:

[0006] S1, performing ground segmentation and clustering processing on the original point cloud data obtained based on the laser radar to obtain the target detection result of the laser radar;

[0007] S2, performing target detection on the image data obtained based on the camera using a deep learning algorithm to obtain the target detection result of the camera;

[0008] S3, performing joint calibration using the camera and the laser radar, and obtaining the target detection result, and using a graph matching algorithm to obtain a coarse perception fusion result;

[0009] S4, detecting the line of sight direction of the safety officer using an eye tracking algorithm, and performing fine detection on the original point cloud data in the line of sight direction region to obtain a secondary perception result;

[0010] S5, performing target fusion on the coarse perception fusion result and the secondary perception result to obtain an environment perception result based on human-machine consistency.

[0011] As a further scheme of the present application: the specific steps in S1 include:

[0012] S11, according to the line beam distribution relationship of the laser radar, and setting each line of the line beam in the middle of the grid, a ring-shaped grid graph is established;

[0013] The lowest point of each grid of the search grid graph is searched to establish a ground base point, and the ground base point of each grid is corrected according to the four-neighbor relationship between the grids;

[0014] The height difference of each point in each grid to the grid is calculated, all points with a height difference less than a height threshold are marked as ground points, and after all ground points are filtered out from the original point cloud, obstacle points are obtained;

[0015] S12, the obtained obstacle points are clustered, first, clustering operation is performed according to a distance threshold between points on a line, and then final clustering operation is performed according to a distance threshold between classes, to obtain a clustering result.

[0016] As a further scheme of the present application: the specific method in S2 is: based on the Faster R-CNN target detection algorithm, using the KITTI data set, training the traffic participation categories of vehicles, people and cyclists, to obtain the detection frame of the target and the category information of the target.

[0017] As a further scheme of the present application: the specific steps in S3 include:

[0018] S31, in the joint calibration process of the camera and the laser radar, the internal parameter calibration of the camera adopts Zhang Zhengyou calibration method, different pose images of the checkerboard collected at different positions are used to identify the corner points of the checkerboard, and multiple constraint equations are established, and the internal parameter matrix of the camera is fitted by using least squares; and

[0019] S32, the external parameter calibration of the camera uses a calibration tool with a specific reflectivity, accurately selects the center position of the calibration board in the point cloud data, and selects the corresponding center position in the camera image, to complete the selection of the control point pair, and the external parameter calibration parameters of the camera are solved by using the N-point perspective pose algorithm;

[0020] S33, a target-level back-end fusion strategy of the camera and the laser radar based on graph matching, first, the camera detection frame is projected to form a camera graph node connected with all targets , and a laser radar graph node formed by the laser radar clustering target Two subgraphs are formed;

[0021] Each camera graph node is traversed using a tree-based search Local node connection information is obtained, and then a camera graph node with the global node index list of the laser radar graph node , and finally judging whether the two targets match according to the constraint condition to complete the matching of the camera detection target and the laser radar clustering result.

[0022] As a further scheme of the present application: the specific steps of the S4 include:

[0023] S41, in the positioning detection of the safety officer's eyes, first, the safety officer's face position in the vehicle is imaged, and the eye position detection model trained in the BioID dataset by using the small target detection network RetinaNet is used for detection to extract the image of the safety officer's eyes;

[0024] S42, in the process of determining the iris center position of the safety officer's eyes, first, pre-processing is required, including binarization processing of the eye image obtained in S41, and open operation and close operation are performed to remove noise;

[0025] The filtered binarized eye image is projected in the horizontal and vertical directions respectively, and the maximum value after the horizontal and vertical projection is taken as the coordinate of the iris center position of the eyes;

[0026] S43, the obtained left and right eye iris positions are used to determine the line of sight direction according to the calibrated line of sight estimation model;

[0027] S44, the line of sight direction obtained by S43 in the original laser radar point cloud data is subjected to secondary perception, and all point clouds in the line of sight direction region are subjected to clustering processing again, in which the distance threshold between points on the line and the class distance threshold between lines are reduced to obtain more accurate clustering results, and the attitude estimation of each clustering target is performed.

[0028] As a further scheme of the present application: the specific steps of the S5 include:

[0029] S51, the area of the boundary box of each target attitude estimation of the twice perception result is calculated;

[0030] S52, for any coarse perception fusion result, the distance information from the attitude estimation center position to each secondary perception result attitude estimation center position is calculated;

[0031] S53, when any coarse perception fusion result and the currently searched secondary perception result distance is less than the distance threshold, and the ratio of the respective boundary box areas is less than the area threshold, the twice search results are merged;

[0032] S54, after traversing search on all coarse perception fusion results, the secondary perception result not being merged is added to the clustering list of the coarse perception fusion result as a new target, and the human-machine consistent human-machine co-driving environment perception algorithm is completed.

[0033] Compared with the prior art, the present application has the following technical effects:

[0034] By using the above technical scheme, the human-machine consistent human-machine co-driving environment perception is used to detect the dynamic and static traffic elements around the unmanned vehicle by using the multi-source multi-modal information fusion method to construct the human-machine consistent environment perception model. The most important feature of the method is to introduce the line of sight direction of the safety officer, and to perform secondary perception on the point cloud data of the line of sight direction of the safety officer, thereby improving the perception accuracy of short-term and close-range road conditions and ensuring driving safety. BRIEF DESCRIPTION OF DRAWINGS

[0035] The specific embodiments of the present application will be described in detail below with reference to the accompanying drawings:

[0036] Figure 1 The steps of the human-machine consistent human-machine co-driving environment perception method of the disclosed embodiments of the present application are shown in the figure;

[0037] Figure 2 The system structure diagram of the human-machine consistent human-machine co-driving environment perception method of the disclosed embodiments of the present application is shown in the figure;

[0038] Figure 3 The point cloud clustering algorithm based on the wire harness relationship of the disclosed embodiments of the present application is shown in the figure;

[0039] Figure 4 The camera and laser radar external parameter calibration tool of the disclosed embodiments of the present application is shown in the figure;

[0040] Figure 5 The camera target detection graph based on the graph of the disclosed embodiments of the present application is shown in the figure;

[0041] Figure 6 The camera and laser radar back-end fusion matching based on the graph of the disclosed embodiments of the present application is shown in the figure;

[0042] Figure 7 The laser radar clustering graph based on the graph of the disclosed embodiments of the present application is shown in the figure. DETAILED DESCRIPTION

[0043] With reference to the drawings of the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments of the present application, all the other embodiments obtained by a person of ordinary skill in the art without creative work are within the protection scope of the present application.

[0044] Please refer to Figure 1 and Figure 2 In the embodiments of the present application, a human-computer consistent human-computer co-driving environment perception method comprises:

[0045] S1, ground segmentation and clustering processing are performed on original point cloud data obtained based on a laser radar to obtain a target detection result of the laser radar, and the specific steps comprise:

[0046] S11, a ring-shaped grid graph is established according to the line bundle distribution relationship of the laser radar, and each line of the line bundle is arranged in the middle of the grid;

[0047] For any grid , wherein and represent the horizontal and vertical indexes of the grid, and the lowest point in the grid is searched as is a ground base point, and is corrected according to the four-neighbor relationship between the grids, and the four-neighbor ground base point estimation value of the lowest point of the grid can be expressed as:

[0048] ;

[0049] When the difference between and is less than a set four-neighbor height difference threshold , the ground base point is replaced by the value of , and the correction of the ground base point of the grid is completed;

[0050] It is set that there are points in the grid , and for any point , the height of the point to the corrected ground base point of the grid is calculated:

[0051] ;

[0052] When is less than a grid height threshold , the point is marked as a ground point, otherwise, it is marked as an obstacle point;

[0053] When the traversal operation is performed on all the grids, all the ground points can be filtered out using the obtained labels, and all the obstacle points are obtained;

[0054] S12, the obstacle points obtained in step S11 are subjected to clustering processing, first, clustering operation is performed according to the distance threshold between points on the line, and then final clustering operation is performed according to the distance threshold between lines, and the clustering result is obtained;

[0055] In the clustering processing, first, clustering operation is performed on each line, as shown in Figure 3 for the first line, by judging the Euclidean distance between the first class and the first class is greater than the set distance threshold between points on the line, so the first class and the first class are divided into two classes, otherwise, they are merged into one class;

[0056] Secondly, the inter-line judgment is performed, as shown in Figure 3 for the first line, the clustering results of the upper and lower two lines, i.e. the first line and the first line, are needed to be judged, since the Euclidean distance between the first class and the first class is less than the inter-line class distance threshold, finally, the two classes are merged into one class, otherwise, since the Euclidean distance between the first class and the first class is greater than the inter-line class distance threshold, it is determined that there are two classes.

[0057] S2, based on the image data obtained by the camera, a deep learning algorithm is used for target detection to obtain the target detection result of the camera, and the specific steps include;

[0058] S21, data enhancement: the puzzleMix data enhancement algorithm is used to process the KITTI dataset, first, the salient region of each input sample is calculated, then only the salient region in the input sample is cropped, and a new sample is spliced to enrich the data amount of the sample;

[0059] S22, target proposal preprocessing is performed on the image input into the network: first, the lateral inhibition mechanism existing in the perception of human eyes is simulated, and the proposal box with different "excitability" scores is learned and generated by combining the enhanced frequency feature with the linear classifier;

[0060] A closed-loop decomposition strategy with adaptive key information compensation is constructed to make the decomposition basis vectors as few as possible and not lose key information to speed up the proposal box generation rate. Then the proposal boxes are projected into a non-zero matrix in descending order of score, and the values of the projection position and the neighborhood coordinates are set to 0 to suppress the area around the proposal box;

[0061] When the subsequent proposals are projected to the suppression region, it is judged whether the scores of the two proposals are similar, if the score difference is large, the proposal frame with lower score is deleted, and finally the eight-neighbor boundary merging algorithm is used to merge the proposal frames;

[0062] S23, input the proposal frame obtained in S22 into the Faster R-CNN detector, replace the RPN network in the Faster R-CNN network, wherein the detector uses the VGGNET-16 network pre-trained on the VOC2007 dataset to detect the target bounding box and category information of the collected image.

[0063] S3, joint calibration of the camera and the laser radar, and obtaining the target detection result, using a graph matching algorithm to obtain a coarse perception fusion result, the specific steps comprising:

[0064] S31, in the joint calibration process of the camera and the laser radar, the internal parameter calibration of the camera uses Zhang Zhengyou calibration method, different position collected images of the checkerboard in different poses are used to identify the corner points of the checkerboard, and multiple constraint equations are established, and the internal parameter matrix of the camera is fitted by using least squares;

[0065] S32, the external parameter calibration of the camera uses a calibration tool with specific reflectivity, as shown in Figure 4 The calibration tool has the characteristics that the center reflection of the calibration tool has specific reflectivity, and the center position of the calibration board can be accurately selected in the point cloud data, and due to the obvious color difference between red and white, the corresponding center position can be selected in the camera image to complete the selection of the control point pair, and the external parameter calibration parameters of the camera are obtained by using the N-point perspective pose algorithm;

[0066] S33, the target-level back-end fusion strategy of the camera and the laser radar based on graph matching, specifically as follows:

[0067] Firstly, the camera target detection result is stretched to the bird's eye view by using the trapezoidal correction algorithm, and the target detection result is saved in the queue from bottom to top in the bird's eye view, wherein the arrangement from bottom to top can approximately obtain the order of the target from near to far;

[0068] According to the overlapping relationship of each two target detection results, it is judged whether the two targets constitute a connection relationship, and according to the above arrangement order, the camera target detection frame is connected to form a camera graph As shown in Figure 5 , the camera graph contains and ;

[0069] Secondly, the camera-laser radar joint calibration parameters are used to project the laser radar clustering results to the camera plane, as shown in Figure 6 , and the laser radar clustering results in each camera target detection frame are taken out, and the camera graph is established according to the one-to-one correspondence relationship, and the laser radar clustering target graph is constructed.

[0070] At this time there are two laser radar clustering results in the camera target detection frame and , therefore, all the laser radar clustering targets and in the camera target detection frame are constructed into a local full-connection node, as shown in Figure 7 , according to the set constraint condition, it is judged which one of and is the actual corresponding matching result of the camera target detection frame, and the specific constraint condition is as follows:

[0071] Degree constraint: according to the node degree characteristics of the subgraph isomorphism problem, when matching, the degree of two nodes in the laser radar graph should be greater than or equal to the degree between the corresponding two nodes in the camera graph .

[0072] Neighborhood constraint: when matching, the connected laser radar clustering result node number in the laser radar graph should meet the corresponding laser radar clustering result number in the matched camera graph .

[0073] Completeness constraint: the projection area of the clustering result in the laser radar graph on the camera plane should be similar to the size of the target detection frame area corresponding to the corresponding camera graph .

[0074] According to the above three constraint conditions, it can be judged that satisfies all three constraint conditions, and does not satisfy the completeness constraint, so the final determination is that is the matching object.

[0075] S4, the gaze direction of the safety officer is detected by using the eye tracking algorithm, and the original point cloud data in the gaze direction area is finely detected to obtain secondary perception results, and the specific steps include:

[0076] ​​S41, in the positioning detection of the safety officer's eyes, first, the safety officer's face position is imaged in the vehicle, and the eye position detection model trained in the BioID dataset by using the small target detection network RetinaNet is used for detection to extract the image of the safety officer's eyes;

[0077] S42, in the process of determining the iris center position of the safety officer's eyes, first, pre-processing is needed, including binarization processing of the eye image obtained in S41, the binarization processing adopts the iterative optimal binarization algorithm, and the specific steps are as follows:

[0078] Initialization of binarization threshold : the eye image obtained in S41 is converted into a gray image, and each pixel value in the eye gray image is traversed to seek the maximum gray value of the image and the minimum gray value , then the binarization threshold is initialized , which can be expressed as:

[0079] ;

[0080] Iterative update of binarization threshold: according to the current binarization threshold , each pixel value of the eye image obtained in S41 is traversed, the pixel value greater than is marked as a foreground point, and the pixel value less than is marked as a background point.

[0081] According to the foreground points and the background points obtained at present, the average gray values of the foreground points and the background points are calculated respectively and , then the th binarization threshold can be updated as:

[0082] ;

[0083] Iterative solution , until , then the iteration is stopped, at this time is the optimal binarization threshold.

[0084] Binarization of the eye image: according to the obtained , each pixel value in the eye gray image is traversed, the pixel value greater than is marked as 255, otherwise, it is marked as 0, the binarization of the eye image is completed, and opening operation and closing operation are performed to remove noise.

[0085] In order to determine the left and right eye iris center positions, the filtered binarized eye image is projected in the horizontal and vertical directions respectively, and the maximum value after horizontal and vertical projection is the coordinate of the iris center position.

[0086] S43, using the obtained left and right eye iris positions, determining a gaze direction according to a calibrated gaze estimation model;

[0087] S44, performing secondary perception on the gaze direction obtained by S43 in the original laser radar point cloud data, re-performing clustering processing on all point clouds in the gaze direction area, reducing the distance threshold between points on the line and the distance threshold between lines in the clustering processing, obtaining more accurate clustering results, and performing pose estimation on each clustering target.

[0088] S5, performing target fusion on the obtained coarse perception fusion result and secondary perception result to finally obtain an environment perception result based on human-machine consistency, and the specific steps include:

[0089] S51, calculating the bounding box area of each target pose estimation of the twice perception results;

[0090] S52, for any coarse perception fusion result, calculating the distance information from the center position of the pose estimation to the center position of the pose estimation of each secondary perception result;

[0091] S53, when any coarse perception fusion result is less than the distance threshold from the current searched secondary perception result, and the ratio of the respective bounding box areas is less than the area threshold, merging the twice search results;

[0092] S54, after searching all coarse perception fusion results, the secondary perception results that are not merged are added to the clustering list of the coarse perception fusion results as new targets, and the human-machine co-driving environment perception algorithm based on human-machine consistency is completed.

[0093] Although the embodiments of the present application have been shown and described, it can be understood by those of ordinary skill in the art that various changes, modifications, replacements and variations can be made to the embodiments without departing from the principles and spirits of the present application, the scope of the present application is defined by the appended claims and their equivalents, and all should be included in the protection scope of the present application.

Claims

1. A human-machine co-driving environment perception method based on human-computer consistency, characterized in that, The method comprises the following steps: S1, ground segmentation and clustering processing are performed on raw point cloud data obtained based on a laser radar to obtain a target detection result of the laser radar; S2, a deep learning algorithm is used to perform target detection on image data obtained based on a camera to obtain a target detection result of the camera; S3, joint calibration is performed on the camera and the laser radar, and a graph matching algorithm is used to obtain a coarse perception fusion result from the target detection results; S4, a gaze direction of a safety officer is detected by using an eye tracking algorithm, and secondary perception is performed on raw point cloud data in the gaze direction region to obtain a secondary perception result, and the specific steps comprise: S41, in the positioning detection of the safety officer's eyes, first, an image of the safety officer's face position is collected in the vehicle, and a human eye position detection model trained in a BioID data set by using a small target detection network RetinaNet is used for detection to extract an image of the safety officer's eyes; S42, in the determination of the iris center positions of the safety officer's eyes, first, preprocessing is performed, including binarization processing of the human eye image obtained in S41, and open operation and close operation are performed on the binarized human eye image to remove noise; the binarized human eye image is projected in the horizontal and vertical directions respectively, and the maximum values after the projection in the horizontal and vertical directions are taken as the coordinates of the iris center positions of the eyes; S43, the gaze direction is determined according to a calibrated gaze estimation model by using the obtained iris positions of the left and right eyes; S44, secondary perception is performed on the gaze direction obtained in S43 in the raw laser radar point cloud data, and all point clouds in the gaze direction region are re-clustered, in the clustering process, the distance threshold between points on a line and the class distance threshold between lines are reduced to obtain more accurate clustering results, and the attitude of each cluster target is estimated; S5, target fusion is performed on the coarse perception fusion result and the secondary perception result to finally obtain an environment perception result based on human-machine consistency. 2.The human-machine consistent environment perception method based on human-machine co-driving according to claim 1, wherein, The specific steps in S1 comprise: S11, a ring-shaped grid graph is established according to the line bundle distribution relationship of the laser radar, and each line of the line bundle is arranged in the middle of the grid; ground base points are established by searching for the lowest points of each grid of the grid graph, and the ground base points of each grid are corrected according to the four-neighbor relationship between the grids; the height difference of each point in each grid to the grid is calculated, and all points with a height difference less than a height threshold are marked as ground points, and all ground points are filtered from the raw point cloud to obtain obstacle points; S12, clustering processing is performed on the obtained obstacle points, first, clustering operation is performed according to a distance threshold between points on a line, and then final clustering operation is performed according to a class distance threshold between lines to obtain a clustering result. 3.The human-machine consistent environment perception method based on human-machine co-driving according to claim 1, wherein, The specific method in S2 is that a Faster R-CNN target detection algorithm is used to train vehicle, human and cyclist traffic participant categories based on a KITTI data set to obtain a detection frame of the target and class information of the target. 4.The human-machine consistent environment perception method based on human-machine co-driving according to claim 1, wherein, The specific steps in S3 comprise: In the joint calibration process of the camera and the laser radar, the internal parameter calibration of the camera adopts Zhang Zhengyou calibration method, different position collected images of the checkerboard in different postures are used to identify the corner points of the checkerboard, and a plurality of constraint equations are established, and the internal parameter matrix of the camera is fitted by using the least square method. In the external parameter calibration of the camera, a calibration tool with a specific reflectivity is used, the center position of the calibration board in the point cloud data is accurately selected, and the corresponding center position in the camera image is selected, so as to complete the selection of the control point pair, and the N-point perspective pose algorithm is used to solve the external parameter calibration parameters of the camera. S33, a camera and lidar target-level backend fusion strategy based on graph matching, first project the camera detection boxes to form a camera graph node connected with all targets , and a lidar graph node formed by the lidar clustered targets , to form two subgraphs; Tree-based search is used to traverse the graph nodes of each camera Local node connection information is obtained, and then the camera graph nodes are established Global node index list of the lidar graph nodes Finally, it is determined whether the two targets match according to the constraint conditions, and the matching of the camera detected targets and the lidar clustering results is completed.

5. The human-robot co-driving environment perception method based on human consistency according to claim 1, wherein, The specific steps of S5 include: S51, calculating the area of the bounding box of each target pose estimation of the twice perception result; S52, for any coarse perception fusion result, calculating the distance information from the center position of the pose estimation to the center position of the pose estimation of each secondary perception result; S53, when any coarse perception fusion result and the currently searched secondary perception result are less than the distance threshold, and the ratio of the area of the respective bounding box is less than the area threshold, the twice search results are merged; S54, after searching all the coarse perception fusion results, the secondary perception result not merged is added to the clustering list of the coarse perception fusion result as a new target, and the human-machine consistent human-machine co-driving environment perception algorithm is completed.

Images