Expansion point cloud recognition method, device, computer equipment and storage medium

Abstract

The present application relates to an expansion point cloud recognition method, device, and computer equipment. The method includes: obtaining environmental point cloud data and environmental image data corresponding to the current field of view, performing first target object detection based on the environmental image data, and obtaining the first target object area corresponding to the environmental image data; performing obstacle clustering based on the environmental point cloud data to obtain each obstacle point cloud cluster, filtering the environmental point cloud data based on each obstacle point cloud cluster to obtain filtered point cloud data; determining the second target object point cloud data from the filtered point cloud data based on the reflection intensity information corresponding to the filtered point cloud data; performing projection based on the second target object point cloud data to obtain the second target object projection area, calculating the matching degree between the first target object area and the second target object projection area; and determining whether there is expansion point cloud data in the environmental point cloud data based on the matching degree. The use of this method can improve the accuracy of expansion point cloud recognition.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and in particular to an expansion point cloud recognition method, apparatus, computer equipment, storage medium and computer program product. Background Art

[0002] With the development of autonomous driving technology, it is necessary to detect obstacles in the environment around autonomous driving vehicles. Usually, laser radar is used to scan the environment around the vehicle to obtain a three-dimensional point cloud of the environment, and then detect obstacles based on the three-dimensional point cloud. However, when the vehicle encounters an object with high reflectivity during driving, such as a traffic sign, the laser radar hardware in the vehicle will cause the point cloud to expand in all directions when processing the high-reflection signal, resulting in an expansion point cloud that appears out of thin air, causing misjudgment of obstacle identification, and judging the originally passable area as impassable, causing the autonomous driving vehicle to brake suddenly or slow down and stop for no reason. The existing method of removing the expansion point cloud is to improve the hardware design and signal processing algorithm of the laser radar manufacturer. However, the existing method has a high design cost, and the laser radar is affected by the environment and cannot accurately identify the expansion point cloud, resulting in low efficiency in identifying the expansion point cloud. Summary of the invention

[0003] Based on this, it is necessary to provide an expansion point cloud recognition method, device, computer equipment, computer-readable storage medium and computer program product that can improve the accuracy of obstacle detection in response to the above technical problems.

[0004] In a first aspect, the present application provides a method for identifying an expansion point cloud. The method comprises:

[0005] Acquire environmental point cloud data and environmental image data corresponding to the current field of view, perform first target object detection based on the environmental image data, and obtain a first target object area corresponding to the environmental image data;

[0006] Perform obstacle clustering based on the environmental point cloud data to obtain various obstacle point cloud clusters, and filter the environmental point cloud data based on the various obstacle point cloud clusters to obtain filtered point cloud data;

[0007] determining second target object point cloud data from the filtered point cloud data based on reflection intensity information corresponding to the filtered point cloud data;

[0008] Performing projection based on the point cloud data of the second target object to obtain a projection area of ​​the second target object, and calculating a matching degree between the first target object area and the projection area of ​​the second target object;

[0009] It is determined whether there is expansion point cloud data in the environment point cloud data based on the matching degree.

[0010] In a second aspect, the present application also provides an expansion point cloud recognition device. The device comprises:

[0011] An image detection module is used to obtain environmental point cloud data and environmental image data corresponding to the current field of view, perform first target object detection based on the environmental image data, and obtain a first target object area corresponding to the environmental image data;

[0012] A filtering module is used to cluster obstacles based on the environmental point cloud data to obtain each obstacle point cloud cluster, and filter the environmental point cloud data based on each obstacle point cloud cluster to obtain filtered point cloud data;

[0013] a point cloud determination module, configured to determine point cloud data of a second target object from the filtered point cloud data based on reflection intensity information corresponding to the filtered point cloud data;

[0014] A matching module, used for performing projection based on the point cloud data of the second target object to obtain a projection area of ​​the second target object, and calculating a matching degree between the first target object area and the projection area of ​​the second target object;

[0015] The determination module is used to determine whether there is expansion point cloud data in the environment point cloud data based on the matching degree.

[0016] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0017] Acquire environmental point cloud data and environmental image data corresponding to the current field of view, perform first target object detection based on the environmental image data, and obtain a first target object area corresponding to the environmental image data;

[0018] Perform obstacle clustering based on the environmental point cloud data to obtain various obstacle point cloud clusters, and filter the environmental point cloud data based on the various obstacle point cloud clusters to obtain filtered point cloud data;

[0019] determining second target object point cloud data from the filtered point cloud data based on reflection intensity information corresponding to the filtered point cloud data;

[0020] Performing projection based on the point cloud data of the second target object to obtain a projection area of ​​the second target object, and calculating a matching degree between the first target object area and the projection area of ​​the second target object;

[0021] It is determined whether there is expansion point cloud data in the environment point cloud data based on the matching degree.

[0022] In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0023] Acquire environmental point cloud data and environmental image data corresponding to the current field of view, perform first target object detection based on the environmental image data, and obtain a first target object area corresponding to the environmental image data;

[0024] Perform obstacle clustering based on the environmental point cloud data to obtain various obstacle point cloud clusters, and filter the environmental point cloud data based on the various obstacle point cloud clusters to obtain filtered point cloud data;

[0025] determining second target object point cloud data from the filtered point cloud data based on reflection intensity information corresponding to the filtered point cloud data;

[0026] Performing projection based on the point cloud data of the second target object to obtain a projection area of ​​the second target object, and calculating a matching degree between the first target object area and the projection area of ​​the second target object;

[0027] It is determined whether there is expansion point cloud data in the environment point cloud data based on the matching degree.

[0028] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0029] Acquire environmental point cloud data and environmental image data corresponding to the current field of view, perform first target object detection based on the environmental image data, and obtain a first target object area corresponding to the environmental image data;

[0030] Perform obstacle clustering based on the environmental point cloud data to obtain various obstacle point cloud clusters, and filter the environmental point cloud data based on the various obstacle point cloud clusters to obtain filtered point cloud data;

[0031] determining second target object point cloud data from the filtered point cloud data based on reflection intensity information corresponding to the filtered point cloud data;

[0032] Performing projection based on the point cloud data of the second target object to obtain a projection area of ​​the second target object, and calculating a matching degree between the first target object area and the projection area of ​​the second target object;

[0033] It is determined whether there is expansion point cloud data in the environment point cloud data based on the matching degree.

[0034] The above-mentioned expansion point cloud recognition method, device, computer equipment, storage medium and computer program product can obtain image data corresponding to the first target object area by using environmental image data to detect the first target object; then use the environmental point cloud data to cluster and filter obstacles to obtain filtered point cloud data, and determine the second target object point cloud data in the filtered point cloud data according to the reflection intensity information. The projection area of ​​the second target object is obtained by projecting the second target object point cloud data. By calculating the matching degree between the projection area of ​​the second target object and the area of ​​the first target object, it is possible to quickly determine whether there is expansion point cloud data in the environmental point cloud. Furthermore, it is possible to avoid the situation where the target object is misjudged due to the existence of expansion point cloud data, thereby improving the accuracy of the expansion point cloud data recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A diagram showing an application environment of an expansion point cloud recognition method in an embodiment;

[0036] Figure 2 A schematic diagram of a process of an expansion point cloud recognition method in one embodiment;

[0037] Figure 3 A schematic diagram of a process for removing expanded point cloud data in one embodiment;

[0038] Figure 4 is a structural block diagram of an expansion point cloud recognition device in one embodiment;

[0039] Figure 5 is an internal structure diagram of a computer device in one embodiment;

[0040] Figure 6 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0042] The expansion point cloud recognition method provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The terminal of the moving carrier obtains the environmental point cloud data and environmental image data corresponding to the current field of view, performs first target object detection based on the environmental image data, and obtains the first target object area corresponding to the environmental image data; the moving carrier terminal performs obstacle clustering based on the environmental point cloud data to obtain each obstacle point cloud cluster, and filters the environmental point cloud data based on each obstacle point cloud cluster to obtain filtered point cloud data; the moving carrier terminal determines the second target object point cloud data from the filtered point cloud data based on the reflection intensity information corresponding to the filtered point cloud data; the moving carrier terminal performs projection based on the second target object point cloud data to obtain the second target object projection area, and calculates the matching degree between the first target object area and the second target object projection area; the moving carrier terminal determines whether there is expansion point cloud data in the environmental point cloud data based on the matching degree. The terminal 102 may be an autonomous driving terminal. In one embodiment, the terminal 102 may be an unmanned vehicle or a manned vehicle with autonomous driving capability. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers.

[0043] In one embodiment, Figure 2 As shown, a method for recognizing an expanded point cloud is provided, and the method is applied to Figure 1 The method is described by taking the mobile carrier terminal in the example as an example. It can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0044] Step 202: Acquire environmental point cloud data and environmental image data corresponding to the current field of view, perform first target object detection based on the environmental image data, and obtain a first target object region corresponding to the environmental image data.

[0045] Among them, the current field of view refers to the maximum range of the surrounding environment of the moving carrier collected by the acquisition device at the current moment. Environmental point cloud data refers to the original point cloud data of the surrounding environment of the moving carrier collected, and the surrounding environment may include obstacles. Environmental image data refers to the original image of the surrounding environment of the moving carrier collected. The moving carrier refers to a movable object, which can be an unmanned moving carrier or a manned moving carrier. The first target object detection refers to the detection of the target object using the environmental image data, and the target object refers to a pre-set specific detection object. The first target object area refers to the area corresponding to the target object in the image data.

[0046] Specifically, the moving carrier is equipped with a point cloud data acquisition device and an image acquisition device. The moving carrier terminal can obtain the original point cloud data of the moving carrier's surrounding environment in the current field of view through the point cloud data acquisition device, and obtain the environmental image data of the moving carrier's surrounding environment in the same field of view as the environmental point cloud data through the image acquisition device. The moving carrier can be a vehicle, an aircraft, a robot, etc. Then the moving carrier terminal performs a first target object detection on the environmental image data to obtain the first target object area in the environmental image data. Then the moving carrier terminal stores the first target object area for subsequent area matching.

[0047] Step 204 , clustering obstacles based on the environmental point cloud data to obtain various obstacle point cloud clusters, and filtering the environmental point cloud data based on the various obstacle point cloud clusters to obtain filtered point cloud data.

[0048] Among them, obstacle clustering refers to the point cloud clusters obtained by aggregating the point cloud data that need to be clustered. Obstacle point cloud clusters are point cloud data that represent obstacles, and different point cloud clusters can represent different obstacles. Filtered point cloud data refers to the point cloud data remaining after filtering the environmental point cloud data.

[0049] Specifically, the moving carrier terminal can use a pre-trained obstacle detection algorithm to perform obstacle detection on the environmental point cloud data, and then use a clustering algorithm to aggregate the point cloud data of each obstacle detected in the environmental point cloud data to obtain each obstacle point cloud cluster. Then, the moving carrier terminal filters each obstacle point cloud cluster in the environmental point cloud data to obtain the remaining filtered point cloud data.

[0050] Step 206 : determining second target object point cloud data from the filtered point cloud data based on the reflection intensity information corresponding to the filtered point cloud data.

[0051] The reflection intensity information refers to information characterizing the reflection intensity of the point cloud, which may be the reflectivity of the point cloud. The second target object point cloud data refers to a point cloud cluster whose reflection intensity meets a preset reflection requirement.

[0052] Specifically, the motion carrier terminal can then use a clustering algorithm to cluster the filtered point cloud data to obtain point cloud clusters corresponding to the filtered point cloud data, and then obtain the reflection intensity information corresponding to each point cloud cluster, and obtain the pre-set reflection requirements. The motion carrier terminal compares the reflection intensity information corresponding to each point cloud cluster with the preset reflection threshold, and uses the point cloud cluster that meets the preset reflection requirements as the point cloud data of the second target object.

[0053] Step 208 , performing projection based on the point cloud data of the second target object to obtain a projection area of ​​the second target object, and calculating a matching degree between the first target object area and the projection area of ​​the second target object.

[0054] Among them, projection refers to the process of projecting the three-dimensional point cloud data of the second target object point cloud data onto a two-dimensional plane to obtain point cloud projection data. In one embodiment, the point cloud data is projected onto the same two-dimensional coordinate system as the image data to obtain the corresponding point cloud projection data. The point cloud projection data refers to the point cloud data projected onto the two-dimensional plane. The point cloud projection data exists in the form of two-dimensional data. The second target object projection area refers to the point cloud data projection area corresponding to the second target object point cloud data on the two-dimensional plane. The matching degree refers to the degree of overlap between the second target object projection area and the first target object area.

[0055] Specifically, the moving carrier terminal converts the second target object point cloud data into point cloud projection data in the same plane coordinate system as the image data according to the three-dimensional coordinates of the second target object point cloud data to obtain the second target object projection area. Then the moving carrier terminal matches the second target object projection area with the first target object area, and calculates the degree of overlap between the second target object projection area and the first target object area.

[0056] Step 208 : determining whether there is expansion point cloud data in the environment point cloud data based on the matching degree.

[0057] The expanded point cloud data refers to point cloud data in which expansion occurs in the environmental point cloud data.

[0058] Specifically, the moving carrier terminal can obtain a pre-set matching degree threshold. When the moving carrier terminal detects that the matching degree meets the matching degree threshold, it determines that there is expanded point cloud data in the environmental point cloud data; when the moving carrier terminal detects that the matching degree does not meet the matching degree threshold, it determines that there is no expanded point cloud data in the environmental point cloud data.

[0059] In the above-mentioned expansion point cloud recognition method, by using the environment image data to detect the first target object, the image data corresponding to the first target object area can be obtained; then the environment point cloud data is used to cluster and filter obstacles to obtain filtered point cloud data, and the second target object point cloud data is determined in the filtered point cloud data according to the reflection intensity information. The second target object point cloud data is projected to obtain the second target object projection area. By calculating the matching degree between the second target object projection area and the first target object area, it is possible to quickly determine whether there is expansion point cloud data in the environment point cloud. Furthermore, it is possible to avoid the situation where the target object is misjudged due to the existence of expansion point cloud data, thereby improving the accuracy of expansion point cloud data recognition.

[0060] In one embodiment, step 210, determining whether there is an expansion point cloud in the environment point cloud data based on the matching degree, includes:

[0061] When the matching degree is greater than a preset matching degree threshold, determining that there is expansion point cloud data in the environment point cloud data; and

[0062] A second target object projection sub-region in the second target object projection region that does not overlap with the first target object region is determined, and point cloud data corresponding to the second target object projection sub-region is expanded point cloud data.

[0063] The preset matching degree threshold refers to a preset matching degree threshold. The target object projection sub-area refers to a sub-area of ​​the second target object projection area that does not overlap with the first target object area, and is a sub-area in the second target object projection area.

[0064] Specifically, the moving carrier terminal obtains a preset matching degree threshold, and when it detects that the matching degree is greater than the preset matching threshold, it determines that the second target object projection area is matched with the first target object area, and determines that there is expanded point cloud data in the environmental point cloud data corresponding to the second target object projection area.

[0065] The moving carrier terminal scans the projection area of ​​the second target object and the corresponding first target object area, determines the overlapping area and the non-overlapping area between the projection area of ​​the second target object and the first target object area, uses the non-overlapping area in the projection area of ​​the second target object as the projection sub-area of ​​the second target object, and determines the point cloud projection data in the projection sub-area of ​​the second target object as the expanded point cloud data. Further, the moving carrier terminal determines the point cloud data corresponding to the projection sub-area of ​​the second target object in the point cloud data of the second target object as the expanded point cloud data, and then the moving carrier terminal can filter and remove the expanded point cloud data in the point cloud data of the second target object.

[0066] The motion carrier terminal can also first determine the overlapping area in the projection area of ​​the second target object, and determine the point cloud data area corresponding to the overlapping area in the point cloud data of the second target object, and then determine the point cloud data outside the point cloud data area corresponding to the overlapping area in the point cloud data of the second target object as expanded point cloud data, and then the motion carrier terminal can filter and clear the expanded point cloud data in the point cloud data of the second target object.

[0067] In this embodiment, when the matching degree is greater than the preset matching degree threshold, it is possible to quickly determine that there is expanded point cloud data in the environmental point cloud data, and the position of the expanded point cloud can be quickly determined through the non-overlapping area between the second target object projection area and the first target object area, thereby improving the recognition efficiency of the expanded point cloud data.

[0068] In one embodiment, the method further comprises:

[0069] Identifying the corresponding original expanded point cloud data in the environment point cloud data based on the point cloud data corresponding to the projection sub-region of the second target object; and

[0070] The environmental point cloud data is filtered based on the original expanded point cloud data.

[0071] The original expanded point cloud data refers to the expanded point cloud data existing in the environment point cloud data.

[0072] Specifically, the moving carrier terminal determines the original expanded point cloud data corresponding to the expanded point cloud data in the environmental point cloud data according to the point cloud projection data corresponding to the projection sub-area of ​​the second target object, that is, the expanded point cloud data, and then determines the original expanded point cloud data corresponding to the expanded point cloud data in the environmental point cloud data according to the projection relationship between the point cloud projection data and the environmental point cloud data, and the projection relationship refers to the association relationship between the point cloud data and the point cloud projection data established during the projection process of the point cloud data. Then, the moving carrier terminal filters and clears the original expanded point cloud data in the environmental point cloud data to obtain the filtered environmental point cloud data, performs obstacle detection on the filtered environmental point cloud data, and determines the movement route and movement speed according to the detection results, so that the moving carrier moves according to the movement route and the calculation speed.

[0073] In this embodiment, the corresponding original expanded point cloud data in the environmental point cloud data can be accurately identified based on the point cloud data corresponding to the projection sub-area of ​​the second target object, and the original expanded point cloud data in the environmental point cloud data is filtered, so that the misidentification of the target object due to the existence of the expanded point cloud data can be avoided, thereby improving the accuracy of the target object recognition.

[0074] In one embodiment, step 206, determining the second target object point cloud data from the filtered point cloud data based on the reflection intensity information corresponding to the filtered point cloud data, includes:

[0075] Clustering the filtered point cloud data to obtain at least two point cloud clusters;

[0076] The point cloud data of the second target object is determined from at least two point cloud clusters based on the reflection intensity information corresponding to each point cloud cluster and the preset reflection intensity threshold. The preset reflection threshold refers to a pre-set threshold of the point cloud reflectivity. The point cloud cluster refers to a point cloud cluster obtained by clustering the point cloud data.

[0077] Specifically, the motion carrier terminal uses a preset deep learning network to detect moving obstacles such as vehicles, pedestrians and cyclists in the environmental point cloud data, and obtains the category of each three-dimensional point in the environmental point cloud data output by the deep learning network. The motion carrier terminal can also detect the ground in the environmental point cloud data through the deep learning network, and the category of each three-dimensional point output also includes the ground category. The deep learning network can be a network such as Birdeye view, Cylinder view, 3D voxel, PointNet, etc. that can perform full-frame point cloud target detection and ground point recognition.

[0078] The computing carrier terminal clusters the point cloud data corresponding to the categories of vehicles, pedestrians, cyclists and ground in the environmental point cloud data according to the categories of each three-dimensional point output by the deep learning network. The motion carrier terminal can cluster the data using the Euclidean clustering algorithm, etc., to obtain the point cloud data corresponding to the clustered vehicles, pedestrians, cyclists and ground. Then, the motion carrier terminal filters and clears the point cloud data corresponding to the clustered vehicles, pedestrians, cyclists and ground in the environmental point cloud data to obtain the filtered point cloud data.

[0079] Then the moving carrier terminal uses a preset clustering algorithm to cluster the filtered point cloud data. The moving carrier terminal can use a European clustering algorithm to aggregate the filtered point cloud data into point cloud clusters to obtain at least two point cloud clusters, such as point cloud clusters corresponding to static objects such as trees, buildings, and traffic signs. Then the moving carrier terminal estimates the three-dimensional bounding box of each point cloud cluster, and can use a bounding box estimation algorithm, such as a principal component analysis fitted rectangle, a search algorithm-based fitted rectangle, etc., to obtain the corresponding three-dimensional bounding box of each point cloud cluster, and the three-dimensional bounding box represents the spatial position area of ​​the point cloud cluster.

[0080] The motion carrier terminal obtains the reflection intensity information corresponding to the three-dimensional point data of each point cloud cluster, and obtains a pre-set reflection threshold, compares the reflection intensity information corresponding to the three-dimensional point data of each point cloud cluster with the reflection requirement, determines the number of three-dimensional points in each point cloud cluster that meets the reflection requirement, and uses the data of the point cloud cluster corresponding to the three-dimensional points that meet the reflection requirement as the point cloud data of the second target object.

[0081] In a specific embodiment, the moving carrier terminal compares the reflectivity corresponding to each 3D point data in each point cloud cluster with a preset reflection threshold, determines the 3D point data in each obstacle point cloud data that reaches the preset reflection threshold, and uses the 3D point data that reaches the preset reflection threshold as the target 3D point data. Then the moving carrier terminal counts the number of target 3D point data in each point cloud cluster, compares the number of points of the target 3D point data in each point cloud cluster with the preset point number threshold, and uses the point cloud cluster whose number of points of the target 3D point data in each point cloud cluster reaches the preset point number threshold as the second target object point cloud data.

[0082] The moving carrier terminal may also estimate a bounding box for the second target object point cloud data after determining the second target object point cloud data, and then store the second target object point cloud data and its corresponding three-dimensional bounding box for subsequent use.

[0083] In this embodiment, by first filtering the ground points and moving obstacle points in the environmental point cloud data and then clustering the remaining point clouds, it is possible to avoid interference between the point cloud data corresponding to the moving obstacles and the ground during the clustering process, thereby improving the accuracy of the obstacle point cloud data and further improving the accuracy of identifying the obstacle area.

[0084] In one embodiment, the first target object area includes at least two, and the second target object point cloud data includes at least two;

[0085] The moving carrier terminal determines at least two second target object point cloud data from each obstacle point cloud data according to the reflection intensity information corresponding to each point cloud cluster, and then performs plane projection on the at least two second target object point cloud data respectively. The moving carrier terminal obtains the pre-stored internal parameters of the camera and the external parameters between the camera and the laser radar. The internal parameters and external parameters can be obtained in advance through the calibration algorithm. The camera is used to collect environmental image data, and the laser radar is used to collect environmental point cloud data. Then the moving carrier terminal obtains the eight vertex coordinates of the three-dimensional bounding box corresponding to each second target object point cloud data, and projects the eight vertex coordinates of the three-dimensional bounding box and the point cloud data in the bounding box into the two-dimensional plane data through the internal parameter matrix and external parameter matrix of the camera and the laser radar, and then obtains the minimum and maximum row and column coordinates to form the projected two-dimensional box in the two-dimensional plane data and the point cloud projection data in the two-dimensional data form in the two-dimensional box, and obtains at least two second target object projection areas. Then the moving carrier terminal matches the at least two second target object projection areas with the at least two first target object areas respectively, and calculates the matching degree of the at least two second target object projection areas with the corresponding first target object areas. The moving carrier terminal screens at least two second target object projection areas and their corresponding first target object areas that meet the matching requirements according to the matching degree between the at least two second target object projection areas and their corresponding first target object areas and the pre-set matching requirements. The moving carrier terminal determines the second target object projection sub-area according to the non-overlapping area between the at least two second target object projection areas and their corresponding first target object areas, and then determines the point cloud projection data corresponding to the second target object projection sub-area as the expanded point cloud data.

[0086] In this embodiment, when the first target object area and the second target object point cloud data include at least two, the motion carrier terminal uses multiple second target object point cloud data for projection to obtain multiple second target object projection areas, and then uses the multiple second target object projection areas and the multiple first target object areas for matching, which can improve the detection rate of the target object and avoid missed detection, thereby improving the recognition accuracy of the expanded point cloud of the target object.

[0087] In one embodiment, step 208, performing projection based on the point cloud data of the second target object to obtain a projection area of ​​the second target object includes:

[0088] The point cloud data of the second target object is projected to the same coordinate system as the environment image data to obtain a projection area of ​​the second target object.

[0089] Specifically, the motion carrier terminal obtains the pre-set internal and external parameter matrices of the camera and lidar, projects the three-dimensional point coordinates in the second target point cloud data through the internal and external parameter matrices, converts each three-dimensional point coordinate to the same coordinate system as the environmental image data, that is, the camera coordinate system, obtains the point cloud projection data of the point cloud data on the two-dimensional plane, and obtains the second target projection area corresponding to the point cloud data of the second target object on the two-dimensional plane.

[0090] In a specific embodiment, the motion carrier terminal obtains a relationship list between point cloud projection data and point cloud data, which relationship list records the association relationship between the point cloud projection data and the point cloud data. The correspondence between the point cloud projection data and the point cloud data is generated in the process of projecting the point cloud data of the second target object to obtain the projection area of ​​the second target object, and after the projection is completed, a relationship list is generated according to the association relationship between the point cloud projection data and the point cloud data, and then the relationship list is stored.

[0091] The moving carrier terminal obtains the point cloud projection data corresponding to the projection sub-area of ​​the second target object, and determines the expanded point cloud data corresponding to the point cloud projection data in the point cloud data of the second target object according to the relationship list. The moving carrier terminal can also use the expanded point cloud data in the point cloud data of the second target object as the original expanded point cloud data in the environmental point data. Then the moving carrier terminal filters and removes the original expanded point cloud data in the environmental point cloud data.

[0092] In this embodiment, by projecting the second target object point cloud data to the same coordinate system as the environment image data, the second target object projection area can be accurately obtained, and further, the second target object projection sub-area can be accurately obtained, thereby improving the recognition accuracy of the expanded point cloud data.

[0093] In one embodiment, step 208, calculating the matching degree between the target object projection area and the target object area, includes:

[0094] The overlap degree between the first target object region and the second target object projection region is calculated to obtain the matching degree between the target object projection region and the target object region.

[0095] Specifically, the motion carrier terminal can use the overlap calculation formula to calculate the overlap between the first target object area and the second target object projection area. When it is detected that the calculated overlap reaches a preset overlap threshold, it is determined that the matching degree between the first target object area and the second target object projection area meets the matching requirements, and it is determined that there is a second target object projection area matching the first target object area in the environmental image data, and at this time, there is expanded point cloud data in the second target object projection area.

[0096] In this embodiment, by calculating the overlap between the first target object area and the second target object projection area, when the overlap meets the overlap threshold, it can be determined that there is expanded point cloud data in the second target object projection area, so that the original expanded point cloud data in the environmental point cloud data can be removed according to the expanded point cloud data in the second target object projection area to obtain the actual size point cloud data corresponding to the target object, thereby improving the recognition accuracy of the target object.

[0097] In a specific embodiment, the moving carrier terminal performs target object detection on the filtered environmental point cloud data. The target object may be an object with high reflectivity such as a traffic sign. The expanded point cloud data corresponding to the target object is removed from the filtered environmental point cloud data to obtain the true size area of ​​the target object.

[0098] If the moving carrier terminal identifies that the volume occupied by the target object in the moving area exceeds a preset ratio based on the target object area, and the moving road cannot meet the needs of the moving carrier to move, the traffic condition in the moving area is determined to be an impassable area, and then the moving carrier terminal can plan the moving route and moving speed of the moving carrier based on the traffic condition in the moving area. If the moving carrier terminal identifies that the volume occupied by the obstacle in the moving area is less than a preset ratio based on the target object area, the traffic condition in the moving area is determined to be a passable area, and the moving carrier terminal controls the moving carrier to move in the moving area according to the existing movement route and movement speed. The moving terminal can also send the traffic condition of the moving area to the management terminal, and receive the movement control instruction returned by the management terminal, and the moving carrier terminal moves according to the movement control instruction.

[0099] In this embodiment, by determining the traffic condition of the moving area where the obstacle point cloud area is located according to the target object area, it is possible to avoid misjudgment of the traffic condition of the moving area based on the target object point cloud data without clearing the expansion points, thereby improving the recognition accuracy of the passable area.

[0100] In a specific embodiment, Figure 3 As shown, a schematic diagram of a process for removing expanded point cloud data is provided; the target object may be a traffic sign.

[0101] When the autonomous driving vehicle is driving, the on-board motion carrier terminal obtains the full point cloud data of the 360-degree range of the current frame from the lidar, and obtains 360-degree image data around the autonomous driving vehicle from 6 cameras.

[0102] The vehicle terminal uses the deep learning network to detect the two-dimensional image data of the traffic sign using the image data, where the deep learning network includes but is not limited to SSD, YOLO and other networks that can detect two-dimensional image data, and obtains the detection results of the traffic sign in the image data, including the two-dimensional detection box that selects the traffic sign in the image data. The vehicle terminal uses the deep learning network to detect three-dimensional obstacles using the full point cloud data, including the detection of vehicles, pedestrians, cyclists and the ground, where the deep learning network includes but is not limited to Birdeye view, Cylinder view, 3D voxel, PointNet and other networks that can perform full-frame point cloud target detection and ground point recognition, and obtains the point cloud data corresponding to vehicles, pedestrians, cyclists and the ground in the environmental point cloud data.

[0103] After filtering the point cloud data of the ground and obstacles, the vehicle terminal uses clustering algorithms such as Euclidean clustering and DBSCAN to cluster the remaining point cloud data to obtain multiple unclassified obstacle point cloud data. Then the vehicle terminal determines whether the reflectivity of each point in each point cloud cluster exceeds the set threshold, and counts the number of points exceeding the threshold. When the number of points exceeds the set threshold, the point cloud cluster is considered to be a point cloud cluster with high reflectivity. Point cloud clusters with high reflectivity are retained. These point cloud clusters may contain expanded point cloud data, and the bounding box is estimated for the point cloud clusters with high reflectivity.

[0104] The vehicle terminal uses the calibration parameters of the laser radar and the camera to project the clustered bounding box and point cloud into the image, and obtains the projection box of the point cloud cluster in the image data. Then the vehicle terminal detects whether the projection box matches the two-dimensional detection box of the traffic sign, and calculates the overlap between the projection box and the two-dimensional detection box of the traffic sign. When the overlap reaches the set threshold, it is determined that the projection box and its corresponding two-dimensional detection box are matched, meeting the matching requirements, and it is considered that the point cloud cluster corresponding to the projection box contains the point cloud data corresponding to the traffic sign. Then the vehicle terminal determines the overlapping area of ​​the projection box and the two-dimensional detection box that meets the matching requirements. The size of the overlapping area is the same as that of the two-dimensional detection box, which represents the real area of ​​the traffic sign. The vehicle terminal obtains the point cloud projection data in the overlapping area of ​​the projection box, determines the point cloud data corresponding to the point cloud projection data in the point cloud cluster according to the relationship list, determines the real point cloud data area of ​​the traffic sign in the point cloud cluster according to the determined point cloud data, and then determines the point cloud data outside the real point cloud data area of ​​the traffic sign in the point cloud cluster as expanded point cloud data, and filters the expanded point cloud data in the point cloud cluster.

[0105] In this embodiment, since the laser radar is prone to expand point clouds at high-reflectivity objects, the projection frame is obtained by obtaining the point cloud cluster with high reflectivity corresponding to the laser radar and projecting it, and then the two-dimensional detection frame of the traffic sign in the image data is obtained. The projection frame and the two-dimensional detection frame are matched to remove the expansion points of the traffic sign, thereby improving the recognition accuracy of the traffic sign. Furthermore, it can solve the misrecognition of the traffic sign area caused by the expansion point cloud, avoid the situation where the autonomous driving vehicle suddenly brakes and stops, and improve driving safety.

[0106] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0107] Based on the same inventive concept, the embodiment of the present application also provides an expansion point cloud recognition device for implementing the above-mentioned expansion point cloud recognition method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above-mentioned method, so the specific limitations in one or more expansion point cloud recognition device embodiments provided below can refer to the above-mentioned limitations on the expansion point cloud recognition method, and will not be repeated here.

[0108] In one embodiment, Figure 5 As shown, an expansion point cloud recognition device 400 is provided, comprising: an image detection module 402, a filtering module 404, a point cloud determination module 406, a matching module 408 and a determination module 410, wherein:

[0109] An image detection module 402 is used to obtain environmental point cloud data and environmental image data corresponding to the current field of view, perform first target object detection based on the environmental image data, and obtain a first target object area corresponding to the environmental image data;

[0110] The filtering module 404 is used to cluster obstacles based on the environmental point cloud data to obtain various obstacle point cloud clusters, and filter the environmental point cloud data based on the various obstacle point cloud clusters to obtain filtered point cloud data;

[0111] A point cloud determination module 406, configured to determine second target object point cloud data from the filtered point cloud data based on reflection intensity information corresponding to the filtered point cloud data;

[0112] A matching module 408 is used to perform projection based on the point cloud data of the second target object to obtain a projection area of ​​the second target object, and calculate a matching degree between the first target object area and the projection area of ​​the second target object;

[0113] The determination module 410 is used to determine whether there is expansion point cloud data in the environment point cloud data based on the matching degree.

[0114] In one embodiment, the determination module 410 includes:

[0115] The judgment unit is used to determine that there is expanded point cloud data in the environmental point cloud data when the matching degree is greater than a preset matching threshold; and determine a second target object projection sub-area in the second target object projection area that does not overlap with the first target object area, wherein the point cloud data corresponding to the second target object projection sub-area is the expanded point cloud data.

[0116] In one embodiment, the expansion point cloud recognition device 400 further includes:

[0117] The filtering unit is used to identify the corresponding original expanded point cloud data in the environmental point cloud data based on the point cloud data corresponding to the projection sub-region of the second target object; and filter the environmental point cloud data based on the original expanded point cloud data.

[0118] In one embodiment, the point cloud determination module 406 includes:

[0119] The target point cloud determination unit is used to cluster the filtered point cloud data to obtain at least two point cloud clusters; based on the reflection intensity information corresponding to each point cloud cluster and a preset reflection intensity threshold, determine the second target object point cloud data from the at least two point cloud clusters.

[0120] In one embodiment, the matching module 408 includes:

[0121] The projection unit is used to project the point cloud data of the second target object to the same coordinate system as the environment image data to obtain a projection area of ​​the second target object.

[0122] In one embodiment, the matching module 408 includes:

[0123] The matching calculation unit is used to calculate the overlap between the first target object area and the second target object projection area to obtain the matching degree between the target object projection area and the target object area.

[0124] Each module in the above-mentioned expansion point cloud recognition device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0125] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 5As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store full point cloud data and image data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an expansion point cloud recognition method is implemented.

[0126] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 6 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, an expansion point cloud recognition method is implemented. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse, etc.

[0127] Those skilled in the art will understand that Figure 5-Figure 6The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0128] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0129] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0130] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0131] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0132] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0133] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0134] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for recognizing an expanded point cloud, characterized in that: The method comprises: Acquire environmental point cloud data and environmental image data corresponding to the current field of view, perform first target object detection based on the environmental image data, and obtain a first target object area corresponding to the environmental image data, wherein the target object is a pre-set specific detection object; Performing obstacle clustering based on the environmental point cloud data to obtain various obstacle point cloud clusters, and filtering the environmental point cloud data based on the various obstacle point cloud clusters to obtain filtered point cloud data; determining second target object point cloud data from the filtered point cloud data based on reflection intensity information corresponding to the filtered point cloud data; Performing projection based on the point cloud data of the second target object to obtain a second target object projection area, and calculating a matching degree between the first target object area and the second target object projection area, wherein the matching degree is a degree of area overlap; When the matching degree satisfies a preset matching degree threshold, it is determined that there is expansion point cloud data in the environment point cloud data; when the matching degree does not satisfy the matching degree threshold, it is determined that there is no expansion point cloud data in the environment point cloud data.

2. The method according to claim 1, characterized in that When the matching degree satisfies a preset matching degree threshold, determining that there is expansion point cloud data in the environment point cloud data includes: When the matching degree is greater than a preset matching degree threshold, determining that there is expansion point cloud data in the environment point cloud data; and A second target object projection sub-region in the second target object projection region that does not overlap with the first target object region is determined, and point cloud data corresponding to the second target object projection sub-region is expanded point cloud data.

3. The method according to claim 2, characterized in that The method further comprises: Identifying the corresponding original expanded point cloud data in the environmental point cloud data based on the point cloud data corresponding to the projection sub-region of the second target object; and The environmental point cloud data is filtered based on the original expanded point cloud data.

4. The method according to claim 1, characterized in that: The step of determining the second target object point cloud data from the filtered point cloud data based on the reflection intensity information corresponding to the filtered point cloud data comprises: Clustering the filtered point cloud data to obtain at least two point cloud clusters; The second target object point cloud data is determined from the at least two point cloud clusters based on the reflection intensity information corresponding to each point cloud cluster and a preset reflection intensity threshold.

5. The method according to claim 1, characterized in that The projecting based on the second target object point cloud data to obtain a second target object projection area includes: The second target object point cloud data is projected to the same coordinate system as the environment image data to obtain the second target object projection area.

6. The method according to claim 1, characterized in that The calculating the matching degree between the target object projection area and the target object area includes: Based on the overlap calculation between the first target object area and the second target object projection area, the matching degree between the target object projection area and the target object area is obtained.

7. An expansion point cloud recognition device, characterized in that: The device comprises: An image detection module is used to obtain environmental point cloud data and environmental image data corresponding to the current field of view, perform first target object detection based on the environmental image data, and obtain a first target object area corresponding to the environmental image data, wherein the target object is a pre-set specific detection object; A filtering module, configured to cluster obstacles based on the environment point cloud data to obtain individual obstacle point cloud clusters, and filter the environment point cloud data based on the individual obstacle point cloud clusters to obtain filtered point cloud data; a point cloud determination module, configured to determine point cloud data of a second target object from the filtered point cloud data based on reflection intensity information corresponding to the filtered point cloud data; a matching module, configured to perform projection based on the point cloud data of the second target object to obtain a projection area of ​​the second target object, and calculate a matching degree between the area of ​​the first target object and the projection area of ​​the second target object, wherein the matching degree is a degree of area overlap; The determination module is used to determine that there is expansion point cloud data in the environmental point cloud data when the matching degree meets a preset matching degree threshold, and to determine that there is no expansion point cloud data in the environmental point cloud data when the matching degree does not meet the matching degree threshold.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images