A point cloud data processing method, device, equipment and storage medium

Abstract

The present disclosure provides a point cloud data processing method, device and equipment and a storage medium. The method projects each point in the point cloud data of a road object on a grid of each image to determine a target road object corresponding to the grid. If at least two road objects correspond to the grid, the method determines a target road object closest to the image acquisition device from the at least two road objects according to the point cloud data. In this way, the problem that the category of the obtained point is inaccurate when at least two road objects overlap due to the lack of depth information after each point in the point cloud data of a road object is projected on an image is solved, thereby improving the accuracy of the category of the point.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field, and particularly relates to a point cloud data processing method and device, equipment and a storage medium. BACKGROUND

[0002] In the field of perception, 3D point cloud data of a scene can be detected by radar. When determining the category of each road object in the point cloud data, each point in the point cloud data of the road object is projected onto a 2D image to obtain a projection point coordinate, and then the category of the projection point coordinate is obtained from the semantic segmentation result of the image, so that the category of the road object can be determined according to the category of the projection point coordinate.

[0003] However, in the case where the projections of two road objects overlap, the category of the road object obtained is not accurate enough. SUMMARY

[0004] To overcome the problems in the related art, the present disclosure provides a point cloud data processing method, device, equipment and a storage medium.

[0005] According to a first aspect of the present disclosure, a point cloud data processing method is provided, the method comprising:

[0006] obtaining point cloud data of a plurality of road objects collected by a radar, and at least one image collected by at least one image collection device, wherein the road objects in the point cloud data at least partially coincide with the road objects contained in each image;

[0007] determining a target road object corresponding to each grid by projecting each point in the point cloud data of the road object in each grid of each image, wherein the grid comprises at least one pixel, and in the case where the grid corresponds to at least two road objects, determining a target road object closest to the image collection device from the at least two road objects according to the point cloud data;

[0008] for each point in the point cloud data of the road object, obtaining a target image in which the target road object corresponding to the grid where the projection point of the point is located in each image is consistent with the road object to which the point belongs, and determining the pixel category corresponding to the projection point in the target image as the category of the point;

[0009] determining the category of the road object according to the category of each point in the point cloud data of the plurality of road objects.

[0010] In combination with any of the embodiments provided in the present disclosure, the determining the target road object corresponding to the grid by projecting each point in the point cloud data of the road object into the grid of each image comprises: determining a projection point coordinate of the point on the image according to a pose relationship between the radar and the image acquisition device; determining the grid to which the projection point coordinate belongs according to the projection point coordinate; and determining the target road object corresponding to the grid according to the road object to which the point belongs in the point cloud data.

[0011] In combination with any of the embodiments provided in the present disclosure, the determining the grid to which the projection point coordinate belongs according to the projection point coordinate comprises: dividing the projection point coordinate by a preset scale and performing an integer processing; obtaining a target grid coordinate matched with the projection point coordinate after the integer processing from a plurality of grid coordinates, the grid coordinate being determined by dividing a pixel coordinate in the image by the preset scale and performing an integer processing; and determining the grid to which the projection point coordinate belongs according to the target grid coordinate.

[0012] In combination with any of the embodiments provided in the present disclosure, after the determining the target road object corresponding to the grid by projecting each point in the point cloud data of the road object into the grid of each image, the method further comprises:

[0013] establishing a correspondence table of the images, the correspondence table being used to indicate a correspondence relationship between the grid and the target road object;

[0014] The obtaining, for each point in the point cloud data of the road object, a target image in which a target road object corresponding to a grid in which a projection point of the point is located is consistent with a road object to which the point belongs in each image comprises: determining the target road object corresponding to the grid in which the projection point of the point is located according to the correspondence table of each image; and determining, as a target image corresponding to the point, an image in which a target road object corresponding to the point in the at least one image is consistent with the road object to which the point belongs.

[0015] In combination with any of the embodiments provided in the present disclosure, the road object in the point cloud data is provided with a first road object ID;

[0016] After the determining the target road object corresponding to the grid by projecting each point in the point cloud data of the road object into the grid of each image, the method further comprises: determining a second road object ID corresponding to the grid according to the target road object corresponding to the grid;

[0017] The determining grid unit is configured to: determine a projection point coordinate of each point in the point cloud data of the road object on the image according to a pose relationship between the radar and the image acquisition device; determine a grid to which the projection point coordinate belongs according to the projection point coordinate; and determine a target road object corresponding to the grid according to a road object to which the point belongs in the point cloud data.

[0018] According to any one of the embodiments provided in the present disclosure, the method further comprises:

[0019] performing semantic recognition on the image to determine a pixel category of each pixel in the image.

[0020] According to a second aspect of the present disclosure, a point cloud data processing apparatus is provided, and the apparatus comprises:

[0021] an acquisition unit configured to acquire point cloud data of a plurality of road objects collected by a radar and at least one image collected by at least one image acquisition device, wherein road objects in the point cloud data at least partially coincide with road objects contained in each image;

[0022] a determining grid unit configured to determine a target road object corresponding to a grid by projecting each point in the point cloud data of the road object in the grid of each image, wherein the grid comprises at least one pixel, and in a case where the grid corresponds to at least two road objects, determining a target road object closest to the image acquisition device from the at least two road objects according to the point cloud data;

[0023] a target image determining unit configured to, for each point in the point cloud data of the road object, acquire a target image in which a target road object corresponding to a grid in which a projection point of the point is located coincides with a road object to which the point belongs in each image, and determine a pixel category corresponding to the projection point in the target image as a category of the point;

[0024] a category determining unit configured to determine a category of the road object according to the category of each point in the point cloud data of the plurality of road objects.

[0025] According to any one of the embodiments provided in the present disclosure, the determining grid unit is configured to: determine a projection point coordinate of each point in the point cloud data of the road object on the image according to a pose relationship between the radar and the image acquisition device; determine a grid to which the projection point coordinate belongs according to the projection point coordinate; and determine a target road object corresponding to the grid according to a road object to which the point belongs in the point cloud data.

[0026] In combination with any of the embodiments provided in the present disclosure, the determining the grid unit is configured to: divide the projection point coordinate by a preset scale and perform an integer processing; in a plurality of grid coordinates, obtain a target grid coordinate matched with the projection point coordinate after the integer processing, the grid coordinate being determined by dividing a pixel coordinate in the image by the preset scale and performing the integer processing; and determine a grid to which the projection point coordinate belongs according to the target grid coordinate.

[0027] In combination with any of the embodiments provided in the present disclosure, after the grid unit is determined, the method further includes: establishing, by an establishing unit, a correspondence table of the images, the correspondence table being used to indicate a correspondence between a grid and a target road object; and determining, by the determining target image unit, a target road object corresponding to the grid in which the projection point of the point is located according to the correspondence table of each image after the establishing unit establishes the correspondence; and determining an image in which a target road object corresponding to the point in the at least one image is consistent with a road object to which the point belongs as a target image corresponding to the point.

[0028] In combination with any of the embodiments provided in the present disclosure, the road object in the point cloud data is provided with a first road object ID.

[0029] The determining grid unit is further configured to determine a second road object ID corresponding to the grid according to the target road object corresponding to the grid after the target road object corresponding to the grid is determined.

[0030] The determining target image unit is configured to: obtain a second road object ID corresponding to the grid in which the projection point of the point is located in each image; and determine an image in which the second road object ID is consistent with the first road object ID to which the point belongs as a target image corresponding to the point.

[0031] According to a third aspect of the present disclosure, an electronic device is provided, the device including: a processor; and a memory for storing processor-executable instructions to perform the method according to any of the embodiments of the present disclosure.

[0032] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, the computer-readable storage medium storing computer program instructions, the computer program instructions being executed by a processor to implement the method according to any of the embodiments of the present disclosure.

[0033] The technical solutions provided in the present disclosure can include the following beneficial effects:

[0034] The point cloud data processing method, device, equipment and storage medium provided by one or more embodiments of the present disclosure project each point in the point cloud data of a road object in a grid of each image to determine a target road object corresponding to the grid, and if at least two road objects corresponding to the grid appear, determine a target road object closest to the image acquisition device from the at least two road objects according to the point cloud data. In this way, the problem that the category of the point is inaccurate when at least two road objects overlap due to the lack of depth information after each point in the point cloud data of a road object is projected to an image can be solved, thereby improving the accuracy of the category of the point.

[0035] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, rather than limiting the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0036] The accompanying drawings incorporated in and forming a part of the specification illustrate embodiments consistent with the present disclosure and serve to explain the technical solutions of the present disclosure together with the specification.

[0037] Figure 1 FIG. 1 is a schematic diagram of projection coverage leading to category error according to an exemplary embodiment of the present disclosure.

[0038] Figure 2 FIG. 2 is a flowchart of a point cloud data processing method according to an exemplary embodiment of the present disclosure.

[0039] Figure 3 FIG. 3 is a schematic diagram of a grid on an image according to an exemplary embodiment of the present disclosure.

[0040] Figure 4 FIG. 4 is a flowchart of a method for determining a target road object corresponding to a grid according to an exemplary embodiment of the present disclosure.

[0041] Figure 5 FIG. 5 is a flowchart of a method for determining a category of a road object according to an exemplary embodiment of the present disclosure.

[0042] Figure 6 FIG. 6 is a structural schematic diagram of a point cloud data processing device according to an exemplary embodiment of the present disclosure.

[0043] Figure 7 FIG. 7 is a structural schematic diagram of an electronic device for point cloud data processing according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0044] The exemplary embodiments will be described in detail herein with reference to the attached drawings. The description of the exemplary embodiments is intended to apply to all alternative embodiments, as would be understood by persons skilled in the art. To that end, the following description is not intended to limit the exemplary embodiments to a particular application or applications in conjunction with a particular device or group of devices. As such, the following description is not intended to limit the scope of the claims to the normal meaning of the claims and equivalents thereof.

[0045] The terminology used in this description is for the purpose of describing particular embodiments only and is not intended to limit the description. As used in this description and the appended claims, the singular forms "a," "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0046] It will be understood that, although the terms first, second, third, etc. can be used herein to describe various information, these terms are not intended to denote a particular order or hierarchy among the information. The terms are used only to distinguish one piece of information from another. For example, a first piece of information can be termed a second piece of information, and similarly, a second piece of information can be termed a first piece of information without departing from the scope of the present description. As used herein, the term "if' can be interpreted to mean "when" or "upon" or "in response to determining" taking into account the context in which the term is used.

[0047] The present disclosure can be applied to determining the category of a road object in point cloud data, which can include obstacles encountered by an autonomous vehicle during autonomous driving. A radar is usually installed on the autonomous vehicle, which can be used to obtain point cloud data, through which complete spatial information around the vehicle can be reproduced, providing a basis for subsequent driving decisions. A plurality of image acquisition devices can also be installed on the autonomous vehicle to obtain images around the vehicle. The image acquisition devices can include cameras, video cameras, etc. It can be understood that the scenes in which the image acquisition devices obtain images and the scenes in which the radar obtains point cloud data at least partially overlap.

[0048] The point cloud data collected by the radar is pre-processed for segmentation, that is, the points in the point cloud data belonging to the same road object can be clustered to obtain point cloud data of different road objects, at which time the categories of the road objects are not known. When determining the category of a road object, each point in the point cloud data of the road object can be projected onto an image to obtain a corresponding projection point according to a point cloud-camera calibration matrix, and then a pixel category at a position of the projection point can be obtained from a semantic recognition result of the image, and the category of each point can be determined according to the pixel category corresponding to the projection point of the point, thereby determining the category of the road object.

[0049] But in the process of projecting each point in the point cloud data of the road object to the image, due to the lack of depth information, the projection points of the points of different road objects on the image may be overlapped, and the category of the road object far away from the camera is actually the category of the road object close to the camera, which will lead to the inaccuracy of the category of the point cloud data of the road object far away from the camera.

[0050] In order to more clearly illustrate the phenomenon of category error caused by projection overlap, the present disclosure combines Figure 1 to make a specific description. Figure 1 is a schematic diagram of projection overlap leading to category error shown by the present disclosure according to an exemplary embodiment. As Figure 1 shown, the first image 10 can be collected by using the first image collection device, the second image 11 can be collected by using the second image collection device, and the third image 12 can be collected by using the third image collection device. The point cloud data 13 can be collected by using the radar, and the point cloud data of multiple road objects is included in the point cloud data 13. The actual category of the road object 101 is determined to be a pedestrian through the second image 11, and the actual category of the road object 102 is determined to be a bicycle through the third image 12. Ideally, the category of the point of the road object 101 is expected to be a pedestrian, and the category of the point of the road object 102 is expected to be a bicycle.

[0051] But when the road object 101 and the road object 102 in the point cloud data 13 are projected onto the second image 11, it can be seen that the road object 101 covers the road object 102. At this time, since the road object 101 is closer to the second image collection device than the road object 102, the category of the road object 102 is theoretically obtained to be a bicycle, but the category of the road object 102 is actually obtained to be a pedestrian. In other words, when the point cloud data of the road object 102 is projected onto the second image 11, the category of the road object 102 is inaccurate due to the occlusion of the road object 101.

[0052] In view of this, the present disclosure provides a point cloud data processing method. In the present disclosure, by projecting each point in the point cloud data of the road object in each grid of the image, the target road object corresponding to the grid is determined, and in the case where the grid corresponds to at least two road objects, the target road object closest to the image collection device among the at least two road objects is determined according to the point cloud data. In this way, the problem that when each point in the point cloud data of the road object is projected to the image, the category of the point obtained is inaccurate due to the overlap of at least two road objects caused by the lack of depth information can be solved, thereby improving the accuracy of the category of the point.

[0053] The following embodiments, in conjunction with the accompanying drawings, will provide a detailed description of a point cloud data processing method.

[0054] Figure 2 This is a flowchart illustrating a point cloud data processing method according to an exemplary embodiment of this disclosure. Figure 2 As shown, the present disclosure provides a point cloud data processing method, which includes the following steps 201 to 204.

[0055] In step 201, point cloud data of multiple road objects collected by radar and at least one image collected by at least one image acquisition device are obtained.

[0056] In some embodiments, the radar and image acquisition device can be mounted on the same object, which can be an autonomous vehicle. Point cloud data of multiple road objects acquired by the radar are obtained. These road objects may include other objects involved in road traffic within a defined range of the autonomous vehicle, such as pedestrians, bicycles, electric vehicles, and other vehicles. The radar can be a lidar, millimeter-wave radar, or ultrasonic radar, etc.

[0057] Typically, three image acquisition devices can be installed on an autonomous vehicle to acquire images collected by each of the three devices. Usually, the road objects in the point cloud data at least partially overlap with the road objects contained in each of the images.

[0058] by Figure 1 Taking the scenario shown as an example, some road objects in point cloud data 13 overlap with some road objects in each of the images (first image 10, second image 11, and third image 12). In other words, the scene captured by each image acquisition device at least partially overlaps with the scene scanned by radar.

[0059] In step 202, the target road object corresponding to the grid is determined by projecting each point in the point cloud data of the road object onto the grid of each image. The grid includes at least one pixel. When the grid corresponds to at least two road objects, the target road object that is closest to the image acquisition device among the at least two road objects is determined based on the point cloud data.

[0060] For each road object, the point cloud data of the road object may include multiple points, and each point in the point cloud data of the road object is projected onto a separate image.

[0061] In some embodiments, the pixels on each image can be pre-divided into multiple grids according to a division criterion. The division criterion can be set to one grid per pixel, or multiple pixels (e.g., 20*20) per grid.

[0062] Figure 3 is a schematic diagram of grids on an image according to an exemplary embodiment of the present disclosure. After processing the first image 10 according to the division basis, a plurality of grids 30 corresponding to the image can be obtained, as shown in the effect. Figure 3 It should be noted that, Figure 3 is to intuitively show the relationship between the image and the grid, so as to facilitate the understanding of the scheme, and does not represent that the similar grids are drawn on the image in the actual execution process, that is, the grid can be a virtual concept for the understanding of the scheme, and does not represent that the grid shown in Figure 3 For example, the grid 30 can be represented by the pixel coordinates on the image, for example, the grid 30 can be represented by the pixel coordinates (708, 0).

[0063] The points in the point cloud data of the road object are projected on each image respectively, and the projection points of the points usually fall into a certain grid in the image, so that the grid to which the projection point of the point belongs can be determined, and the road object of the point is determined as the target road object corresponding to the grid. In the case where the grid corresponds to at least two road objects, the target road object closest to the image acquisition device is determined from the at least two road objects according to the point cloud data.

[0064] In some embodiments, the point cloud data of a plurality of road objects and a plurality of images can be included, and in the projection process, the projection points of the points in the point cloud data of each road object on the image can be obtained, the grid to which the projection point belongs can be determined, and the target road object corresponding to the grid can be determined until all images are traversed. The target road object corresponding to the grid of each image can be obtained through step 202.

[0065] In order to more clearly illustrate step 202, one of the cases is set as follows: road object A and point a in the point cloud data of road object A, road object B and point b in the point cloud data of road object B, point a and point b are projected onto the first image 10, and the first image 10 corresponds to the first image acquisition device.

[0066] Assuming that the acquisition point a is projected to the first image 10, the grid to which the projection point of the point a belongs is grid 30, and in a case where it is determined that the grid 30 does not correspond to a target road object, it can be determined that the target road object corresponding to the grid 30 is road object A. In a possible embodiment: after traversing all road objects, no projection point of a point of another road object falls into the grid 30, at this time, it is indicated that the target road object corresponding to the grid 30 is road object A. In another possible embodiment: when the point b in road object B is projected on the first image 10, the grid to which the projection point of the point b belongs can also be grid 30, since the target road object corresponding to the grid 30 is road object A, at this time, the distance A between road object A and the first image acquisition device in the 3D space and the distance B between road object B and the first image acquisition device need to be determined. If the distance A is less than the distance B, it is indicated that road object A is closer to the first image acquisition device, and then the target road object corresponding to the grid 30 is kept as road object A; if the distance A is greater than the distance B, it is indicated that road object B is closer to the first image acquisition device, and then the target road object corresponding to the grid 30 is updated to road object B.

[0067] In step 203, for each point in the point cloud data of the road object, a target image in which the target road object corresponding to the grid to which the projection point of the point belongs is consistent with the road object to which the point belongs is acquired, and the pixel category corresponding to the projection point in the target image is determined as the category of the point.

[0068] Through step 202, the target road object corresponding to each grid on each image can be determined, and then for each point in the point cloud data of each road object, a target image in which the target road object corresponding to the grid to which the projection point of the point belongs is consistent with the road object to which the point belongs can be acquired. For example, for the point a of road object A, the images in which the target road object corresponding to the grid to which the projection point of the point a belongs is road object A are searched through each image, and the image is taken as the target image. Assuming that the pixel category corresponding to the projection point in the target image is a pedestrian, it is determined that the category of the point is a pedestrian.

[0069] In some embodiments, the images can be subjected to semantic recognition to determine the pixel category of each pixel in the image.

[0070] In step 204, the category of the road object is determined according to the categories of the points in the point cloud data of the plurality of road objects.

[0071] The point cloud data of each road object can include a plurality of points, and after the categories of the points of the road object are determined, the categories of all points can be classified, and the category with the largest number of points is taken as the category of the road object.

[0072] In some embodiments, step 202 of determining the target road object corresponding to the grid based on the projection of each point in the point cloud data of the road object in each image includes steps 2021-2023.

[0073] In step 2021, the projection point coordinates of the points on the image are determined according to the pose relationship between the radar and the image acquisition device.

[0074] In general, the projection point coordinates of each point in the point cloud data of the road object on the first image 10 can be determined according to the pose relationship between the radar and the first image acquisition device, and the projection point coordinates of each point in the point cloud data of the road object on the second image 11 can also be determined according to the pose relationship between the radar and the second image acquisition device.

[0075] In some embodiments, the projection point coordinates of each point in the point cloud data of the road object on the image corresponding to the image acquisition device can be determined according to the calibration matrix between the point cloud data of the road object and the image acquisition device.

[0076] For example, for each camera, use cam_x (1≤x≤n, n is the number of cameras) to represent camera x, and let the point cloud data-camera calibration matrix of cam_x be M_x, and iterate through all road objects. Assuming that the point cloud data of road object i is points_i, and let the jth point in the point cloud data be points_ij, calculate the projection of points_ij on the image through M_x, and let the projection point be image_point_ij, and the coordinates of the projection point are (image_point_ij.x, image_point_ij.y).

[0077] In some embodiments, after determining the projection point coordinates of the points, the points and the projection point coordinates of the points can be recorded, which facilitates direct use in step 203, eliminates the need for repeated calculation, and reduces the amount of calculation.

[0078] In step 2022, the grid to which the projection point coordinates belong is determined according to the projection point coordinates.

[0079] In general, the grid coordinates corresponding to the grid can be obtained when the grid is known, and then when the projection point coordinates are known, the grid to which the projection point coordinates belong can be determined by judging whether the projection point coordinates fall within the range corresponding to the grid coordinates.

[0080] In step 2023, the target road object corresponding to the grid is determined according to the road object to which the point in the point cloud data belongs.

[0081] In the case of determining the grid to which the projection point coordinate belongs, the road object to which the point in the point cloud data belongs can be determined as the target road object corresponding to the grid. For example, the projection point coordinate of point a belongs to grid 30 (that is, the projection point coordinate of point a falls into grid 30), and the road object A to which point a belongs is determined as the target road object corresponding to grid 30.

[0082] In some embodiments, according to the projection point coordinate, the grid to which the projection point coordinate belongs is determined, including: dividing the projection point coordinate by a preset scale and performing an integer processing; in a plurality of grid coordinates, obtaining a target grid coordinate matched with the projection point coordinate after the integer processing, the grid coordinates being determined by dividing the pixel coordinates in the image by the preset scale and performing an integer processing; and according to the target grid coordinate, determining the grid to which the projection point coordinate belongs.

[0083] In the embodiment, the pixel coordinates in each image can be divided by a preset scale and an integer processing can be performed to obtain the grid coordinates of a plurality of grids corresponding to an image. The preset scale can be 20. For example, the first image 10 can include 1920*1080 pixels, and after dividing each pixel coordinate by the preset scale 20 and performing an integer processing, 96*54 grids can be obtained, and each grid can be represented by a grid coordinate. For example, the pixel coordinate (20, 10) divided by the preset scale 20 can obtain (1, 0.5), and after the integer processing of (1, 0.5) downward, the grid coordinate (1, 0) can be obtained, and (1, 0) is used to represent a grid. From the perspective of the grid, it can be understood that the pixel coordinates in the range of {[1-2), [0-1)} can be represented by the grid (1, 0).

[0084] Continuing the example of the projection point coordinate being (image_point_ij.x, image_point_ij.y), the projection point coordinate is divided by a preset scale and an integer processing is performed, wherein the preset scale is the preset scale 20 used to obtain the grid coordinate, and the projection point coordinate divided by 20 can obtain (image_point_ij.x / 20, image_point_ij.y / 20), and then the integer processing of (image_point_ij.x / 20, image_point_ij.y / 20) downward can obtain the integer coordinate of the projection point. For example, assuming that the projection point coordinate (10, 35) divided by the preset scale 20 can obtain (0.5, 1.75), and after the integer processing, (0, 1) can be obtained, that is, the projection point between {[0-1), [1-2)} can be represented by (0, 1).

[0085] In the case that the plurality of grid coordinates and the projection point coordinates after the rounding processing are known, the target grid coordinate matching the projection point coordinate after the rounding processing can be obtained, and the grid to which the projection point coordinate belongs is determined according to the target grid coordinate.

[0086] By processing the projection point coordinates and the pixel coordinates according to the preset proportion, the calculation amount can be reduced, and the space for storing the grid can be reduced.

[0087] In some embodiments, after determining the target road object corresponding to the grid by projecting each point in the point cloud data of the road object in each grid of each image, a correspondence table of the images can be established, and the correspondence table is used to indicate the correspondence between the grid and the target road object. For example, a correspondence table T x of the image collected by the camera cam x can be established, T x has a size of 96*54, and the grid and the target road object corresponding to the grid are recorded in T x.

[0088] In some embodiments, for each point in the point cloud data of the road object, the target image in which the target road object corresponding to the grid in which the projection point of the point is located is consistent with the road object to which the point belongs is obtained, including: determining the target road object corresponding to the grid in which the projection point of the point is located according to the correspondence table of each image; and determining the image in which the target road object corresponding to the point in the at least one image is consistent with the road object to which the point belongs as the target image corresponding to the point.

[0089] In the case that the correspondence table of each image is known, the correspondence table of each image can be traversed to determine the target road object corresponding to the grid where the projection point of the point is located, and the image in which the target road object corresponding to the point is consistent with the road object to which the point belongs is determined as the target image corresponding to the point. For example, assuming that there are a first image 10, a second image 11 and a third image 12, the correspondence table 1-0 corresponding to the first image 10, the correspondence table 1-1 corresponding to the second image 11 and the correspondence table 1-2 corresponding to the third image 12 can be obtained. For each point in the point cloud data of the road object, the target road object corresponding to the grid where the projection point of the point is located is determined. Assuming that for point a, the target road object corresponding to the grid where the projection point of the point a is located in the correspondence table 1-0 is road object A, the first image 10 can be determined as the target image corresponding to the point a, and in this case, the correspondence table 1-1 and the correspondence table 1-2 are no longer traversed. Assuming again that for point a, the correspondence table 1-0, the correspondence table 1-1 and the correspondence table 1-2 are all traversed and no image in which the target road object corresponding to the grid is road object A is found, the category of the point a can be set as a default value, which indicates that the category of the point a is not found, and it can also be understood that there is no projection point corresponding to the point a on the first image 10, the second image 11 and the third image 12.

[0090] In some embodiments, a first road object ID can be set for the road object in the point cloud data. In the case that the first road object ID is set for the road object, after the target road object corresponding to the grid where each point in the point cloud data of the road object is projected in each image is determined, the second road object ID corresponding to the grid is determined according to the target road object corresponding to the grid. It can be understood that the correspondence between the grid and the second road object ID is established, and the target road object corresponding to the grid is represented by the second road object ID. The target road object is represented by the road object ID, which can reduce the occupied storage space.

[0091] In some embodiments, for each point in the point cloud data of the road object, the target image in which the target road object corresponding to the grid where the projection point of the point is located in each image is consistent with the road object to which the point belongs is obtained, including: obtaining the second road object ID corresponding to the grid where the projection point of the point is located in each image; and determining the image in which the second road object ID is consistent with the first road object ID of the point as the target image corresponding to the point.

[0092] In order to more clearly illustrate the present disclosure, the following embodiments are specifically described by taking the camera cam_x as an example.

[0093] For each camera, a correspondence table T_x is established for the image collected by camera cam_x:

[0094] With the point cloud data of cam_x - camera calibration matrix M_x, all road objects are traversed, assuming that the first road object ID of road object i is i, and the point cloud data of road object i is points_i, and the jth point in the point cloud data points_i is points_ij.

[0095] Referring to Figure 4 In step 401, points_ij is calculated by M_x to obtain the projection point on the image, denoted as image_point_ij; in step 402, the grid (image_point_ij.x / 20, image_point_ij.y / 20) in T_x is searched, if the grid has no corresponding record of the second road object ID, that is, points_ij is the first point projected to this position, step 403 is executed to establish the second road object ID corresponding to the grid as i, that is, T_x (image_point_ij.x / 20, image_point_ij.y / 20) can be directly assigned as i; if the grid has already recorded the second road object ID, assuming that the second road object ID originally recorded by T_x (image_point_ij.x / 20, image_point_ij.y / 20) is i', step 404 is executed to compare the distance of road object i and road object i' from the camera (the spatial position information obtained by 3D point cloud is compared), if road object i is closer to the camera, step 405 is executed to update the second road object ID recorded by the grid T_x (image_point_ij.x / 20, image_point_ij.y / 20) from i' to i; if road object i' is closer to the camera, step 406 is executed to maintain the second road object ID recorded by the grid as i'.

[0096] For each point points_ij of each road object i, referring to Figure 5For the camera cam_x, the projection point coordinates (image_point_ij.x, image_point_ij.y) of points_ij on the image collected by the camera are obtained in step 501, and it is checked in the correspondence table T_x corresponding to the image whether the second road object ID corresponding to the grid (image_point_ij.x / 20, image_point_ij.y / 20) is consistent with the first road object ID of the road object i to which the point belongs. If consistent, step 502 is performed to obtain the pixel point category of (image_point_ij.x, image_point_ij.y), and if inconsistent, step 503 is performed to calculate the next camera until the category is found or all cameras are enumerated, and if all are enumerated and the category is not found, the category of the point points_ij can be determined as the default value. After step 502 is performed, step 505 can be performed, in which it is judged whether the categories of all points of the road object i are obtained. If the categories of all points of the road object i are obtained, step 506 is performed to determine the category of the road object i. If the categories of all points of the road object i are not obtained, step 507 is performed to obtain the category of the next point of the road object i.

[0097] Corresponding to the embodiments of the foregoing method, the disclosure also provides embodiments of devices and terminals to which the devices are applied.

[0098] Figure 6 A structural schematic diagram of a point cloud data processing device according to an exemplary embodiment of the disclosure is shown. As shown in Figure 6 A point cloud data processing device, the device comprising:

[0099] An acquisition unit 601 is configured to acquire point cloud data of a plurality of road objects collected by a radar and at least one image collected by at least one image collection device, and the road objects in the point cloud data at least partially coincide with the road objects contained in each image;

[0100] A determination grid unit 602 is configured to determine a target road object corresponding to a grid by projecting each point in the point cloud data of the road object in the grid of each image, wherein the grid comprises at least one pixel, and in the case where the grid corresponds to at least two road objects, the target road object closest to the image collection device is determined from the at least two road objects according to the point cloud data;

[0101] determine, for each point in the point cloud data of the road object, a target image in which a target road object corresponding to a grid in which a projection point of the point in each image is located is consistent with a road object to which the point belongs, and determine a pixel category corresponding to the projection point in the target image as a category of the point;

[0102] determine, for each point in the point cloud data of the road object, a target image in which a target road object corresponding to a grid in which a projection point of the point in each image is located is consistent with a road object to which the point belongs, and determine a pixel category corresponding to the projection point in the target image as a category of the point;

[0103] In some embodiments, the determining grid unit 602 is configured to: determine a projection point coordinate of the point projected on the image according to a pose relationship between the radar and the image acquisition device; determine a grid to which the projection point coordinate belongs according to the projection point coordinate; and determine a target road object corresponding to the grid according to a road object to which the point belongs in the point cloud data.

[0104] In some embodiments, the determining grid unit 602 is configured to: divide the projection point coordinate by a preset scale and perform an integer processing; obtain a target grid coordinate matched with the projection point coordinate after the integer processing from a plurality of grid coordinates, the grid coordinates being determined by dividing pixel coordinates in the image by the preset scale and performing an integer processing; and determine the grid to which the projection point coordinate belongs according to the target grid coordinate.

[0105] In some embodiments, after the determining grid unit 602, the method further includes: establishing a correspondence table of the images, the correspondence table being configured to indicate a correspondence relationship between a grid and a target road object; and determining, by the determining target image unit 603, a target road object corresponding to a grid in which a projection point of the point is located according to the correspondence table of each image after the establishing unit establishes the correspondence, and determining an image in which the target road object corresponding to the point in the at least one image is consistent with a road object to which the point belongs as a target image corresponding to the point.

[0106] In some embodiments, the road objects in the point cloud data are provided with first road object IDs;

[0107] The determining grid unit 602 is further configured to determine a second road object ID corresponding to the grid according to the target road object corresponding to the grid after determining the target road object corresponding to the grid.

[0108] The determining target image unit 603 is configured to obtain a second road object ID corresponding to a grid in which a projection point of the point is located in each image, and determine an image in which the second road object ID is consistent with a first road object ID to which the point belongs as a target image corresponding to the point.

[0109] Figure 7 An electronic device structure schematic diagram for point cloud data processing provided by at least one embodiment of the present disclosure is shown in FIG. 1. As shown in the figure, the electronic device includes a memory and a processor. The memory is configured to store computer instructions executable on the processor, and the processor is configured to implement the point cloud data processing method according to any one of the embodiments of the present disclosure when executing the computer instructions. Figure 7

[0110] The present disclosure also provides a computer readable storage medium having a computer program stored thereon, wherein the program is executed by a processor to implement the point cloud data processing method according to any one of the embodiments of the present disclosure.

[0111] Those skilled in the art should understand that one or more embodiments of the present disclosure can be provided as a method, a system or a computer program product. Therefore, one or more embodiments of the present disclosure can be in the form of a completely hardware embodiment, a completely software embodiment or an embodiment combining software and hardware aspects. Moreover, one or more embodiments of the present disclosure can be in the form of a computer program product implemented on one or more computer usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer usable program code.

[0112] In the present specification, "and / or" means at least one of the two, for example, "A and / or B" includes three cases: A, B, and "A and B".

[0113] Each embodiment in the present specification is described in a progressive manner, and the same or similar parts between each embodiment can be referred to each other. Each embodiment mainly describes the difference from other embodiments. In particular, for the data processing device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiment.

[0114] The above describes specific embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the acts or steps recited in the claims can be performed in a different order than the order in which they are recited and still achieve desirable results. In addition, the processes depicted in the figures do not necessarily require the particular order shown, or sequential order, to achieve the desired results. In certain implementations, multitasking and parallel processing can be advantageous.

[0115] ​Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory program carrier for execution by, or to control the operation of, data processing apparatus. Alternatively or additionally, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

[0116] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit), and the apparatus can be implemented as special purpose logic circuitry.

[0117] Computers suitable for the execution of a computer program include, by way of example, general and / or special purpose microprocessors, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read-only memory and / or a random access memory. The essential elements of a computer are a central processing unit for performing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name just a few.

[0118] Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0119] While the specification contains many specifics, these should not be construed as limiting the scope of any invention or application, but as merely providing illustrations of some of the embodiments of the inventions. Certain features that are, for clarity, described above in the context of separate embodiments, can also be provided in combination in a single embodiment. Conversely, various features that are, for brevity, described above in the context of a single embodiment, can also be provided separately or in any suitable subcombination. In addition, while features can be described above as being implemented in digital electronic circuitry, forms of the inventions can also be implemented using analog circuitry, as one of ordinary skill in the art will recognize. Furthermore, forms of the inventions can be implemented in one or more computer programs, which can be executed on one or more programmable computers, each computer including its own volatile memory (e.g., random access memory) and non-volatile memory (e.g., read-only memory). Each computer also includes a processor, which can be a general purpose processor or a special purpose processor, and a computer readable medium, which can be a volatile memory, a non-volatile memory, or a combination of the two. The computer readable medium includes instructions stored thereon, which, when executed by the processor, cause the computer to perform the methods described above. The computer readable medium can be removable and / or non-removable memory including, but not limited to, floppy disks, magnetic disks, optical disks, DVDs, CD-ROMs, CD-RWs, CD-Rs, DVDs-R, DVDs-RWs, flash memory, USB flash drives, external hard drives, memory cards, memory sticks, and any other medium which can be used to store desired information.

[0120] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring or implying that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing can be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0121] Accordingly, particular embodiments of the subject matter have been described. Other embodiments within the scope of the following claims can be apparent to those of ordinary skill in the art. Under the heading of "A Method of" one or more steps can be performed in different order and still achieve desirable results. Also, the processes depicted in the accompanying figures need not be performed in the particular order described or in sequential order, to achieve desirable results. In certain implementations, multitasking and parallel processing can be advantageous.

[0122] The foregoing description of one or more implementations will be better understood with reference to the accompanying drawings, which are used to provide a description of the various implementations. The description is made with reference to the accompanying drawings, in which:

Claims

1. A method for processing point cloud data, characterized in that, The method comprises: acquiring point cloud data of a plurality of road objects collected by a radar and at least one image collected by at least one image collection device, the road objects in the point cloud data at least partially coinciding with the road objects contained in each image; determining a target road object corresponding to a grid by projecting each point in the point cloud data of the road object in the grid of each image, wherein the grid comprises at least one pixel, and in the case where the grid corresponds to at least two road objects, determining a target road object closest to the image collection device from the at least two road objects according to the point cloud data; for each point in the point cloud data of the road object, acquiring a target image in which the target road object corresponding to the grid in which the projection point of the point is located coincides with the road object to which the point belongs, and determining the pixel category corresponding to the projection point in the target image as the category of the point; determining the category of the road object according to the category of each point in the point cloud data of the plurality of road objects.

2. The method of claim 1, wherein, The method comprises: determining the projection point coordinate of the point on the image according to the pose relationship between the radar and the image collection device; determining the grid to which the projection point coordinate belongs according to the projection point coordinate; determining the target road object corresponding to the grid according to the road object to which the point belongs in the point cloud data.

3. The method of claim 2, wherein, The method comprises: dividing the projection point coordinate by a preset scale and performing integer processing; acquiring a target grid coordinate matching the projection point coordinate after the integer processing from a plurality of grid coordinates, wherein the grid coordinate is determined by dividing the pixel coordinate in the image by the preset scale and performing integer processing; determining the grid to which the projection point coordinate belongs according to the target grid coordinate.

4. The method of claim 1, wherein, After determining the target road object corresponding to the grid by projecting each point in the point cloud data of the road object in the grid of each image, the method further comprises: establishing a correspondence table of the images, wherein the correspondence table is used to indicate the correspondence between the grid and the target road object; The method comprises: determining the target road object corresponding to the grid in which the projection point of the point is located according to the correspondence table of each image; determining the image in which the target road object corresponding to the point in the at least one image coincides with the road object to which the point belongs as the target image corresponding to the point.

5. The method of claim 1, wherein, The road objects in the point cloud data are provided with a first road object ID; After determining the target road object corresponding to the grid by projecting each point in the point cloud data of the road object in the grid of each image, the method further comprises: According to the target road object corresponding to the grid, a second road object ID corresponding to the grid is determined; The target image in which the target road object corresponding to the grid where the projection point of the point is located is consistent with the road object to which the point belongs is obtained for each point in the point cloud data of the road object in each image, including: The target image in which the second road object ID corresponding to the grid where the projection point of the point is located is consistent with the first road object ID to which the point belongs is obtained. The image in which the second road object ID is consistent with the first road object ID to which the point belongs is determined as the target image corresponding to the point.

6. The method of claim 1, wherein, The method further comprises: The image is subjected to semantic recognition to determine the pixel category of each pixel in the image.

7. A point cloud data processing apparatus, characterized by comprising: The device comprises: An acquisition unit is configured to acquire point cloud data of a plurality of road objects collected by a radar and at least one image collected by at least one image collection device, wherein the road objects in the point cloud data at least partially coincide with the road objects contained in each image; A grid determination unit is configured to determine a target road object corresponding to a grid by projecting each point in the point cloud data of the road objects in each image, wherein the grid comprises at least one pixel, and in the case where the grid corresponds to at least two road objects, a target road object closest to the image collection device is determined from the at least two road objects according to the point cloud data; A target image determination unit is configured to, for each point in the point cloud data of the road objects, obtain a target image in which the target road object corresponding to the grid where the projection point of the point is located is consistent with the road object to which the point belongs in each image, and determine the pixel category corresponding to the projection point in the target image as the category of the point; A category determination unit is configured to determine the category of the road objects according to the category of each point in the point cloud data of the plurality of road objects.

8. The apparatus of claim 7, wherein, The grid determination unit is configured to: Determine the projection point coordinates of the point projected on the image according to the pose relationship between the radar and the image collection device; Determine the grid to which the projection point coordinates belong according to the projection point coordinates; Determine the target road object corresponding to the grid according to the road object to which the point belongs in the point cloud data.

9. An electronic device, comprising: The device comprises: A processor; A memory for storing processor-executable instructions to execute the method of any one of claims 1 to 6.

10. A computer readable storage medium having stored thereon computer program instructions, wherein, The computer program instructions are executed by the processor to implement the method of any one of claims 1 to 6. The computer program instructions are executed by the processor to implement the method of any one of claims 1 to 6.

Images