Point cloud data optimization method, device, equipment and autonomous vehicle

Abstract

The disclosure provides an optimization method and device of point cloud data, equipment, a storage medium and a vehicle, relates to the technical field of image processing, and particularly relates to the technical field of automatic driving, computer vision, autonomous parking, cloud computing and deep learning. The specific implementation scheme is as follows: according to first semantic labels of each pixel point of a first image, determining that a plurality of first laser points of point cloud data are projected to an initial semantic label corresponding to the first image; determining that a first obstacle region of the first image contains a first clustering cluster, wherein the first clustering cluster includes a second laser point, and the second laser point belongs to the plurality of first laser points; and updating the initial semantic labels of the plurality of first laser points by using laser point information of the second laser point to obtain optimized point cloud data. According to the disclosure, the point cloud data can obtain accurate semantic labels, for example, rain and fog noise in the projection process of the point cloud data, missing of long-distance point cloud and the like can be solved, and the robustness of obstacle segmentation is improved.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and more particularly to the fields of autonomous driving, computer vision, autonomous parking, cloud computing, and deep learning technology. Background Technology

[0002] When autonomous vehicles are operating autonomously, they need to accurately understand the surrounding environment in real time, especially to accurately identify obstacles on the road, in order to ensure that autonomous vehicles can drive safely and stably. Summary of the Invention

[0003] This disclosure provides a method, apparatus, device, storage medium, and vehicle for optimizing point cloud data.

[0004] According to one aspect of this disclosure, a method for optimizing point cloud data is provided, comprising:

[0005] Based on the first semantic label of each pixel in the first image, determine the projection of multiple first laser points of the point cloud data onto the initial semantic label corresponding to the first image;

[0006] Determine a first cluster comprising a first obstacle region in the first image, wherein the first cluster includes a second laser point, and the second laser point belongs to a plurality of first laser points; and

[0007] Using the laser point information of the second laser point, the initial semantic labels of multiple first laser points are updated to obtain optimized point cloud data.

[0008] According to another aspect of this disclosure, an optimization apparatus for point cloud data is provided, comprising:

[0009] The first determining module is used to determine the initial semantic labels corresponding to the first image by projecting multiple first laser points of the point cloud data onto the first image based on the first semantic labels of each pixel of the first image.

[0010] The second determining module is used to determine a first cluster contained in the first obstacle region of the first image, wherein the first cluster includes second laser points, and the second laser points belong to multiple first laser points; and

[0011] The optimization module is used to update the initial semantic labels of multiple first laser points using the laser point information of the second laser point, so as to obtain optimized point cloud data.

[0012] According to another aspect of this disclosure, an autonomous vehicle is provided, comprising:

[0013] Sensors are used to collect point cloud data;

[0014] Image acquisition device, used to acquire the first image;

[0015] The computing unit, connected to the sensor and the image acquisition device, is used to execute any of the methods in the embodiments of this disclosure.

[0016] According to another aspect of this disclosure, an electronic device is provided, comprising:

[0017] At least one processor; and

[0018] The memory is communicatively connected to the at least one processor; wherein,

[0019] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform any of the methods described in the present disclosure.

[0020] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform any of the methods according to embodiments of this disclosure.

[0021] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements any of the methods according to embodiments of this disclosure.

[0022] According to the technology disclosed herein, point cloud data can be made to obtain accurate semantic labels.

[0023] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0024] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0025] Figure 1 This is a schematic diagram of a point cloud data optimization method according to an embodiment of the present disclosure;

[0026] Figure 2 This is a schematic diagram of a first image of a point cloud data optimization method according to an embodiment of the present disclosure;

[0027] Figure 3 This is a schematic diagram of the projection of point cloud data according to the point cloud data optimization method according to an embodiment of the present disclosure;

[0028] Figure 4 This is a schematic diagram of the projection of point cloud data according to the point cloud data optimization method according to an embodiment of the present disclosure;

[0029] Figure 5This is a schematic diagram of a point cloud data optimization method according to an embodiment of the present disclosure;

[0030] Figure 6 This is a schematic diagram of an autonomous vehicle according to an embodiment of the present disclosure;

[0031] Figure 7 This is a schematic diagram of a point cloud data optimization apparatus according to an embodiment of the present disclosure;

[0032] Figure 8 This is a block diagram of an electronic device used to implement the point cloud data optimization method of the embodiments of this disclosure. Detailed Implementation

[0033] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0034] like Figure 1 As shown, this disclosure provides a method for optimizing point cloud data, which can be applied to autonomous vehicles, including:

[0035] Step S101: Based on the first semantic label of each pixel in the first image, determine the initial semantic label corresponding to the first image by projecting multiple first laser points of the point cloud data onto the first image.

[0036] Step S102: Determine the first cluster contained in the first obstacle region of the first image, wherein the first cluster includes a second laser point, and the second laser point belongs to multiple first laser points.

[0037] Step S103: Use the laser point information of the second laser point to update the initial semantic labels of multiple first laser points to obtain optimized point cloud data.

[0038] According to the embodiments of this disclosure, it should be noted that:

[0039] The first image can be understood as an image captured by an image acquisition device (e.g., a monocular camera, a binocular camera, or a sensor-based camera). The first semantic label of each pixel in the first image can be identified and determined using semantic segmentation techniques. Based on the first semantic label of each pixel, it is possible to determine what types of obstacles are present in the first image and which pixels correspond to each obstacle. This allows the location of the obstacles to be obtained.

[0040] Point cloud data can be understood as data collected by sensors (e.g., LiDAR). The timing of sensor acquisition of point cloud data is roughly the same as the timing of image acquisition device acquisition of the first image, in order to ensure the consistency of the acquired external environmental information and improve the accuracy of assigning semantic labels to the LiDAR points in the point cloud data.

[0041] Determining the initial semantic labels corresponding to the projection of multiple first laser points from the point cloud data onto the first image can be understood as, after the multiple first laser points are projected onto the first image (e.g., Figure 3 , Figure 4 As shown, the first semantic label of each pixel in the first image covered by the first laser point is assigned to the first laser point as its initial semantic label. Based on the initial semantic labels of each first laser point, three-dimensional point cloud data containing semantic information can be obtained. The method of projecting the first laser points of the point cloud data onto the first image can adopt any method of projecting point cloud data onto a planar image in the prior art, and is not specifically limited here. For example, a rotation and translation matrix can be generated using the pose relationship of the sensor, so that multiple first laser points can be projected into the first image through coordinate transformation.

[0042] The first obstacle region can be understood as the region corresponding to the connected component formed by the pixels of each obstacle identified in the first image; that is, the region enclosed by the outer edge contour of each obstacle in the first image. For example, as Figure 2 As shown, cars, signs, and roadblocks each correspond to a first obstacle region. Based on the first semantic label of each pixel, the obstacle category of the first obstacle region can be determined, and each obstacle region can have a unique identification document (ID). Depending on the captured image, the first image may include one or more obstacles, that is, it may include one or more first obstacle regions. When multiple first obstacle regions are included, each first obstacle region can be applied to the method of this disclosure embodiment.

[0043] The first cluster can be understood as the set of laser points that fall within the first obstacle region after the point cloud data is projected onto the first image. For ease of distinction and description, each first laser point falling into the first obstacle region is named a second laser point.

[0044] Laser point information can be understood as any information in the point cloud data collected by the sensor. For example, the coordinates of the first laser point, the height of the first laser point, the relative distance to the first laser point, the laser point reflection length of the first laser point, etc., without specific limitations here.

[0045] Updating the initial semantic labels of multiple first laser points can be understood as updating the initial semantic labels of at least some of the multiple first laser points.

[0046] The technology according to the embodiments of this disclosure can effectively solve the problem of projection of point cloud data onto images. It enables point cloud data to obtain accurate semantic labels, thereby avoiding the misidentification or underidentification of obstacles in the surrounding environment by autonomous vehicles based on erroneous semantic labels, which could affect driving safety. This invention addresses the semantic segmentation requirements of autonomous vehicles for obstacles by proposing a semantic segmentation projection optimization method based on image acquisition devices (cameras) and sensors (LiDAR). It specifically solves the problem of point cloud data being assigned erroneous semantic labels during projection, improves the robustness of obstacle segmentation, and achieves high projection accuracy and stability.

[0047] In one embodiment, the point cloud data optimization method of this disclosure includes steps S101 to S103, and further includes:

[0048] Using a semantic segmentation network, the first semantic label of each pixel in the first image is determined.

[0049] According to the embodiments of this disclosure, it should be noted that:

[0050] The semantic segmentation network can adopt any network model result that can achieve semantic segmentation technology in the existing technology, and there is no specific limitation here. For example, HRNet (High-Resolution Net) and FCNs (Fully Convolutional Networks) can be used.

[0051] The first semantic label can be understood as an obstacle label. A semantic segmentation network is used to determine whether each pixel is an obstacle. Based on the first semantic label of each pixel, it is possible to determine which image regions in the first image correspond to obstacles, and the specific type of obstacle.

[0052] The method of this disclosure can be executed in parallel by multiple processes of a GPU (graphics processing unit), thereby improving computational real-time performance and efficiency to meet the application requirements of autonomous vehicles. For example, a first image is segmented into multiple image blocks, which are then processed in parallel by multiple processes of the GPU using a semantic segmentation network, thereby enabling faster semantic recognition of each pixel in the first image.

[0053] According to the technology of the embodiments of this disclosure, by using a semantic segmentation network, the first semantic label of each pixel of the first image can be obtained quickly and accurately.

[0054] In one embodiment, the point cloud data optimization method of this disclosure includes steps S101 to S103, and further includes:

[0055] Based on the first semantic label of each pixel in the first image, multiple pixels belonging to the same obstacle are identified.

[0056] Generate an identification number for each obstacle and assign this identification number to the multiple pixels that make up the obstacle. Implement an association index between pixels and obstacles. Each obstacle has its own unique identification number.

[0057] According to the embodiments of this disclosure, it should be noted that:

[0058] The obstacle, the first obstacle region corresponding to the obstacle, each pixel constituting the first obstacle region, and the first point cloud cluster corresponding to the first obstacle region all have the same identification number.

[0059] The first image can be understood as an obstacle ID map, that is, the first obstacle area of ​​each obstacle is displayed in the first image, and the obstacle type and identification number of each first obstacle area are marked.

[0060] In one embodiment, the point cloud data optimization method of this disclosure includes steps S101 to S103, wherein step S101: determining the initial semantic labels corresponding to the first image by projecting a plurality of first laser points of the point cloud data onto the first image based on the first semantic labels of each pixel of the first image, including:

[0061] Project multiple first laser points from the point cloud data onto the first image.

[0062] Based on the first semantic label of each pixel in the first image, the initial semantic labels of multiple first laser points are determined.

[0063] According to the embodiments of this disclosure, it should be noted that:

[0064] The method of projecting the first laser point of the point cloud data onto the first image can be any of the existing methods for projecting point cloud data onto a planar image, and is not specifically limited here. For example, a rotation and translation matrix can be generated using the pose relationship of the sensor, so that multiple first laser points can be projected into the first image through coordinate transformation.

[0065] Determining the initial semantic labels for multiple first laser points can be understood as follows: after the multiple first laser points are projected onto the first image, the first semantic label of each pixel covered by the first laser point is assigned to the first laser point as its initial semantic label. Based on the initial semantic labels of each first laser point, 3D point cloud data containing semantic information can be obtained.

[0066] The method of this disclosure can be executed in parallel by multiple processes of the GPU, thereby improving computational real-time performance and efficiency, and meeting the application requirements of autonomous vehicles. For example, the first image is segmented into multiple image blocks, and the multiple image blocks are processed in parallel by multiple processes of the GPU using a semantic segmentation network, thereby more quickly assigning initial semantic labels to each first laser point projected onto the first image.

[0067] According to the technology of the embodiments of this disclosure, each first laser point of point cloud data can be assigned a semantic label, thereby obtaining three-dimensional point cloud data containing semantic information.

[0068] Because the wavelength of the sensors collecting point cloud data is between infrared and visible light, and is in the nanometer range, water mist in the autonomous driving environment (such as water mist from sprinkler trucks or vehicle exhaust in winter) will form suspended point cloud clusters in the sensor's field of view, thus being collected as noise laser points as point cloud data. However, the image acquisition device uses a longer wavelength of visible light, allowing it to penetrate the water mist and see obstacles behind it. Therefore, when the point cloud data is projected onto the first image, noise laser points formed by the water mist will be projected onto the obstacles and assigned initial semantic labels. If these noise laser points, already assigned obstacle semantics, are directly sent to the autonomous vehicle as point cloud data, the vehicle will perceive a sudden obstacle ahead and brake sharply, leading to false obstacle detection.

[0069] To address the aforementioned problems, in one embodiment, the point cloud data optimization method of this disclosure includes steps S101 to S103, wherein step S103: using the laser point information of the second laser point to update the initial semantic labels of multiple first laser points to obtain optimized point cloud data, including:

[0070] Step S1031: Using the sliding window algorithm and the first feature index, determine the first laser point subset from the multiple second laser points of the first cluster, wherein the spatial positions of the multiple second laser points of the first laser point subset are continuous.

[0071] Step S1032: Based on the second feature index and the laser point information of multiple second laser points in the first laser point subset, determine the second semantic labels of multiple second laser points in the first laser point subset, wherein the second semantic labels include water mist noise semantic labels and obstacle semantic labels.

[0072] Step S1033: Based on the second semantic label, update the initial semantic label of the first laser point corresponding to multiple second laser points of the first laser point subset to obtain optimized point cloud data.

[0073] According to the embodiments of this disclosure, it should be noted that:

[0074] In the case where the first image includes multiple first obstacle regions, the method of the present disclosure embodiment can be applied to the first cluster corresponding to each first obstacle region.

[0075] The first characteristic indicator can be understood as a key characteristic indicator describing the spatial distribution of laser points corresponding to obstacles. It can be used to initially determine whether a laser point is water mist. The specific first characteristic indicator used can be selected and adjusted as needed. For example, the floating motion and distribution of water mist in the air are usually different from those of static obstacles. Therefore, characteristic indicators such as, but not limited to, the number of laser points in clusters, the height range threshold of laser points acting as obstacles, and the distance threshold to the laser points on the main vehicle can be used as judgment indicators.

[0076] The sliding window algorithm is used to determine whether the beginning and end of each second laser point are continuous based on the position information of the laser point, thereby identifying multiple laser points with continuous spatial positions from the multiple second laser points of the first cluster.

[0077] Based on the first feature index and the sliding window algorithm, laser points that may correspond to the obstacles can be initially screened from the first cluster corresponding to the first obstacle area, i.e., the first subset of laser points.

[0078] The second characteristic indicator can be understood as a key characteristic indicator describing the spatial distribution of laser points corresponding to obstacles. This indicator can accurately determine whether a laser point is water mist. The specific second characteristic indicator used can be selected and adjusted as needed. For example, the floating motion and distribution of water mist in the air are usually different from those of static obstacles. Therefore, characteristic indicators such as, but not limited to, the number of laser points in clusters, the height range threshold of laser points acting as obstacles, the distance threshold to the laser point on the main vehicle, and the reflection length threshold of the laser point can be used as judgment indicators.

[0079] The laser point information for the second laser point can be obtained from the point cloud data collected by the sensor. The specific laser point information used is selected based on the second feature index. For example, when the second feature index is a threshold for the height range of the laser point, the laser point information is the laser point height information. When the second feature index is a threshold for the distance of the laser point to the main vehicle, the laser point information is the laser point distance information.

[0080] Based on the second feature index and the laser point information of the second laser point, it is possible to accurately determine which second laser points in the first laser point subset are truly collected based on obstacles, and which are water mist noise laser points collected due to misidentification of water mist in the environment. If, based on the second feature index and the laser point information of the second laser point, the second laser point is determined to be water mist noise and not an obstacle, then the initial semantic label of its corresponding first laser point is updated from an obstacle label to a water mist noise label (or a non-obstacle label).

[0081] According to the technology of the present disclosure embodiments, water mist point cloud laser points that are mistakenly marked as obstacles can be accurately identified, improving the accuracy of obtaining semantic labels by projecting point cloud data onto the first image. This effectively solves the problem of semantic label errors in point cloud data, which leads to false obstacle reports caused by autonomous vehicles judging water mist as obstacles.

[0082] In one example, when an autonomous vehicle is traveling in the lane of its primary direction of travel, if a water truck passes by in the adjacent lane traveling in the opposite direction, water mist will drift around the autonomous vehicle. The sensors will then collect this water mist as point cloud data. For example, as... Figure 3 As shown in the diagram, the laser point in the circled area represents water mist noise, not an obstacle.

[0083] In one embodiment, the point cloud data optimization method of this disclosure includes steps S101 to S103, and steps S1031 to S1033, and further includes:

[0084] Step S1034: Determine the first feature index and the second feature index based on the distance between the first obstacle area and the target object. The target object is the object on which sensors are deployed to collect point cloud data.

[0085] According to the embodiments of this disclosure, it should be noted that:

[0086] The target object can be understood as the autonomous vehicle itself, which acts as the master vehicle.

[0087] The distance between the first obstacle area and the target object can be understood as the distance between the obstacle and the main vehicle. This distance information can be directly obtained through point cloud data and information such as the sensor's position in the vehicle coordinate system.

[0088] Sensors that collect point cloud data exhibit a pattern of denser data acquisition at closer distances and sparser data acquisition at farther distances. Therefore, it is necessary to select a first and second feature index based on the distance between the obstacle and the vehicle to make judgments. For example, the first feature index is significantly different when the obstacle is 2 meters away from the vehicle compared to when it is 10 meters away; the same applies to the second feature index.

[0089] According to the technology of the embodiments of this disclosure, based on the distance between the first obstacle area and the target object, a first feature index and a second feature index that better meet the judgment requirements can be selected in a targeted manner.

[0090] In one embodiment, the point cloud data optimization method of this disclosure includes steps S101 to S103, and steps S1031 to S1033, wherein step 1031: using a sliding window algorithm and a first feature index, determining a subset of first laser points from a plurality of second laser points in a first cluster, including:

[0091] Step 10311: Based on the laser point information of multiple second laser points in the first cluster, sort the multiple second laser points in the first cluster according to their distance to the target object. The target object is the object on which the sensors collecting point cloud data are deployed.

[0092] Step 10312: Using the sliding window algorithm, determine a subset of second laser points from the sorted set of second laser points. The spatial positions of the multiple second laser points within this subset are continuous.

[0093] Step 10313: Determine the first laser point subset from the second laser point subset based on the laser point information of multiple second laser points within the first feature index and the second laser point subset.

[0094] According to the embodiments of this disclosure, it should be noted that:

[0095] Laser point information can be understood as the distance from the second laser point to the target object (main vehicle).

[0096] Sort the multiple second laser points in the first cluster according to their distance from the target object. This can be understood as sorting the multiple second laser points from closest to furthest from the main vehicle, or from furthest to closest. Through this sorting operation, the sliding window algorithm can be used to find spatially continuous second laser points more quickly.

[0097] The second laser point subset can be understood as multiple second laser points with consecutive spatial positions initially selected using the sliding window algorithm. Since water mist is usually discretely distributed, if the spatial positions are consecutive, it can be preliminarily determined that the second laser points in the second laser point subset are likely laser points corresponding to obstacles, rather than water mist noise laser points.

[0098] The first subset of laser points can be understood as a further judgment of the second subset of laser points based on the first feature index, and the selected points may correspond to the second laser points of obstacles. Since the spatial distribution of water mist and obstacles is quite different, some second laser points that do not meet the requirements can be screened out using the first feature index.

[0099] According to the technology of the embodiments of this disclosure, water mist noise laser points that may be mislabeled as obstacles can be initially screened out.

[0100] In one embodiment, the point cloud data optimization method of this disclosure includes steps S101 to S103, steps S1031 to S1033, and steps S10311 to S10313, wherein step S10313: determining a first laser point subset from the second laser point subset based on the laser point information of multiple second laser points within the second laser point subset, including:

[0101] Determine the laser point height information of multiple second laser points within the second laser point subset.

[0102] Determine the height range threshold for the first feature indicator.

[0103] Based on the height range threshold and the laser point height information, the first laser point subset is determined from the second laser point subset.

[0104] According to the embodiments of this disclosure, it should be noted that:

[0105] Based on the height range threshold and the laser point height information, the first laser point subset is determined from the second laser point subset. This can be understood as follows: if the value corresponding to the laser point height information is within the height range threshold, it means that the second laser point corresponding to the laser point height information meets the requirements. It may be the laser point corresponding to the obstacle, rather than the laser point corresponding to the water mist noise.

[0106] The height information of the laser point can be directly obtained from the laser point information of the first laser point corresponding to the second laser point. When the sensor collects point cloud data, the height information of each first laser point can be directly obtained.

[0107] According to the technology of the present disclosure embodiments, by utilizing laser point height information and height range threshold, water mist noise laser points that may be mislabeled as obstacles can be initially screened out.

[0108] In one embodiment, the point cloud data optimization method of this disclosure includes steps S101 to S103, steps S1031 to S1033, and steps S10311 to S10313, wherein step S10313: determining a first laser point subset from the second laser point subset based on the laser point information of multiple second laser points within the second laser point subset, including:

[0109] Determine the laser point distance information of multiple second laser points within the second laser point subset.

[0110] Determine the distance threshold for the first feature index.

[0111] Based on the distance threshold and the distance information of the laser points, the first laser point subset is determined from the second laser point subset.

[0112] According to the embodiments of this disclosure, it should be noted that:

[0113] Based on the distance threshold and laser point distance information, the first laser point subset is determined from the second laser point subset. This can be understood as follows: if the value corresponding to the laser point distance information meets the distance threshold, it means that the second laser point corresponding to that distance information meets the requirements and is likely the laser point corresponding to an obstacle, rather than the laser point corresponding to water mist noise. Because the distances between water mist and static obstacles and the main vehicle are different, it is possible to initially determine that the laser point is likely to be water mist.

[0114] The distance information of the laser points can be directly obtained from the laser point information of the first laser point corresponding to the second laser point. When the sensor collects point cloud data, the distance information of each first laser point can be directly obtained.

[0115] According to the technology of the present disclosure embodiments, by utilizing distance thresholds and laser point distance information, water mist noise laser points that may be mislabeled as obstacles can be initially screened out.

[0116] In one embodiment, the point cloud data optimization method of this disclosure includes steps S101 to S103, steps S1031 to S1033, and steps S10311 to S10313, wherein step S10313: determining a first laser point subset from the second laser point subset based on the laser point information of multiple second laser points within the second laser point subset, including:

[0117] Determine the laser point height information and laser point distance information of multiple second laser points within the second laser point subset.

[0118] Determine the height range threshold and distance threshold for the first feature index.

[0119] Based on the distance threshold, height range threshold, laser point distance information, and laser point height information, the first laser point subset is determined from the second laser point subset.

[0120] According to the embodiments of this disclosure, it should be noted that:

[0121] Based on the height range threshold and the laser point height information, the first laser point subset is determined from the second laser point subset. This can be understood as follows: if the value corresponding to the laser point height information is within the height range threshold, it means that the second laser point corresponding to the laser point height information meets the requirements. It may be the laser point corresponding to the obstacle, rather than the laser point corresponding to the water mist noise.

[0122] After filtering based on the height range threshold and laser point height information, for the second laser point that meets the height range threshold requirement, further judgment is made based on the distance threshold and laser point distance information. Specifically, if the value corresponding to the laser point distance information meets the distance threshold, it means that the second laser point corresponding to that laser point distance information meets the requirements, and it is likely the laser point corresponding to the obstacle, rather than the laser point corresponding to the water mist noise.

[0123] According to the technology of this disclosure, by combining feature indicators of different dimensions, water mist noise laser points that may be mislabeled as obstacles can be initially screened out.

[0124] In one embodiment, the point cloud data optimization method of this disclosure includes steps S101 to S103, and steps S1031 to S1033, wherein step S1032 includes: determining a second semantic label for the multiple second laser points of the first laser point subset based on a second feature index and laser point information of multiple second laser points of a first laser point subset, including:

[0125] Determine the laser point height information, laser point distance information, and number of second laser points within the first laser point subset for multiple second laser points.

[0126] Determine the height range threshold, distance threshold, and quantity threshold for the second feature indicator.

[0127] Based on the laser point height information, laser point distance information, quantity information, height range threshold, distance threshold, and quantity threshold, the second semantic labels of multiple second laser points in the first laser point subset are determined.

[0128] According to the embodiments of this disclosure, it should be noted that:

[0129] The quantity threshold can be understood as the theoretical number of laser points projected on the obstacle when the first obstacle area is a certain distance away from the target object (main vehicle).

[0130] If the laser point height information of the second laser point in the first laser point subset meets the height range threshold, the laser point distance information of the second laser point meets the distance threshold, and the number information of the second laser points within the first laser point subset meets the number threshold, then it indicates that the second laser point is a laser point corresponding to an obstacle, meaning that the second semantic label of the second laser point should be an obstacle semantic label. Conversely, if none of these conditions are met, then it indicates that the second laser point is a laser point corresponding to water mist noise, meaning that the second semantic label of the second laser point should be a water mist noise semantic label (or a non-obstacle semantic label).

[0131] According to the technology of this disclosure, by combining feature indicators of different dimensions, it is possible to accurately determine the first laser point in the point cloud data that was mistakenly marked as an obstacle but is actually water mist noise.

[0132] In one embodiment, the point cloud data optimization method of this disclosure includes steps S101 to S103, and steps S1031 to S1033, wherein step S1032 includes: determining a second semantic label for the multiple second laser points of the first laser point subset based on a second feature index and laser point information of multiple second laser points of a first laser point subset, including:

[0133] Determine the laser point height information, laser point distance information, laser point reflection length information, and the number of second laser points within the first laser point subset for multiple second laser points.

[0134] Determine the height range threshold, distance threshold, quantity threshold, and reflection length threshold for the second feature index.

[0135] Based on the laser point height information, laser point distance information, laser point reflection length information, quantity information, height range threshold, distance threshold, quantity threshold, and reflection length threshold, the second semantic labels of multiple second laser points in the first laser point subset are determined.

[0136] According to the embodiments of this disclosure, it should be noted that:

[0137] The quantity threshold can be understood as the theoretical number of laser points projected on the obstacle when the first obstacle area is a certain distance away from the target object (main vehicle).

[0138] The laser point reflection length information can be directly obtained from the laser point information of the first laser point corresponding to the second laser point.

[0139] If the laser point height information of the second laser point in the first laser point subset meets the height range threshold, the laser point distance information of the second laser point meets the distance threshold, the number information of the second laser points within the first laser point subset meets the number threshold, and the laser point reflection length information of the second laser point meets the reflection length threshold, then it is indicated that the second laser point is the laser point corresponding to the obstacle, that is, the second semantic label of the second laser point should be the obstacle semantic label. Conversely, it is indicated that the second laser point is the laser point corresponding to water mist noise, that is, the second semantic label of the second laser point should be the water mist noise semantic label (or the non-obstacle semantic label).

[0140] According to the technology of this disclosure, by combining feature indicators of different dimensions, it is possible to accurately determine the first laser point in the point cloud data that was mistakenly marked as an obstacle but is actually water mist noise.

[0141] In one embodiment, the point cloud data optimization method of this disclosure includes steps S101 to S103, and steps S1031 to S1033, wherein step S1032 includes: determining a second semantic label for the multiple second laser points of the first laser point subset based on a second feature index and laser point information of multiple second laser points of a first laser point subset, including:

[0142] Determine the laser point height information, laser point distance information, and laser point reflection length information of multiple second laser points in the first laser point subset.

[0143] Determine the height range threshold, distance threshold, and reflection length threshold for the second feature index.

[0144] Based on the laser point height information, laser point distance information, laser point reflection length information, height range threshold, distance threshold, and reflection length threshold, the second semantic labels of multiple second laser points in the first laser point subset are determined.

[0145] According to the embodiments of this disclosure, it should be noted that:

[0146] The quantity threshold can be understood as the theoretical number of laser points projected on the obstacle when the first obstacle area is a certain distance away from the target object (main vehicle).

[0147] The laser point reflection length information can be directly obtained from the laser point information of the first laser point corresponding to the second laser point.

[0148] If the laser point height information of the second laser point in the first laser point subset meets the height range threshold, the laser point distance information of the second laser point meets the distance threshold, and the laser point reflection length information of the second laser point meets the reflection length threshold, then it is indicated that the second laser point is the laser point corresponding to the obstacle, that is, the second semantic label of the second laser point should be the obstacle semantic label. Conversely, it is indicated that the second laser point is the laser point corresponding to water mist noise, that is, the second semantic label of the second laser point should be the water mist noise semantic label (or the non-obstacle semantic label).

[0149] According to the technology of this disclosure, by combining feature indicators of different dimensions, it is possible to accurately determine the first laser point in the point cloud data that was mistakenly marked as an obstacle but is actually water mist noise.

[0150] Because sensors collecting point cloud data exhibit a "dense near, sparse far" characteristic in their laser point reflections, distant obstacles typically reflect very few laser points, resulting in a lack of distant point cloud (laser point) data. Furthermore, if projection errors caused by sensor calibration are added on top of this, it becomes difficult to obtain stable projection results for distant obstacles. This leads to laser points that should be assigned semantic labels to obstacles being missed, causing the vehicle to detect the presence of distant obstacles but failing to plan a safe driving trajectory in a timely manner.

[0151] To address the aforementioned problems, in one embodiment, the point cloud data optimization method of this disclosure includes steps S101 to S103, wherein step S103: using the laser point information of the second laser point to update the initial semantic labels of multiple first laser points to obtain optimized point cloud data, including:

[0152] Step S1035: Based on the third semantic labels of each pixel in the second image, determine the fourth semantic labels corresponding to the projection of multiple first laser points onto the second image. The second image is generated from the first image through image dilation processing.

[0153] Step S1036: If the laser point information of the second laser point determines that the first obstacle area is a distant obstacle, determine the second obstacle area associated with the first obstacle area from the second image.

[0154] Step S1037: Determine the second cluster contained in the associated second obstacle region, wherein the second cluster includes a third laser point, and the third laser point belongs to multiple first laser points.

[0155] Step S1038: Update the initial semantic label of the first laser point corresponding to the third laser point according to the fourth semantic label of the third laser point to obtain optimized point cloud data.

[0156] According to the embodiments of this disclosure, it should be noted that:

[0157] The second image is generated from the first image through image dilation. It can be understood as dilating each of the first obstacle regions in the first image to obtain the second image. Each second obstacle region in the second image corresponds one-to-one with each of the first obstacle regions in the first image. That is, each second obstacle region is generated from its corresponding first obstacle region through image dilation. Both have the same identifier. This identifier allows for the indexing of the first obstacle region and its corresponding second obstacle region. The difference is that the image size (total number of pixels) of the second obstacle region is larger than that of the first obstacle region. Through image dilation, the semantic labels of peripheral pixels that did not originally belong to the first obstacle region can also be identified as obstacle semantic labels.

[0158] Through image dilation processing, a laser point that was originally projected outside the obstacle (first obstacle area) corresponding to ID number A in the first image may be projected inside the obstacle (second obstacle area) corresponding to ID number A in the second image. For example, as... Figure 4 As shown, the left side is the first image and the right side is the second image. The box in the first image represents the first obstacle area, and the box in the second image represents the second obstacle area. It can be clearly seen that the second obstacle area contains more laser points.

[0159] The second obstacle region can be understood as the region corresponding to the connected block formed by each pixel of each obstacle identified in the second image, that is, the region enclosed by the outer edge contour of each obstacle in the second image.

[0160] The third semantic labels of each pixel in the second image can be identified and determined using semantic segmentation techniques. Based on these labels, the types of obstacles in the second image can be determined, along with the corresponding pixels for each obstacle. This allows the location of the obstacles to be determined.

[0161] A fourth semantic label is determined corresponding to the projection of multiple first laser points onto the second image. After the multiple first laser points are projected onto the second image, the third semantic label of each pixel in the second image covered by the first laser point is assigned to the first laser point as its fourth semantic label. The method of projecting the first laser points of the point cloud data onto the second image can adopt any method of projecting point cloud data onto a planar image in the prior art, and is not specifically limited here. For example, a rotation and translation matrix can be generated using the pose relationship of the sensor, so that multiple first laser points can be projected into the second image through coordinate transformation.

[0162] It should be noted that the semantic labels of the first laser point projected onto the same coordinate position in the first and second images may differ. This is because the second image is obtained through image dilation. The first laser point, which originally did not fall within the obstacle image region in the first image, has a non-obstacle semantic label (initial semantic label). However, when projected onto the second image, if it falls within the obstacle image region, its semantic label changes to an obstacle semantic label (fourth semantic label). In this case, the same first laser point possesses different semantic labels.

[0163] A long-distance obstacle can be understood as a first obstacle area being more than a threshold distance from the target object (the main vehicle).

[0164] Determining the first obstacle area as a distant obstacle using the laser point information of the second laser point can be understood as calculating the average distance from the second laser point in the first obstacle area to the target object based on the laser point distance information of each second laser point in the first obstacle area (distance from the second laser point to the target object). If the average distance is greater than a threshold distance, it indicates that the first obstacle area is far from the target object, and the obstacle corresponding to the first obstacle area is a distant obstacle.

[0165] The second obstacle region associated with the first obstacle region can be understood as having the same identification number, meaning that the second obstacle region is obtained by image dilation processing of the first obstacle region.

[0166] The second cluster can be understood as the set of laser points that fall within the second obstacle region after the point cloud data is projected onto the second image. For ease of distinction and description, each first laser point falling into the second obstacle region is named a third laser point.

[0167] Updating the initial semantic labels of multiple first laser points can be understood as assigning the fourth semantic label of the third laser point to the corresponding first laser point, so that the initial semantic label of the first laser point becomes the fourth semantic label.

[0168] According to the technology of this disclosure, for distant obstacles, the expanded second obstacle region increases the obstacle area, thereby increasing the probability that the first laser point will be projected onto the obstacle. This alleviates the projection problem caused by the sparsity of the distant point cloud (first laser point), allowing the distant laser points in the point cloud data to be assigned accurate obstacle semantic labels. This avoids incorrect semantic labels based on point cloud data, preventing autonomous vehicles from misidentifying or missing obstacles in the surrounding environment. It helps autonomous vehicles accurately perform path planning, obstacle avoidance, and other functions, improving driving safety and the robustness of distant obstacle segmentation results.

[0169] In one embodiment, the point cloud data optimization method of this disclosure includes steps S101 to S103, and steps S1035 to S1038, wherein the generation process of the second image includes:

[0170] The first obstacle region in the first image is dilated to obtain the second obstacle region. The first and second obstacle regions share the same identification number.

[0171] A second image is generated based on the second obstacle region.

[0172] According to the technology of this disclosure embodiment, by performing image dilation processing on the first obstacle area, the laser point corresponding to the actual obstacle can be determined in a targeted manner, thereby increasing the probability of the laser point hitting the obstacle after projection from a distance.

[0173] In one embodiment, the point cloud data optimization method of this disclosure includes steps S101 to S103, and steps S1035 to S1038, wherein step S1036: when the first obstacle region is determined to be a distant obstacle using the laser point information of the second laser point, determining a second obstacle region associated with the first obstacle region from the second image, including:

[0174] If the laser point information of the second laser point is used to determine that the first obstacle area is a distant obstacle, the identification number of the first obstacle area is obtained.

[0175] Based on the identification number of the first obstacle area, a second obstacle area associated with the first obstacle area is determined from the second image.

[0176] According to the technology of this disclosure embodiment, the second obstacle area corresponding to the first obstacle area can be quickly and accurately indexed by the identification number.

[0177] In one example, the execution entity of this embodiment of the disclosure can be the GPU (graphics processing unit) of an autonomous vehicle. The methods of this embodiment can be executed in parallel by multiple processes of the GPU, thereby improving computational real-time performance and efficiency, and meeting the application requirements of autonomous vehicles. For example, a first image and a second image can be segmented into multiple image blocks, which are then processed in parallel by multiple processes of the GPU.

[0178] In one embodiment, this disclosure provides a method for optimizing point cloud data, which can be applied to autonomous vehicles, including:

[0179] Based on the first semantic label of each pixel in the first image, the first laser points of the point cloud data are projected onto the initial semantic label corresponding to the first image.

[0180] A first cluster is determined to be contained within a first obstacle region of a first image, wherein the first cluster includes a second laser point, and the second laser point belongs to a plurality of first laser points.

[0181] Using a sliding window algorithm and a first feature index, a subset of first laser points is determined from multiple second laser points in a first cluster, wherein the spatial positions of multiple second laser points in the subset of first laser points are continuous.

[0182] Based on the second feature index and the laser point information of multiple second laser points in the first laser point subset, the second semantic labels of multiple second laser points in the first laser point subset are determined, wherein the second semantic labels include water mist noise semantic labels and obstacle semantic labels.

[0183] Based on the second semantic label, the initial semantic labels of the first laser points corresponding to multiple second laser points in the first laser point subset are updated to obtain optimized first point cloud data.

[0184] Based on the third semantic labels of each pixel in the second image, a fourth semantic label corresponding to the projection of multiple first laser points onto the second image is determined. The second image is generated from the first image through image dilation processing.

[0185] If the laser point information of the second laser point is used to determine that the first obstacle area is a distant obstacle, a second obstacle area associated with the first obstacle area is determined from the second image.

[0186] The associated second obstacle region is identified as containing a second cluster, wherein the second cluster includes a third laser point, which belongs to multiple first laser points.

[0187] Based on the fourth semantic label of the third laser point, the initial semantic label of the first laser point corresponding to the third laser point in the first point cloud data is updated to obtain the final optimized point cloud data.

[0188] To address the semantic segmentation requirements of autonomous vehicles for obstacles, this disclosure proposes a camera- and LiDAR-based semantic segmentation projection optimization method. This method specifically addresses issues such as rain and fog noise and missing long-distance point clouds during point cloud data projection, improving the robustness of obstacle segmentation. According to the technology of this disclosure, water mist point cloud laser points that are mistakenly labeled as obstacles can be accurately identified, improving the accuracy of obtaining semantic labels from point cloud data projection onto the first image. This effectively solves the problem of semantic label errors in point cloud data, which leads to false obstacle detection by autonomous vehicles that mistake water mist for obstacles. Furthermore, for long-distance obstacles, an expanded second obstacle region is used to increase the obstacle area, increasing the probability that the first laser point will be projected onto the obstacle. This alleviates the projection problem caused by the sparsity of the long-distance point cloud (first laser point), allowing long-distance laser points in the point cloud data to be assigned accurate obstacle semantic labels. This avoids erroneous semantic labels based on point cloud data, preventing autonomous vehicles from misidentifying or missing obstacles in the surrounding environment. It helps autonomous vehicles accurately perform functions such as path planning and obstacle avoidance, improving driving safety and the robustness of long-distance obstacle segmentation results.

[0189] In one example, such as Figure 5 As shown in the embodiments of this disclosure, a method for optimizing point cloud data is provided, including:

[0190] Acquire camera images (first image) captured by the cameras of the autonomous vehicle;

[0191] A semantic segmentation network is used to perform semantic segmentation on camera images to obtain a first image with semantic segmentation results; wherein, the semantic segmentation results include the semantic label of each pixel in the camera image, as well as the obstacle information contained therein;

[0192] Obstacle information initialization process: The first image is dilated to obtain the second image; the first and second images are processed in parallel by multiple processes of the GPU to assign an obstacle pixel number (identification code) to each obstacle pixel identified in the first and second images, and obstacle traversal is performed to determine the identification code of each obstacle; based on the identification code of each obstacle, an obstacle ID map (first image) and a dilated obstacle ID map (second image) are obtained;

[0193] Point cloud projection and clustering: Point cloud data is projected onto a first image and a second image through parallel processing of multiple processes on the GPU; initial semantic labels are assigned to the first laser points of the point cloud data projected onto the first image, and fourth semantic labels are assigned to the first laser points of the point cloud data projected onto the second image; obstacle clusters corresponding to each first obstacle region in the first image and dilated obstacle clusters corresponding to each second obstacle region in the second image are determined.

[0194] By using obstacle clustering, rain and fog denoising is performed on point cloud data to determine which initial semantic labels of the first laser points in the point cloud data are actually rain and fog, rather than obstacles;

[0195] By utilizing obstacle clustering and dilated obstacle clustering, long-distance obstacle dilation recognition is performed on point cloud data. The obstacle semantic label of the third laser point corresponding to the long-distance obstacle in the second image is assigned to the corresponding first laser point, thereby completing the optimization of point cloud data.

[0196] like Figure 6 As shown, this disclosure provides an autonomous driving vehicle 600, including:

[0197] Sensor 610 is used to collect point cloud data.

[0198] Image acquisition device 620 is used to acquire the first image.

[0199] The computing unit 630, connected to the sensor 610 and the image acquisition device 620, is used to execute the point cloud data optimization method of any embodiment of the present disclosure.

[0200] According to embodiments of this disclosure, problems such as rain and fog noise and missing long-distance point clouds are specifically addressed during point cloud data projection, improving the robustness of obstacle segmentation. The technology of this disclosure can accurately identify water mist laser points that are mistakenly labeled as obstacles, improving the accuracy of obtaining semantic labels from point cloud data projection onto the first image. This effectively solves the problem of semantic label errors in point cloud data, which could lead to false obstacle reports caused by autonomous vehicles mistaking water mist for obstacles. According to embodiments of this disclosure, for long-distance obstacles, an expanded second obstacle region is used to increase the obstacle area, increasing the probability that the first laser point will be projected onto the obstacle. This alleviates the projection problem caused by the sparseness of the long-distance point cloud (first laser point), allowing long-distance laser points in the point cloud data to be assigned accurate obstacle semantic labels. This avoids the autonomous vehicle misidentifying or missing obstacles in the surrounding environment based on erroneous semantic labels from point cloud data, helping the autonomous vehicle accurately achieve path planning, obstacle avoidance, and other functions, improving driving safety and the robustness of long-distance obstacle segmentation results.

[0201] like Figure 7 As shown in the figure, this disclosure provides a point cloud data optimization device, including:

[0202] The first determining module 710 is used to determine the initial semantic labels corresponding to the first image by projecting multiple first laser points of the point cloud data onto the first image based on the first semantic labels of each pixel of the first image.

[0203] The second determining module 720 is used to determine a first cluster contained in the first obstacle region of the first image, wherein the first cluster includes second laser points, and the second laser points belong to multiple first laser points.

[0204] The optimization module 730 is used to update the initial semantic labels of multiple first laser points using the laser point information of the second laser point, so as to obtain optimized point cloud data.

[0205] In one implementation, the optimization module 730 includes:

[0206] The first determining submodule is used to determine a subset of first laser points from multiple second laser points in a first cluster using a sliding window algorithm and a first feature index, wherein the spatial positions of the multiple second laser points in the subset of first laser points are continuous.

[0207] The second determining submodule is used to determine the second semantic labels of the multiple second laser points in the first laser point subset based on the second feature index and the laser point information of the multiple second laser points in the first laser point subset. The second semantic labels include water mist noise semantic labels and obstacle semantic labels.

[0208] The first optimization submodule is used to update the initial semantic labels of the first laser points corresponding to multiple second laser points in the first laser point subset based on the second semantic labels, so as to obtain optimized point cloud data.

[0209] In one implementation, the optimization module 730 further includes:

[0210] The third determination submodule is used to determine the first feature index and the second feature index based on the distance between the first obstacle area and the target object. The target object is the object on which sensors are deployed to collect point cloud data.

[0211] In one implementation, the first determining submodule includes:

[0212] The sorting unit is used to sort the multiple second laser points of the first cluster according to their distance from the target object, based on the laser point information of the multiple second laser points in the first cluster. The target object is the object on which the sensors collecting point cloud data are deployed.

[0213] The first determining unit is used to determine a subset of second laser points from a sorted set of second laser points using a sliding window algorithm. The spatial positions of the multiple second laser points within the subset are continuous.

[0214] The second determining unit is used to determine the first laser point subset from the second laser point subset based on the laser point information of multiple second laser points within the second laser point subset and the first feature index.

[0215] In one implementation, the second determining unit is used to:

[0216] Determine the laser point height information of multiple second laser points within the second laser point subset.

[0217] Determine the height range threshold for the first feature indicator.

[0218] Based on the height range threshold and the laser point height information, the first laser point subset is determined from the second laser point subset.

[0219] or

[0220] Determine the laser point distance information of multiple second laser points within the second laser point subset.

[0221] Determine the distance threshold for the first feature index.

[0222] Based on the distance threshold and the distance information of the laser points, the first laser point subset is determined from the second laser point subset.

[0223] In one implementation, the second determining unit is used to:

[0224] Determine the laser point height information and laser point distance information of multiple second laser points within the second laser point subset.

[0225] Determine the height range threshold and distance threshold for the first feature index.

[0226] Based on the distance threshold, height range threshold, laser point distance information, and laser point height information, the first laser point subset is determined from the second laser point subset.

[0227] In one implementation, the second determining submodule is used to:

[0228] Determine the laser point height information, laser point distance information, and number of second laser points within the first laser point subset for multiple second laser points.

[0229] Determine the height range threshold, distance threshold, and quantity threshold for the second feature indicator.

[0230] Based on the laser point height information, laser point distance information, quantity information, height range threshold, distance threshold, and quantity threshold, the second semantic labels of multiple second laser points in the first laser point subset are determined.

[0231] In one implementation, the second determining submodule is used to:

[0232] Determine the laser point height information, laser point distance information, laser point reflection length information, and the number of second laser points within the first laser point subset for multiple second laser points.

[0233] Determine the height range threshold, distance threshold, quantity threshold, and reflection length threshold for the second feature index.

[0234] Based on the laser point height information, laser point distance information, laser point reflection length information, quantity information, height range threshold, distance threshold, quantity threshold, and reflection length threshold, the second semantic labels of multiple second laser points in the first laser point subset are determined.

[0235] In one implementation, the optimization module 730 includes:

[0236] The fourth determining submodule is used to determine the fourth semantic labels corresponding to the projection of multiple first laser points onto the second image based on the third semantic labels of each pixel in the second image. The second image is generated from the first image through image dilation processing.

[0237] The fifth determination submodule is used to determine, from the second image, a second obstacle region associated with the first obstacle region when the laser point information of the second laser point is used to determine that the first obstacle region is a distant obstacle.

[0238] The sixth determining submodule is used to determine the second cluster contained in the associated second obstacle region, wherein the second cluster includes a third laser point, and the third laser point belongs to multiple first laser points.

[0239] The second optimization submodule is used to update the initial semantic label of the first laser point corresponding to the third laser point based on the fourth semantic label of the third laser point, so as to obtain optimized point cloud data.

[0240] In one implementation, the process of generating the second image includes:

[0241] An image dilation process is applied to the first obstacle region of the first image to obtain a second obstacle region. The first obstacle region and the second obstacle region share the same identification number.

[0242] A second image is generated based on the second obstacle region.

[0243] In one implementation, the fifth determining submodule is used to:

[0244] If the laser point information of the second laser point is used to determine that the first obstacle area is a distant obstacle, the identification number of the first obstacle area is obtained.

[0245] Based on the identification number of the first obstacle area, a second obstacle area associated with the first obstacle area is determined from the second image.

[0246] In one embodiment, the point cloud data optimization device further includes:

[0247] The third determining module is used to determine the first semantic label of each pixel in the first image using a semantic segmentation network.

[0248] In one implementation, the first determining module 710 includes:

[0249] The projection submodule is used to project multiple first laser points from the point cloud data onto the first image.

[0250] The seventh determination submodule is used to determine the initial semantic labels of multiple first laser points based on the first semantic labels of each pixel in the first image.

[0251] The specific functions and examples of each module and submodule of the apparatus in this disclosure can be found in the relevant descriptions of the corresponding steps in the above method embodiments, and will not be repeated here.

[0252] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0253] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0254] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0255] like Figure 8As shown, device 800 includes a computing unit 801, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 802 or a computer program loaded from storage unit 808 into random access memory (RAM) 803. RAM 803 may also store various programs and data required for the operation of device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via bus 804. Input / output (I / O) interface 805 is also connected to bus 804.

[0256] Multiple components in device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of monitors, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0257] The computing unit 801 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as the point cloud data optimization method. For example, in some embodiments, the point cloud data optimization method can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the point cloud data optimization method described above can be performed. Alternatively, in other embodiments, the computing unit 801 can be configured to perform the point cloud data optimization method by any other suitable means (e.g., by means of firmware).

[0258] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0259] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0260] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0261] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0262] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0263] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0264] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0265] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for optimizing point cloud data, comprising: Based on the first semantic label of each pixel in the first image, determine the projection of multiple first laser points of the point cloud data onto the initial semantic label corresponding to the first image; Determine a first cluster comprising a first obstacle region in the first image, wherein the first cluster includes a second laser point, the second laser point belonging to the plurality of first laser points; and Using the laser point information of the second laser point, the initial semantic labels of the plurality of first laser points are updated to obtain optimized point cloud data, including: Using a sliding window algorithm and a first feature index, a first laser point subset is determined from multiple second laser points in the first cluster, wherein the spatial positions of multiple second laser points in the first laser point subset are continuous. Based on the second feature index and the laser point information of multiple second laser points in the first laser point subset, the second semantic labels of multiple second laser points in the first laser point subset are determined, wherein the second semantic labels include water mist noise semantic labels and obstacle semantic labels. Based on the second semantic label, the initial semantic labels of the first laser points corresponding to multiple second laser points in the first laser point subset are updated to obtain optimized point cloud data.

2. The method according to claim 1, further comprising: Based on the distance between the first obstacle area and the target object, a first feature index and a second feature index are determined; wherein, the target object is the object on which the sensors that collect the point cloud data are deployed.

3. The method according to claim 1, wherein, The step of determining a subset of first laser points from multiple second laser points in the first cluster using a sliding window algorithm and a first feature index includes: Based on the laser point information of multiple second laser points in the first cluster, the multiple second laser points in the first cluster are sorted according to their distance from the target object; wherein, the target object is the object on which the sensor for collecting the point cloud data is deployed; Using a sliding window algorithm, a subset of second laser points is determined from a sorted set of second laser points; wherein the spatial positions of the multiple second laser points within the subset are continuous. Based on the first feature index and the laser point information of multiple second laser points within the second laser point subset, a first laser point subset is determined from the second laser point subset.

4. The method according to claim 3, wherein, The step of determining the first laser point subset from the second laser point subset based on the first feature index and the laser point information of multiple second laser points within the second laser point subset includes: Determine the laser point height information of multiple second laser points within the second laser point subset; Determine the height range threshold of the first feature indicator; Based on the height range threshold and the laser point height information, a first laser point subset is determined from the second laser point subset; or Determine the laser point distance information of multiple second laser points within the second laser point subset; Determine the distance threshold for the first feature index; Based on the distance threshold and the laser point distance information, a first laser point subset is determined from the second laser point subset.

5. The method according to claim 3, wherein, The step of determining the first laser point subset from the second laser point subset based on the first feature index and the laser point information of multiple second laser points within the second laser point subset includes: Determine the laser point height information and laser point distance information of multiple second laser points within the second laser point subset; Determine the height range threshold and distance threshold for the first feature indicator; Based on the distance threshold, the height range threshold, the laser point distance information, and the laser point height information, a first laser point subset is determined from the second laser point subset.

6. The method according to any one of claims 1 to 5, wherein, The step of determining the second semantic labels of the multiple second laser points in the first laser point subset based on the second feature index and the laser point information of the multiple second laser points in the first laser point subset includes: Determine the laser point height information, laser point distance information, and number of second laser points within the first laser point subset; Determine the height range threshold, distance threshold, and quantity threshold for the second feature indicator; Based on the laser point height information, the laser point distance information, the quantity information, the height range threshold, the distance threshold, and the quantity threshold, a second semantic label is determined for a plurality of second laser points in the first laser point subset.

7. The method according to any one of claims 1 to 5, wherein, The step of determining the second semantic labels of the multiple second laser points in the first laser point subset based on the second feature index and the laser point information of the multiple second laser points in the first laser point subset includes: Determine the laser point height information, laser point distance information, laser point reflection length information, and the number of second laser points within the first laser point subset for multiple second laser points; Determine the height range threshold, distance threshold, quantity threshold, and reflection length threshold for the second characteristic index; Based on the laser point height information, the laser point distance information, the laser point reflection length information, the quantity information, the height range threshold, the distance threshold, the quantity threshold, and the reflection length threshold, a second semantic label is determined for a plurality of second laser points in the first laser point subset.

8. The method according to claim 1, wherein, The step of updating the initial semantic labels of the plurality of first laser points using the laser point information of the second laser point to obtain optimized point cloud data further includes: Based on the third semantic label of each pixel in the second image, the fourth semantic label corresponding to the projection of the plurality of first laser points onto the second image is determined; wherein, the second image is generated from the first image through image dilation processing; If the laser point information of the second laser point is used to determine that the first obstacle area is a distant obstacle, a second obstacle area associated with the first obstacle area is determined from the second image; A second cluster is determined to include the associated second obstacle region, wherein the second cluster includes a third laser point, the third laser point belonging to the plurality of first laser points; Based on the fourth semantic label of the third laser point, the initial semantic label of the first laser point corresponding to the third laser point is updated to obtain optimized point cloud data.

9. The method according to claim 8, wherein, The process of generating the second image includes: The first obstacle region of the first image is subjected to image dilation processing to obtain the second obstacle region; wherein the first obstacle region and the second obstacle region have the same identification number; The second image is generated based on the second obstacle region.

10. The method according to claim 8 or 9, wherein, When determining that the first obstacle region is a distant obstacle using the laser point information of the second laser point, determining the second obstacle region associated with the first obstacle region from the second image includes: If the laser point information of the second laser point is used to determine that the first obstacle area is a distant obstacle, the identification number of the first obstacle area is obtained; Based on the identification number of the first obstacle area, a second obstacle area associated with the first obstacle area is determined from the second image.

11. The method according to any one of claims 1 to 5, wherein, Before determining the projection of multiple first laser points from the point cloud data onto the initial semantic labels corresponding to the first image based on the first semantic labels of each pixel in the first image, the method further includes: Using a semantic segmentation network, the first semantic label of each pixel in the first image is determined.

12. The method according to any one of claims 1 to 5, wherein, The step of determining the projection of multiple first laser points from the point cloud data onto the initial semantic labels corresponding to the first image based on the first semantic labels of each pixel in the first image includes: Project multiple first laser points from the point cloud data onto the first image; The initial semantic labels of the plurality of first laser points are determined based on the first semantic labels of each pixel in the first image.

13. An optimization device for point cloud data, comprising: The first determining module is used to determine, based on the first semantic label of each pixel of the first image, the projection of multiple first laser points of the point cloud data onto the initial semantic label corresponding to the first image; The second determining module is configured to determine a first cluster contained in the first obstacle region of the first image, wherein the first cluster includes a second laser point, and the second laser point belongs to the plurality of first laser points; and An optimization module is used to update the initial semantic labels of the plurality of first laser points using the laser point information of the second laser point, so as to obtain optimized point cloud data; The optimization module includes: The first determining submodule is used to determine a first laser point subset from multiple second laser points of the first cluster using a sliding window algorithm and a first feature index, wherein the spatial positions of the multiple second laser points of the first laser point subset are continuous. The second determining submodule is used to determine the second semantic labels of the multiple second laser points in the first laser point subset based on the second feature index and the laser point information of the multiple second laser points in the first laser point subset, wherein the second semantic labels include water mist noise semantic labels and obstacle semantic labels. The first optimization submodule is used to update the initial semantic labels of the first laser points corresponding to multiple second laser points in the first laser point subset according to the second semantic labels, so as to obtain optimized point cloud data.

14. The apparatus of claim 13, further comprising: The third determining submodule is used to determine a first feature index and a second feature index based on the distance between the first obstacle area and the target object; wherein the target object is the object on which the sensors that collect the point cloud data are deployed.

15. The apparatus according to claim 13, wherein, The first determining submodule includes: The sorting unit is used to sort the multiple second laser points of the first cluster according to their distance to the target object based on the laser point information of the multiple second laser points of the first cluster; wherein, the target object is the object on which the sensor for collecting the point cloud data is deployed; The first determining unit is used to determine a subset of second laser points from a sorted plurality of second laser points using a sliding window algorithm; wherein the spatial positions of the plurality of second laser points within the subset of second laser points are continuous. The second determining unit is used to determine the first laser point subset from the second laser point subset based on the first feature index and the laser point information of multiple second laser points in the second laser point subset.

16. The apparatus according to claim 15, wherein, The second determining unit is used for: Determine the laser point height information of multiple second laser points within the second laser point subset; Determine the height range threshold of the first feature indicator; Based on the height range threshold and the laser point height information, a first laser point subset is determined from the second laser point subset; or Determine the laser point distance information of multiple second laser points within the second laser point subset; Determine the distance threshold for the first feature index; Based on the distance threshold and the laser point distance information, a first laser point subset is determined from the second laser point subset.

17. The apparatus according to claim 15, wherein, The second determining unit is used for: Determine the laser point height information and laser point distance information of multiple second laser points within the second laser point subset; Determine the height range threshold and distance threshold for the first feature indicator; Based on the distance threshold, the height range threshold, the laser point distance information, and the laser point height information, a first laser point subset is determined from the second laser point subset.

18. The apparatus according to any one of claims 13 to 17, wherein, The second determining submodule is used for: Determine the laser point height information, laser point distance information, and number of second laser points within the first laser point subset; Determine the height range threshold, distance threshold, and quantity threshold for the second feature indicator; Based on the laser point height information, the laser point distance information, the quantity information, the height range threshold, the distance threshold, and the quantity threshold, a second semantic label is determined for a plurality of second laser points in the first laser point subset.

19. The apparatus according to any one of claims 13 to 17, wherein, The second determining submodule is used for: Determine the laser point height information, laser point distance information, laser point reflection length information, and the number of second laser points within the first laser point subset for multiple second laser points; Determine the height range threshold, distance threshold, quantity threshold, and reflection length threshold for the second characteristic index; Based on the laser point height information, the laser point distance information, the laser point reflection length information, the quantity information, the height range threshold, the distance threshold, the quantity threshold, and the reflection length threshold, a second semantic label is determined for a plurality of second laser points in the first laser point subset.

20. The apparatus according to claim 14, wherein, The optimization module further includes: The fourth determining submodule is used to determine the fourth semantic label corresponding to the projection of the plurality of first laser points onto the second image based on the third semantic label of each pixel of the second image; wherein, the second image is generated by the first image through image dilation processing; The fifth determining submodule is used to determine a second obstacle region associated with the first obstacle region from the second image when the first obstacle region is determined to be a distant obstacle using the laser point information of the second laser point. The sixth determining submodule is used to determine the second cluster contained in the associated second obstacle region, wherein the second cluster includes a third laser point, and the third laser point belongs to the plurality of first laser points; The second optimization submodule is used to update the initial semantic label of the first laser point corresponding to the third laser point according to the fourth semantic label of the third laser point, so as to obtain optimized point cloud data.

21. The apparatus according to claim 20, wherein, The process of generating the second image includes: The first obstacle region of the first image is subjected to image dilation processing to obtain the second obstacle region; wherein the first obstacle region and the second obstacle region have the same identification number; The second image is generated based on the second obstacle region.

22. The apparatus according to claim 20 or 21, wherein, The fifth determination submodule is used for: If the laser point information of the second laser point is used to determine that the first obstacle area is a distant obstacle, the identification number of the first obstacle area is obtained; Based on the identification number of the first obstacle area, a second obstacle area associated with the first obstacle area is determined from the second image.

23. The apparatus according to any one of claims 13 to 17, further comprising: The third determining module is used to determine the first semantic label of each pixel in the first image using a semantic segmentation network.

24. The apparatus according to any one of claims 13 to 17, wherein, The first determining module includes: The projection submodule is used to project multiple first laser points from the point cloud data onto the first image; The seventh determining submodule is used to determine the initial semantic labels of the plurality of first laser points based on the first semantic labels of each pixel point of the first image.

25. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 12.

26. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1 to 12.

27. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 12.

28. An autonomous vehicle, comprising: Sensors are used to collect point cloud data; Image acquisition device, used to acquire the first image; A computing unit, connected to the sensor and the image acquisition device, is used to perform the method according to any one of claims 1 to 12.

Images