Point cloud semantic segmentation method, device, equipment and medium

Abstract

The present application discloses a method, device, equipment and medium for semantic segmentation of point cloud, which belongs to the field of computer technology. The method includes: obtaining the point cloud data to be processed and the corresponding image data; based on the point cloud data, using the recognition strategy, respectively determining the first ground point cloud data and the first obstacle point cloud data in the point cloud data; based on the point cloud data and the image data, using the image semantic segmentation strategy, determining the first point cloud semantic segmentation data corresponding to the point cloud data; based on the first ground point cloud data, filtering the first point cloud semantic segmentation data to obtain the second point cloud semantic segmentation data and the second ground point cloud data; based on the first obstacle point cloud data, filtering the second point cloud semantic segmentation data to obtain the third point cloud semantic segmentation data and the second obstacle point cloud data; based on the second ground point cloud data, the second obstacle point cloud data, and the third point cloud semantic segmentation data, obtaining the point cloud semantic segmentation result.

Description

Technical Field

[0001] The present application relates to the field of computer technology, specifically to technical fields such as perception technology and autonomous driving, and in particular to a method, device, equipment and medium for semantic segmentation of point clouds. Background Art

[0002] Point cloud semantic segmentation technology converts geometric information into understandable semantic information by assigning semantic labels to each point, thereby enhancing the perception system's ability to understand and interpret the three-dimensional environment. The core of the point cloud semantic segmentation task is to classify each point in the point cloud into predefined categories, such as roads, buildings, pedestrians, vehicles, etc. This process usually requires high-precision and high-efficiency segmentation algorithms, as well as a large amount of high-quality annotated data.

[0003] At present, the relevant technologies mainly use manual annotation of point cloud semantic segmentation labels, or use the image semantic segmentation results to annotate the point cloud data corresponding to the image to obtain the point cloud semantic segmentation labels. This method can specifically use the deep learning model to perform semantic segmentation on the RGB image of the scene, obtain the semantic labels of each area in the image, and then map the semantic labels in the image to the corresponding point cloud data through the camera parameters, giving semantic information to each point cloud point. Summary of the invention

[0004] The present application provides a method, device, equipment and medium for point cloud semantic segmentation, which can solve the problem of poor accuracy of point cloud semantic segmentation processing. The technical solution is as follows:

[0005] In a first aspect, a method for semantic segmentation of a point cloud is provided, the method comprising:

[0006] Acquire point cloud data to be processed and image data corresponding to the point cloud data to be processed;

[0007] Based on the point cloud data, using a preset recognition strategy, respectively determining first ground point cloud data and first obstacle point cloud data in the point cloud data;

[0008] Based on the point cloud data and the image data, using a preset image semantic segmentation strategy, determining first point cloud semantic segmentation data corresponding to the point cloud data;

[0009] Based on the first ground point cloud data, filtering the ground semantic point cloud data in the first point cloud semantic segmentation data to obtain second point cloud semantic segmentation data and second ground point cloud data;

[0010] Based on the first obstacle point cloud data, filtering the point cloud data of obstacle semantics in the second point cloud semantic segmentation data to obtain third point cloud semantic segmentation data and second obstacle point cloud data;

[0011] Based on the second ground point cloud data, the second obstacle point cloud data, and the third point cloud semantic segmentation data, a point cloud semantic segmentation result of the point cloud data is obtained.

[0012] In a possible implementation, based on the point cloud data, using a preset recognition strategy, respectively determining first ground point cloud data and first obstacle point cloud data in the point cloud data includes:

[0013] Using a first recognition algorithm, performing ground point cloud recognition processing on the point cloud data to obtain the first ground point cloud data;

[0014] The point cloud data is processed for obstacle recognition using a second recognition algorithm to obtain the first obstacle point cloud data.

[0015] In a possible implementation, the using a first recognition algorithm to perform ground point cloud recognition processing on the point cloud data to obtain the first ground point cloud data includes:

[0016] Get the point cloud data within the specified height range in the point cloud data;

[0017] Performing plane fitting processing on the point cloud data within the specified height range to obtain a ground plane;

[0018] Based on the ground plane, ground point cloud recognition processing is performed on the point cloud data to obtain the first ground point cloud data.

[0019] In a possible implementation, determining first point cloud semantic segmentation data corresponding to the point cloud data by using a preset image semantic segmentation strategy based on the point cloud data and the image data includes:

[0020] Performing semantic segmentation processing on the image data to obtain image semantic segmentation data;

[0021] The point cloud data and the image semantic segmentation data are fused to obtain first point cloud semantic segmentation data.

[0022] In a possible implementation, filtering the point cloud data of obstacle semantics in the second point cloud semantic segmentation data based on the first obstacle point cloud data to obtain third point cloud semantic segmentation data and second obstacle point cloud data includes:

[0023] Based on the first obstacle point cloud data, filtering the point cloud data of obstacle semantics in the second point cloud semantic segmentation data;

[0024] Based on the preset background semantics, updating the semantic label of each point cloud in the result of the filtering process;

[0025] Based on the result of the updating process, obtaining third point cloud semantic segmentation data;

[0026] The filtered point cloud data is used as the second obstacle point cloud data.

[0027] In a second aspect, a device for point cloud semantic segmentation is provided, the device comprising:

[0028] A first acquisition unit, used to acquire point cloud data to be processed and image data corresponding to the point cloud data to be processed;

[0029] A first determining unit, configured to respectively determine first ground point cloud data and first obstacle point cloud data in the point cloud data based on the point cloud data and using a preset recognition strategy;

[0030] A second determining unit, configured to determine first point cloud semantic segmentation data corresponding to the point cloud data based on the point cloud data and the image data and using a preset image semantic segmentation strategy;

[0031] A first filtering unit is used to filter the ground semantic point cloud data in the first point cloud semantic segmentation data based on the first ground point cloud data to obtain second point cloud semantic segmentation data and second ground point cloud data;

[0032] A second filtering unit is used to filter the point cloud data of obstacle semantics in the second point cloud semantic segmentation data based on the first obstacle point cloud data to obtain third point cloud semantic segmentation data and second obstacle point cloud data;

[0033] The first obtaining unit is used to obtain a point cloud semantic segmentation result of the point cloud data based on the second ground point cloud data, the second obstacle point cloud data, and the third point cloud semantic segmentation data.

[0034] In a third aspect, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by a processor to implement the method of the above-mentioned aspect and any possible implementation manner.

[0035] In a fourth aspect, an electronic device is provided, including:

[0036] at least one processor; and

[0037] a memory communicatively connected to the at least one processor; wherein,

[0038] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any possible implementation manner and the aspects described above.

[0039] According to a fifth aspect, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the above-mentioned aspects and any possible implementation method.

[0040] In a sixth aspect, an autonomous driving vehicle is provided, comprising the electronic device as described above.

[0041] The beneficial effects of the technical solution provided by this application include at least:

[0042] It can be seen from the above technical scheme that the embodiment of the present application can obtain the point cloud data to be processed and the image data corresponding to the point cloud data to be processed, and then based on the point cloud data, use a preset recognition strategy to respectively determine the first obstacle point cloud data and the first ground point cloud data in the point cloud data, based on the point cloud data and the image data, use a preset image semantic segmentation strategy to determine the first point cloud semantic segmentation data corresponding to the point cloud data, based on the ground point cloud data, filter the point cloud data of ground semantics in the first point cloud semantic segmentation data to obtain second point cloud semantic segmentation data, based on the first obstacle point cloud data, filter the point cloud data of obstacle semantics in the second point cloud semantic segmentation data to obtain third point cloud semantic segmentation data. Data, so that the point cloud semantic segmentation result of the point cloud data can be obtained based on the ground point cloud data, the second obstacle point cloud data, and the third point cloud semantic segmentation data. Since the first point cloud semantic segmentation data determined by the image semantic segmentation strategy can be filtered and processed respectively according to the determined obstacle point cloud data and the ground point cloud data, the third point cloud semantic segmentation data, as well as the second ground point cloud data and the second obstacle point cloud data are obtained, and then the second ground point cloud data, the second obstacle point cloud data, and the third point cloud semantic segmentation data are used as the point cloud semantic segmentation result of the point cloud data, a more accurate point cloud semantic segmentation result can be obtained by combining the multi-dimensional point cloud semantic recognition result, thereby improving the reliability of the point cloud semantic segmentation result, thereby ensuring the reliability of the point cloud semantic label.

[0043] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0045] Figure 1 It is a flowchart of a method for semantic segmentation of point clouds provided by an embodiment of the present application;

[0046] Figure 2 is a flowchart of a method for semantic segmentation of point clouds provided by another embodiment of the present application;

[0047] Figure 3 is a structural block diagram of a device for point cloud semantic segmentation provided in yet another embodiment of the present application;

[0048] Figure 4 It is a block diagram of an electronic device used to implement the method of point cloud semantic segmentation in an embodiment of the present application. DETAILED DESCRIPTION

[0049] The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0050] Obviously, the described embodiments are only part of the embodiments of the present application, but not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0051] It should be noted that the terminal devices involved in the embodiments of the present application may include but are not limited to mobile phones, personal digital assistants (PDAs), wireless handheld devices, tablet computers and other smart devices; display devices may include but are not limited to personal computers, televisions and other devices with display functions.

[0052] In addition, the term "and / or" in this article is only a description of the association relationship between the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0053] At present, the relevant technologies mainly use manual annotation of point cloud semantic segmentation labels, or use the image semantic segmentation results to annotate the point cloud data corresponding to the image to obtain point cloud semantic segmentation labels, and use the image semantic segmentation deep learning model to perform semantic segmentation on the collected scene images to obtain semantic labels for each area in the image. Then, the semantic labels in the image are mapped to the corresponding point cloud data through camera parameters to give semantics to each point cloud point.

[0054] However, although the traditional method of manually labeling scene point cloud semantic segmentation labels can provide highly accurate labeling data, it is time-consuming, costly, and inefficient, making it difficult to meet the needs of large-scale data processing. The method of using image semantic segmentation results to generate point cloud segmentation labels has limited accuracy and robustness of the image segmentation model used. There are parallax problems between the camera and the lidar due to sensor placement and sensor characteristics. These problems may lead to incomplete transmission of semantic information and reduced accuracy of segmentation results.

[0055] Therefore, there is an urgent need for a point cloud semantic segmentation method to effectively segment point cloud data and thus ensure the reliability and accuracy of point cloud semantic labels.

[0056] Please refer to Figure 1 , which shows a flow chart of a method for semantic segmentation of point clouds provided by an embodiment of the present application. The method for semantic segmentation of point clouds may specifically include:

[0057] Step 101: Obtain point cloud data to be processed and image data corresponding to the point cloud data to be processed.

[0058] Step 102: Based on the point cloud data, using a preset recognition strategy, respectively determine the first ground point cloud data and the first obstacle point cloud data in the point cloud data.

[0059] Step 103: Based on the point cloud data and the image data, a preset image semantic segmentation strategy is used to determine first point cloud semantic segmentation data corresponding to the point cloud data.

[0060] Step 104: Based on the ground point cloud data, filter the ground semantic point cloud data in the first point cloud semantic segmentation data to obtain second point cloud semantic segmentation data and second ground point cloud data.

[0061] Step 105: Based on the first obstacle point cloud data, filter the point cloud data of obstacle semantics in the second point cloud semantic segmentation data to obtain third point cloud semantic segmentation data and second obstacle point cloud data.

[0062] Step 106: Obtain a point cloud semantic segmentation result of the point cloud data based on the second ground point cloud data, the second obstacle point cloud data, and the third point cloud semantic segmentation data.

[0063] It should be noted that the first ground point cloud data may be point cloud data with ground semantic labels, namely ground semantic point cloud data. The first obstacle point cloud data may be point cloud data with obstacle semantic labels, namely obstacle semantic point cloud data.

[0064] It should be noted that the second point cloud semantic segmentation data may be point cloud data after filtering ground point cloud data, namely ground semantic point cloud data. The third point cloud semantic segmentation data may be point cloud data of background semantics, namely background point cloud data.

[0065] It should be noted that the second ground point cloud data may be ground semantic point cloud data filtered out from the first point cloud semantic segmentation data. The second obstacle point cloud data may be obstacle semantic point cloud data filtered out from the second point cloud semantic segmentation data.

[0066] It should be noted that part or all of the execution subject of steps 101 to 106 may be an application located in the local terminal, or may also be a functional unit such as a plug-in or software development kit (SDK) set in the application located in the local terminal, or may also be a processing engine located in a network-side server, or may also be a distributed system located on the network side, for example, a processing engine or distributed system in an autonomous driving platform on the network side, etc. This embodiment does not specifically limit this.

[0067] It is understandable that the application may be a local program (nativeApp) installed on the local terminal, or may also be a webpage program (webApp) of a browser on the local terminal, which is not limited in this embodiment.

[0068] In this way, the point cloud semantic segmentation data determined by the image semantic segmentation strategy can be filtered separately according to the first obstacle point cloud data and the first ground point cloud data determined by the recognition strategy to obtain the third point cloud semantic segmentation data, the second ground point cloud data, and the second obstacle point cloud data. The second ground point cloud data, the second obstacle point cloud data, and the third point cloud semantic segmentation data are then used as the point cloud semantic segmentation result of the point cloud data. The multi-dimensional point cloud semantic recognition result can be combined to obtain a more accurate point cloud semantic segmentation result, thereby improving the reliability of the point cloud semantic segmentation result and ensuring the reliability of the point cloud semantic label.

[0069] Optionally, in a possible implementation of the present embodiment, in step 102, the point cloud data can be subjected to ground point cloud recognition processing using a first recognition algorithm to obtain the first ground point cloud data, and the point cloud data can be subjected to obstacle recognition processing using a second recognition algorithm to obtain the first obstacle point cloud data.

[0070] In this implementation, the first recognition algorithm may include a ground point cloud recognition algorithm.

[0071] In a specific implementation process of this implementation method, first, point cloud data within a specified height range is obtained from the point cloud data. Second, plane fitting processing is performed on the point cloud data within the specified height range to obtain a ground plane. Third, based on the ground plane, ground point cloud recognition processing is performed on the point cloud data to obtain the first ground point cloud data.

[0072] In this specific implementation process, the specified height range may be a point cloud height range determined based on the ground area. For example, the specified height range may be a height value in the range of 0.7 meters to 1.2 meters, and the height value may be a Z value of the point cloud, and point cloud data whose Z value of the point cloud is within the specified height range may be obtained.

[0073] In one case of the specific implementation process, first, a preset plane fitting algorithm is used to perform plane fitting processing on the point cloud data within the specified height range to obtain a ground plane. Then, point cloud data whose distance from the ground plane is within a preset distance range is selected, and the point cloud data is identified as ground point cloud data to obtain the first ground point cloud data.

[0074] In this implementation, the preset plane fitting algorithm may include a plane fitting algorithm such as a RANSAC algorithm.

[0075] Exemplarily, the preset distance range may be a distance range of 2 meters above and below the ground plane, and the preset distance range may be [-2, 2].

[0076] Here, the first ground point cloud data may include point cloud data with ground semantics. After obtaining the first ground point cloud data, ground semantic point cloud data may be obtained.

[0077] In this implementation, the second recognition algorithm may include a target obstacle detection algorithm based on a 3D target detection model.

[0078] In a specific implementation process of this implementation method, first, the target obstacle detection algorithm is used to perform obstacle recognition processing on the point cloud data to obtain the first obstacle point cloud data.

[0079] In this specific implementation process, the first obstacle point cloud data may include a target detection frame and a target detection frame category corresponding to the point cloud data. The target detection frame category may represent the point cloud semantics of the point cloud data within the target detection frame.

[0080] In this way, the point cloud data can be recognized and processed respectively by using the first recognition algorithm and the second recognition algorithm to obtain the point cloud of ground semantics and the point cloud of obstacle semantics, which can further improve the accuracy of semantic recognition of the point cloud.

[0081] Optionally, in a possible implementation of this embodiment, in step 103, first, the image data may be semantically segmented to obtain image semantic segmentation data. Secondly, the point cloud data and the image semantic segmentation data may be fused to obtain first point cloud semantic segmentation data.

[0082] In a specific implementation process of this implementation method, first, the image data is semantically segmented using a preset image semantic segmentation model to obtain image semantic segmentation data. Secondly, the point cloud data is mapped to the image semantic segmentation data to obtain first point cloud semantic segmentation data.

[0083] One situation of the specific implementation process is that, based on the position data of each point cloud point in the point cloud data, the point cloud data is mapped to the image semantic segmentation data to obtain the image semantic segmentation data of the image position corresponding to each point cloud point. Secondly, based on the image semantic segmentation data of the image position corresponding to each point cloud point, the first point cloud semantic segmentation data is obtained.

[0084] Here, the first point cloud semantic segmentation data may include point cloud data of each semantic in the acquisition scene corresponding to the point cloud frame. Exemplarily, the first point cloud semantic segmentation data of any frame may include but is not limited to ground semantic point cloud data, obstacle semantic point cloud data, and background semantic point cloud data.

[0085] In this way, more accurate and effective first point cloud semantic segmentation data can be obtained by mapping and fusing the point cloud data and the image semantic segmentation data determined based on the image data, which can further improve the accuracy of the first point cloud semantic segmentation data.

[0086] It should be noted that the specific implementation process provided in this implementation can be combined with the various specific implementation processes provided in the aforementioned implementation to implement the method for point cloud semantic segmentation of this embodiment. For a detailed description, please refer to the relevant content in the aforementioned implementation, which will not be repeated here.

[0087] Optionally, in a possible implementation of this embodiment, in step 105, first, the point cloud data of obstacle semantics in the second point cloud semantic segmentation data may be filtered based on the first obstacle point cloud data. Secondly, based on the preset background semantics, the semantic label of each point cloud in the result of the filtering process is updated. Thirdly, based on the result of the updating process, third point cloud semantic segmentation data is obtained. Thirdly, the filtered point cloud data is used as the second obstacle point cloud data.

[0088] In this implementation, the result of the filtering process may include point cloud data with background semantics and point cloud data with non-background semantics.

[0089] In a specific implementation process of this implementation method, based on the preset background semantics, semantic update processing is performed on the point cloud data without background semantics in the result of the filtering process to update the point cloud data without background semantics to point cloud data with background semantics, and then the point cloud data with background semantics and the semantically updated point cloud data can be used as the third point cloud semantic segmentation data.

[0090] Here, the second point cloud semantic segmentation data may be point cloud data after filtering the ground point cloud data, that is, the ground semantic point cloud data. The third point cloud semantic segmentation data may be point cloud data of background semantics.

[0091] It can be understood that the filtered point cloud data may be point cloud data with obstacle semantics.

[0092] In this way, by first filtering the point cloud semantic segmentation data using the ground point cloud data, and then filtering the result of the first filtering using the obstacle point cloud data, more accurate background semantic point cloud data can be obtained, which can further improve the accuracy of the third point cloud semantic segmentation data, thereby further improving the accuracy of the final point cloud semantic segmentation result.

[0093] Moreover, by first determining the filtered second ground point cloud data and then determining the filtered second obstacle point cloud data, the misidentified ground point cloud data that may exist in the first obstacle point cloud data can be separated to ensure the accuracy of the final ground point cloud data and obstacle point cloud data, thereby further improving the accuracy of the final point cloud semantic segmentation result.

[0094] It should be noted that the specific implementation process provided in this implementation can be combined with the various specific implementation processes provided in the aforementioned implementation to implement the method for point cloud semantic segmentation of this embodiment. For a detailed description, please refer to the relevant content in the aforementioned implementation, which will not be repeated here.

[0095] In order to better understand the method of the embodiment of the present application, the method of the embodiment of the present application is described below in conjunction with the accompanying drawings and specific application scenarios.

[0096] Figure 2 FIG. 1 is a flow chart of a method for semantic segmentation of point cloud provided by another embodiment of the present application. Figure 2 shown.

[0097] Step 201: Obtain the point cloud data to be processed and the image data corresponding to the point cloud data to be processed.

[0098] In this embodiment, the point cloud data to be processed may be single-frame scene point cloud data. The image data may be an image corresponding to the single-frame scene point cloud data. The single-frame scene point cloud data may be point cloud data obtained by a laser radar scanning cycle.

[0099] Step 202: Use a fitting recognition algorithm to perform ground point cloud recognition processing on the point cloud data to obtain first ground point cloud data.

[0100] In this embodiment, first, point cloud data within the ground height range in the point cloud data can be obtained. Secondly, plane fitting processing is performed on the point cloud data within the ground height range to obtain the ground plane. Thirdly, in response to the distance between the point cloud data and the ground plane meeting the preset distance condition, the point cloud data is identified as the first ground point cloud data.

[0101] Here, the preset distance condition may include that the distance between the point cloud data and the ground plane is less than a preset distance threshold, or that the distance between the point cloud data and the ground plane is within a preset distance range. The preset distance threshold may be an absolute value of the distance. For example, the preset distance threshold may be 2 meters. The preset distance range may be a distance range determined based on the ground plane, for example, the preset distance range may be a range of 2 meters above and below the ground plane as a baseline, that is, a distance range of [-2, 2].

[0102] It is understandable that the preset distance condition can be determined according to the ground position conditions in the actual scene.

[0103] Here, after obtaining the first ground point cloud data, a segmentation label of a ground semantic category may be assigned to the first ground point cloud data.

[0104] Step 203: Use the 3D target detection model to perform obstacle recognition processing on the point cloud data to obtain first obstacle point cloud data.

[0105] In this embodiment, the point cloud data is input into a preset 3D target detection model to output a target detection result, which may include an obstacle detection frame and an obstacle category corresponding to the obstacle detection frame. The point cloud data in the obstacle detection frame may be assigned a segmentation label of the obstacle semantic category to obtain the first obstacle point cloud data.

[0106] It can be understood that the obstacle detection frame is the target detection frame, and the obstacle category corresponding to the obstacle detection frame is the target category, target semantics, etc. corresponding to the target detection frame.

[0107] Here, based on the obstacle category corresponding to the obstacle detection frame, a segmentation label of the obstacle semantic category can be assigned to the point cloud data in the obstacle detection frame to obtain the first obstacle point cloud data. Alternatively, the target detection result can be mapped to the image semantic segmentation data, and based on the obstacle category corresponding to the obstacle detection frame and the image semantic segmentation data at the corresponding position, the obstacle semantic category corresponding to each obstacle detection frame is determined, and the segmentation label of the obstacle semantic category is assigned to the point cloud data in the obstacle detection frame to obtain the first obstacle point cloud data.

[0108] Step 204: perform semantic segmentation processing on the image data using the image semantic segmentation model to obtain image semantic segmentation data, so as to map the point cloud data to the image semantic segmentation data to obtain first point cloud semantic segmentation data.

[0109] In this embodiment, firstly, the point cloud data can also be mapped to the image data to obtain the image data including the point cloud points. Secondly, the image data including the point cloud points is input into the image semantic segmentation model to output the first point cloud semantic segmentation data.

[0110] Step 205: Based on the first ground point cloud data, filter the ground semantic point cloud data in the first point cloud semantic segmentation data to obtain second point cloud semantic segmentation data and second ground point cloud data.

[0111] In this embodiment, firstly, based on the first ground point cloud data, the ground semantic point cloud data in the first point cloud semantic segmentation data is filtered to obtain the point cloud semantic segmentation data from which the ground semantic point cloud data is filtered out and the filtered ground semantic point cloud data. Secondly, the point cloud semantic segmentation data from which the ground semantic point cloud data is filtered out is used as the second point cloud semantic segmentation data, and the filtered ground semantic point cloud data is used as the second ground point cloud data.

[0112] It can be understood that by filtering the ground semantic point cloud data in the first point cloud semantic segmentation data based on the first ground point cloud data, the ground semantic point cloud data in the first point cloud semantic segmentation data can be separated from other semantic point cloud data.

[0113] Step 206: Based on the first obstacle point cloud data, filter the point cloud data of obstacle semantics in the second point cloud semantic segmentation data to obtain background point cloud data and second obstacle point cloud data.

[0114] In this embodiment, firstly, based on the first obstacle point cloud data, the point cloud data of obstacle semantics in the second point cloud semantic segmentation data is filtered to obtain point cloud semantic segmentation data of the point cloud data from which the obstacle semantics are filtered out and the point cloud data of the filtered obstacle semantics. Secondly, the point cloud semantic segmentation data of the point cloud data from which the obstacle semantics are filtered out is used as the third point cloud semantic segmentation data, and the point cloud data of the filtered obstacle semantics is used as the second obstacle point cloud data.

[0115] It can be understood that by filtering the point cloud data with obstacle semantics in the second point cloud semantic segmentation data based on the first obstacle point cloud data, the point cloud data with obstacle semantics in the second point cloud semantic segmentation data can be separated from the point cloud data with other semantics.

[0116] Here, the third point cloud semantic segmentation data may be point cloud data of all background semantics.

[0117] In this way, since there may be some ground semantic point cloud data in the identified obstacle detection frame, the second ground point cloud data that may exist in the first obstacle point cloud data can be separated by first filtering the second ground point cloud data from the first point cloud semantic segmentation data and then filtering the second obstacle point cloud data from the second point cloud semantic segmentation data, so as to ensure the accuracy of the final ground point cloud data and obstacle point cloud data, thereby further improving the accuracy of the final point cloud semantic segmentation result.

[0118] Step 207: Use the second ground point cloud data, the second obstacle point cloud data, and the background point cloud data as the point cloud semantic segmentation result of the point cloud data.

[0119] In this embodiment, after obtaining the point cloud semantic segmentation result of the point cloud data, first, multiple frames of point cloud data with point cloud semantic segmentation results can be used to obtain dense point cloud data. Secondly, based on the position data of each point cloud in the dense point cloud data and the preset voxel grid, each point cloud can be voxelized to determine the position index of the voxel grid corresponding to each point cloud. Thirdly, based on the position index of the voxel grid corresponding to each point cloud and the semantic category of each point cloud, the category of each voxel grid is determined. Finally, based on the category of each voxel grid, an occupied grid label can be obtained.

[0120] By adopting the technical solution in this embodiment, the point cloud of the collected scene can be divided into three parts: ground, obstacle and background by using the ground fitting algorithm, image semantic segmentation model and 3D target detection model. The point cloud segmentation label of the ground is obtained by combining the ground extraction and image segmentation results based on the ground fitting algorithm, the point cloud segmentation label of the obstacle is obtained by combining the 3D target detection result and the image segmentation result, and the point cloud segmentation label of the background is obtained by filtering the image segmentation result based on the ground point cloud and 3D target detection results. Finally, the point cloud segmentation labels of the background, ground and obstacles are combined to form the final point cloud segmentation result of the point cloud data. Thus, by using algorithms such as ground plane extraction, 3D target detection, and image 2D segmentation, the point cloud in the scene is divided into three parts, effectively combining and utilizing the ground plane extraction algorithm, 3D target detection algorithm, and image segmentation algorithm, which can alleviate the problems caused by sensor modality differences and insufficient image segmentation accuracy in traditional methods, has high robustness, and improves the reliability of point cloud semantic segmentation results.

[0121] In addition, by adopting the technical solution in this embodiment, the misidentified ground point cloud data that may exist in the first obstacle point cloud data can be separated by first determining the filtered second ground point cloud data and then determining the filtered second obstacle point cloud data, so as to ensure the accuracy of the final ground point cloud data and obstacle point cloud data, thereby further improving the accuracy of the final point cloud semantic segmentation result.

[0122] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0123] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0124] Figure 3 A structural block diagram of a device for point cloud semantic segmentation provided by an embodiment of the present application is shown. Figure 3 The device 300 for semantic segmentation of point cloud in this embodiment may include a first acquisition unit 301 , a first determination unit 302 , a second determination unit 303 , a first filtering unit 304 , a second filtering unit 305 and a first obtaining unit 306 . Among them, the first acquisition unit 301 is used to acquire the point cloud data to be processed and the image data corresponding to the point cloud data to be processed; the first determination unit 302 is used to determine the first ground point cloud data and the first obstacle point cloud data in the point cloud data respectively based on the point cloud data and using a preset recognition strategy; the second determination unit 303 is used to determine the first point cloud semantic segmentation data corresponding to the point cloud data based on the point cloud data and the image data and using a preset image semantic segmentation strategy; the first filtering unit 304 is used to filter the point cloud data of ground semantics in the first point cloud semantic segmentation data based on the first ground point cloud data to obtain second point cloud semantic segmentation data and second ground point cloud data; the second filtering unit 305 is used to filter the point cloud data of obstacle semantics in the second point cloud semantic segmentation data based on the first obstacle point cloud data to obtain third point cloud semantic segmentation data and second obstacle point cloud data; the first acquisition unit 306 is used to obtain the point cloud semantic segmentation result of the point cloud data based on the second ground point cloud data, the second obstacle point cloud data, and the third point cloud semantic segmentation data.

[0125] It should be noted that part or all of the point cloud semantic segmentation device of this embodiment may be an application located in the local terminal, or may also be a functional unit such as a plug-in or software development kit (SDK) set in the application located in the local terminal, or may also be a processing engine located in a network-side server, or may also be a distributed system located on the network side, for example, a processing engine or distributed system in an autonomous driving platform on the network side, etc. This embodiment does not specifically limit this.

[0126] It is understandable that the application may be a local program (nativeApp) installed on the local terminal, or may also be a webpage program (webApp) of a browser on the local terminal, which is not limited in this embodiment.

[0127] Optionally, in a possible implementation of the present embodiment, the first determination unit 302 may be used to perform ground point cloud recognition processing on the point cloud data using a first recognition algorithm to obtain the first ground point cloud data; and to perform obstacle recognition processing on the point cloud data using a second recognition algorithm to obtain the first obstacle point cloud data.

[0128] Optionally, in a possible implementation of the present embodiment, the first determination unit 302 can be used to obtain point cloud data within a specified height range in the point cloud data; perform plane fitting processing on the point cloud data within the specified height range to obtain a ground plane; and based on the ground plane, perform ground point cloud recognition processing on the point cloud data to obtain the first ground point cloud data.

[0129] Optionally, in a possible implementation of this embodiment, the second determination unit 303 can be specifically used to perform semantic segmentation processing on the image data to obtain image semantic segmentation data; and to perform fusion processing on the point cloud data and the image semantic segmentation data to obtain first point cloud semantic segmentation data.

[0130] Optionally, in a possible implementation of the present embodiment, the second filtering unit 305 can be specifically used to filter the point cloud data of obstacle semantics in the second point cloud semantic segmentation data based on the first obstacle point cloud data; update the semantic label of each point cloud in the result of the filtering process based on preset background semantics; obtain third point cloud semantic segmentation data based on the result of the update process; and use the filtered point cloud data as the second obstacle point cloud data.

[0131] In this embodiment, the point cloud data to be processed and the image data corresponding to the point cloud data to be processed can be obtained by the first acquisition unit, and then the first determination unit can determine the first ground point cloud data and the first obstacle point cloud data in the point cloud data respectively based on the point cloud data by using a preset recognition strategy, and the second determination unit can determine the first point cloud semantic segmentation data corresponding to the point cloud data by using a preset image semantic segmentation strategy based on the point cloud data and the image data, and the first filtering unit can filter the point cloud data of ground semantics in the first point cloud semantic segmentation data based on the first ground point cloud data to obtain second point cloud semantic segmentation data and second ground point cloud data, and the second filtering unit can filter the point cloud data of obstacle semantics in the second point cloud semantic segmentation data based on the first obstacle point cloud data to obtain third point cloud semantic segmentation data and second obstacle point cloud data, so that the first acquisition unit can obtain the point cloud semantic segmentation result of the point cloud data based on the second ground point cloud data, the second obstacle point cloud data and the third point cloud semantic segmentation data.

[0132] Since the first point cloud semantic segmentation data determined by using the image semantic segmentation strategy can be filtered and processed respectively according to the determined obstacle point cloud data and ground point cloud data, the third point cloud semantic segmentation data, the second ground point cloud data, and the second obstacle point cloud data can be obtained, and then the second ground point cloud data, the second obstacle point cloud data, and the third point cloud semantic segmentation data are used as the point cloud semantic segmentation result of the point cloud data. The multi-dimensional point cloud semantic recognition results can be combined to obtain a more accurate point cloud semantic segmentation result, thereby improving the reliability of the point cloud semantic segmentation result, thereby ensuring the reliability of the point cloud semantic label.

[0133] In the technical solution of this application, the user personal information involved, such as the collection, storage, use, processing, transmission, provision and disclosure of user images and attribute data, etc., complies with the provisions of relevant laws and regulations and does not violate public order and good morals.

[0134] According to an embodiment of the present application, the present application also provides an electronic device, a readable storage medium and a computer program product.

[0135] According to an embodiment of the present application, further, an autonomous driving vehicle including the provided electronic device is provided, and the autonomous driving vehicle may include unmanned vehicles of L2 and above levels, such as unmanned delivery vehicles, unmanned logistics vehicles, etc.

[0136] Figure 4 A schematic block diagram of an example electronic device 400 that can be used to implement an embodiment of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.

[0137] like Figure 4As shown, the electronic device 400 includes a computing unit 401, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 402 or a computer program loaded from a storage unit 408 into a random access memory (RAM) 403. In the RAM 403, various programs and data required for the operation of the electronic device 400 can also be stored. The computing unit 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0138] Multiple components in the electronic device 400 are connected to the I / O interface 405, including: an input unit 406, such as a keyboard, a mouse, etc.; an output unit 407, such as various types of displays, speakers, etc.; a storage unit 408, such as a disk, an optical disk, etc.; and a communication unit 409, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 409 allows the electronic device 400 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0139] The computing unit 401 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 401 performs the various methods and processes described above, such as the method of point cloud semantic segmentation. For example, in some embodiments, the method of point cloud semantic segmentation may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 408. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 400 via the ROM 402 and / or the communication unit 409. When the computer program is loaded into the RAM 403 and executed by the computing unit 401, one or more steps of the method of point cloud semantic segmentation described above may be performed. Alternatively, in other embodiments, the computing unit 401 may be configured to perform the method of point cloud semantic segmentation in any other appropriate manner (e.g., by means of firmware).

[0140] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0141] The program code for implementing the method of the present application can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, implements the functions / operations specified in the flow chart and / or block diagram. The program code can be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0142] In the context of the present application, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0143] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0144] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0145] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0146] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution disclosed in this application can be achieved, and this document is not limited here.

[0147] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

Claims

1. A method for semantic segmentation of point clouds, characterized in that: The method comprises: Acquire point cloud data to be processed and image data corresponding to the point cloud data to be processed; Based on the point cloud data, using a preset recognition strategy, respectively determining first ground point cloud data and first obstacle point cloud data in the point cloud data; Based on the point cloud data and the image data, using a preset image semantic segmentation strategy, determining first point cloud semantic segmentation data corresponding to the point cloud data; Based on the first ground point cloud data, filtering the ground semantic point cloud data in the first point cloud semantic segmentation data to obtain second point cloud semantic segmentation data and second ground point cloud data; Based on the first obstacle point cloud data, filtering the point cloud data of obstacle semantics in the second point cloud semantic segmentation data to obtain a filtering result, wherein the filtering result includes point cloud data of background semantics and point cloud data of non-background semantics; Based on the preset background semantics, performing semantic update processing on the point cloud data without background semantics in the result of the filtering processing, so as to update the point cloud data without background semantics into point cloud data with background semantics; The point cloud data of the background semantics and the point cloud data updated to the background semantics are used as third point cloud semantic segmentation data; The filtered point cloud data is used as the second obstacle point cloud data; Based on the second ground point cloud data, the second obstacle point cloud data, and the third point cloud semantic segmentation data, a point cloud semantic segmentation result of the point cloud data is obtained.

2. The method according to claim 1, characterized in that: The step of determining first ground point cloud data and first obstacle point cloud data in the point cloud data based on the point cloud data and using a preset recognition strategy includes: Using a first recognition algorithm, performing ground point cloud recognition processing on the point cloud data to obtain the first ground point cloud data; The point cloud data is processed for obstacle recognition using a second recognition algorithm to obtain the first obstacle point cloud data.

3. The method according to claim 2, characterized in that The step of performing ground point cloud recognition processing on the point cloud data using a first recognition algorithm to obtain the first ground point cloud data includes: Get the point cloud data within the specified height range in the point cloud data; Performing plane fitting processing on the point cloud data within the specified height range to obtain a ground plane; Based on the ground plane, ground point cloud recognition processing is performed on the point cloud data to obtain the first ground point cloud data.

4. The method according to claim 1, characterized in that: The determining, based on the point cloud data and the image data, first point cloud semantic segmentation data corresponding to the point cloud data by using a preset image semantic segmentation strategy comprises: Performing semantic segmentation processing on the image data to obtain image semantic segmentation data; The point cloud data and the image semantic segmentation data are fused to obtain first point cloud semantic segmentation data.

5. A device for point cloud semantic segmentation, characterized in that: The device comprises: A first acquisition unit, used to acquire point cloud data to be processed and image data corresponding to the point cloud data to be processed; A first determining unit, configured to respectively determine first ground point cloud data and first obstacle point cloud data in the point cloud data based on the point cloud data and using a preset recognition strategy; A second determining unit, configured to determine first point cloud semantic segmentation data corresponding to the point cloud data based on the point cloud data and the image data and using a preset image semantic segmentation strategy; A first filtering unit is used to filter the ground semantic point cloud data in the first point cloud semantic segmentation data based on the first ground point cloud data to obtain second point cloud semantic segmentation data and second ground point cloud data; a second filtering unit, configured to filter the point cloud data of obstacle semantics in the second point cloud semantic segmentation data based on the first obstacle point cloud data, to obtain a result of the filtering process, wherein the result of the filtering process includes point cloud data of background semantics and point cloud data of non-background semantics; based on preset background semantics, perform semantic update processing on the point cloud data of non-background semantics in the result of the filtering process to update the point cloud data of non-background semantics to point cloud data of background semantics; use the point cloud data of background semantics and the point cloud data updated to background semantics as third point cloud semantic segmentation data; and use the filtered point cloud data as second obstacle point cloud data; The first obtaining unit is used to obtain a point cloud semantic segmentation result of the point cloud data based on the second ground point cloud data, the second obstacle point cloud data, and the third point cloud semantic segmentation data.

6. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-4.

8. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 4.

9. An autonomous driving vehicle comprising the electronic device as claimed in claim 6.

Images