Point cloud semantic segmentation method and apparatus, and device and storage medium

Abstract

The present application relates to the technical field of computers. Disclosed are a point cloud semantic segmentation method and apparatus, and a device and a storage medium. The method comprises: acquiring point cloud data to be processed, wherein said point cloud data is collected by means of a first LiDAR and a second LiDAR; extracting first range view (RV) image feature information corresponding to point cloud data within a vertical field-of-view range of the first LiDAR, and second RV image feature information corresponding to point cloud data within a vertical field-of-view range of the second LiDAR; performing image compression processing on the second RV image feature information, so as to obtain third RV image feature information; on the basis of the first RV image feature information and the third RV image feature information, obtaining target RV image feature information; and using a pre-trained semantic segmentation model to perform semantic segmentation processing on the target RV image feature information, so as to obtain a point cloud semantic segmentation result. The present application reduces the computational load and time consumption of point cloud semantic segmentation processing.

Description

Point cloud semantic segmentation method, device, equipment and storage medium Technical Field

[0001] The present application relates to the field of computer technology, specifically to technical fields such as intelligent transportation and autonomous driving, and in particular to a method, apparatus, device and storage medium for point cloud semantic segmentation. Background Art

[0002] To achieve environmental awareness, autonomous vehicles are typically equipped with LiDAR (Lidar). LiDAR emits beams of light at a specific frequency. These beams reflect off surrounding objects and are then received by the LiDAR, generating raw point cloud data. Semantic segmentation of point clouds involves assigning distinct semantics to each LiDAR-generated point cloud, enabling classification. For example, semantics can include buildings, roads, vegetation, pedestrians, and vehicles.

[0003] Currently, point cloud semantic segmentation solutions in related technologies include point-based, voxel-based, and range-view (RV)-based methods. However, these solutions still suffer from high computational complexity and time-consuming data processing when processing multi-lidar point cloud data. Summary of the Invention

[0004] This application provides a method, apparatus, device, and storage medium for point cloud semantic segmentation, which solves the problem of high computational complexity and high time consumption in point cloud semantic segmentation processing. The technical solution is as follows:

[0005] In a first aspect, a method for semantic segmentation of a point cloud is provided, the method comprising:

[0006] Acquire point cloud data to be processed; the point cloud data is collected by a first laser radar and a second laser radar;

[0007] Extracting first depth perspective RV image feature information corresponding to the point cloud data within the vertical field of view of the first laser radar and second RV image feature information corresponding to the point cloud data within the vertical field of view of the second laser radar respectively;

[0008] performing image compression processing on the second RV image characteristic information to obtain third RV image characteristic information;

[0009] Obtaining target RV image feature information according to the first RV image feature information and the third RV image feature information;

[0010] The pre-trained semantic segmentation model is used to perform semantic segmentation processing on the target RV image feature information to obtain a point cloud semantic segmentation result.

[0011] In a possible implementation, performing image compression processing on the second RV image feature information to obtain third RV image feature information includes:

[0012] performing image compression processing on the second RV image feature information based on a preset width;

[0013] Based on the result of the image compression process, third RV image feature information is obtained.

[0014] In a possible implementation, obtaining target RV image feature information according to the first RV image feature information and the third RV image feature information includes:

[0015] performing splicing processing on the first RV image feature information and the third RV image feature information;

[0016] Based on the result of the stitching process, the target RV image feature information is obtained.

[0017] In one possible implementation, respectively extracting first depth perspective RV image feature information corresponding to point cloud data within the vertical field of view of the first laser radar, and second RV image feature information corresponding to point cloud data within the vertical field of view of the second laser radar, includes:

[0018] Performing projection processing on the point cloud data to obtain RV image feature information of the point cloud data;

[0019] The first RV image feature information corresponding to the point cloud data within the vertical field of view of the first laser radar and the second RV image feature information corresponding to the point cloud data within the vertical field of view of the second laser radar are extracted from the RV image feature information respectively.

[0020] In one possible implementation, respectively extracting first depth perspective RV image feature information corresponding to point cloud data within the vertical field of view of the first laser radar, and second RV image feature information corresponding to point cloud data within the vertical field of view of the second laser radar, includes:

[0021] Performing projection processing on the point cloud data within the vertical field of view of the first laser radar to extract first RV image feature information;

[0022] Projection processing is performed on the point cloud data within the vertical field of view of the second laser radar to extract the second RV image feature information.

[0023] In a second aspect, a model training method is provided, the method comprising:

[0024] Acquire sample point cloud data; the sample point cloud data is collected by a first laser radar and a second laser radar;

[0025] Extracting first depth perspective RV image feature information corresponding to the sample point cloud data within the vertical field of view of the first laser radar, and second RV image feature information corresponding to the sample point cloud data within the vertical field of view of the second laser radar;

[0026] performing image compression processing on the second RV image characteristic information to obtain third RV image characteristic information;

[0027] Obtaining target RV image feature information according to the first RV image feature information and the third RV image feature information;

[0028] Based on the target RV image feature information, the semantic segmentation model to be trained is trained to obtain a trained semantic segmentation model.

[0029] In a third aspect, a device for point cloud semantic segmentation is provided, the device comprising:

[0030] An acquisition unit, configured to acquire point cloud data to be processed; the point cloud data is collected by a first laser radar and a second laser radar;

[0031] an extraction unit, configured to respectively extract first depth perspective RV image feature information corresponding to the point cloud data within the vertical field of view of the first laser radar, and second RV image feature information corresponding to the point cloud data within the vertical field of view of the second laser radar;

[0032] a compression unit, configured to perform image compression processing on the second RV image characteristic information to obtain third RV image characteristic information;

[0033] an obtaining unit, configured to obtain target RV image feature information according to the first RV image feature information and the third RV image feature information;

[0034] The segmentation unit is used to perform semantic segmentation processing on the target RV image feature information using a pre-trained semantic segmentation model to obtain a point cloud semantic segmentation result.

[0035] In a fourth aspect, a model training device is provided, the device comprising:

[0036] An acquisition unit, configured to acquire sample point cloud data; the sample point cloud data is collected by a first laser radar and a second laser radar;

[0037] an extraction unit, configured to respectively extract first depth perspective RV image feature information corresponding to sample point cloud data within the vertical field of view of the first laser radar, and second RV image feature information corresponding to sample point cloud data within the vertical field of view of the second laser radar;

[0038] a compression unit, configured to perform image compression processing on the second RV image characteristic information to obtain third RV image characteristic information;

[0039] an obtaining unit, configured to obtain target RV image feature information according to the first RV image feature information and the third RV image feature information;

[0040] The training unit is used to train the semantic segmentation model to be trained based on the target RV image feature information to obtain a trained semantic segmentation model.

[0041] In a fifth aspect, a computer-readable storage medium is provided, wherein the storage medium stores at least one instruction, and the at least one instruction is loaded and executed by a processor to implement the method of the above-mentioned aspect and any possible implementation method.

[0042] According to a sixth aspect, an electronic device is provided, including:

[0043] at least one processor; and

[0044] a memory communicatively connected to the at least one processor; wherein,

[0045] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any possible implementation manner and the aspects described above.

[0046] In a seventh aspect, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the above-mentioned aspects and any possible implementation method.

[0047] In an eighth aspect, an autonomous driving vehicle is provided, comprising the electronic device as described above.

[0048] The beneficial effects of the technical solution provided by this application include at least:

[0049] It can be seen from the above technical solution that, on the one hand, the embodiment of the present application can obtain the point cloud data to be processed; the point cloud data is collected by the first laser radar and the second laser radar, and then the first depth perspective RV image feature information corresponding to the point cloud data within the vertical field of view of the first laser radar and the second RV image feature information corresponding to the point cloud data within the vertical field of view of the second laser radar can be extracted respectively, and the second RV image feature information is compressed to obtain the third RV image feature information, and the target RV image feature information is obtained according to the first RV image feature information and the third RV image feature information, so that the pre-trained A semantic segmentation model is used to perform semantic segmentation processing on the target RV image feature information to obtain a point cloud semantic segmentation result. Since the second RV image feature information corresponding to the second lidar can be compressed, and then the third RV image feature information is obtained according to the first RV image feature information and the compression processing, the target RV image feature information for semantic segmentation model processing is obtained. This can effectively reduce the amount of information processed by the semantic segmentation model, and can achieve the goal of reducing the time consumption of the point cloud semantic segmentation task of the semantic segmentation model while ensuring the reliability of the point cloud semantic segmentation result in a multi-lidar scenario of autonomous driving, thereby ensuring the reliability and efficiency of the point cloud semantic segmentation processing.

[0050] It can be seen from the above technical solution that, on the other hand, the embodiment of the present application can obtain sample point cloud data collected by the first laser radar and the second laser radar, and then respectively extract the first depth perspective RV image feature information corresponding to the point cloud data within the vertical field of view of the first laser radar, and the second RV image feature information corresponding to the point cloud data within the vertical field of view of the second laser radar, and perform image compression processing on the second RV image feature information to obtain third RV image feature information. According to the first RV image feature information and the third RV image feature information, the target RV image feature information is obtained, so that the semantic segmentation model to be trained can be trained based on the target RV image feature information to obtain a trained semantic segmentation model. Since the target RV image feature information obtained according to the first RV image feature information and the compressed third RV image feature information can be used to train the semantic segmentation model to be trained, it can be achieved in the multi-lidar scenario of autonomous driving. While ensuring the performance of the trained semantic segmentation model, the amount of feature information processed by the semantic segmentation model training is reduced, the time consumption of the semantic segmentation model training is reduced, thereby ensuring the detection performance and efficiency of the semantic segmentation model.

[0051] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0053] FIG1 is a flow chart of a method for semantic segmentation of a point cloud provided by one embodiment of the present application;

[0054] FIG2 is a flow chart of a model training method provided by another embodiment of the present application;

[0055] FIG3 is a schematic diagram of a method for point cloud semantic segmentation provided by another embodiment of the present application;

[0056] FIG4 is a schematic diagram of an application scenario of a method for point cloud semantic segmentation provided by an embodiment of the present application;

[0057] FIG5 is a structural block diagram of an apparatus for point cloud semantic segmentation provided by another embodiment of the present application;

[0058] FIG6 is a structural block diagram of a model training apparatus provided by another embodiment of the present application;

[0059] FIG7 is a block diagram of an electronic device for implementing the method for point cloud semantic segmentation and the method for model training according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0060] The following description of exemplary embodiments of the present application is made in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0061] Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0062] It should be noted that the terminal devices involved in the embodiments of the present application may include but are not limited to mobile phones, personal digital assistants (PDAs), wireless handheld devices, tablet computers and other smart devices; display devices may include but are not limited to personal computers, televisions and other devices with display functions.

[0063] In this document, the term "and / or" simply describes a relationship between related objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this document generally indicates that the related objects are in an "or" relationship.

[0064] Please refer to Figure 1, which shows a flow chart of a method for semantic segmentation of point clouds provided by an embodiment of the present application. The method for semantic segmentation of point clouds may specifically include:

[0065] Step 101: Acquire point cloud data to be processed; the point cloud data is collected by a first laser radar and a second laser radar.

[0066] Step 102: extract first depth perspective RV image feature information corresponding to the point cloud data within the vertical field of view of the first laser radar, and second RV image feature information corresponding to the point cloud data within the vertical field of view of the second laser radar.

[0067] Step 103: Perform image compression processing on the second RV image feature information to obtain third RV image feature information.

[0068] Step 104: Obtain target RV image feature information according to the first RV image feature information and the third RV image feature information.

[0069] Step 105: Using a pre-trained semantic segmentation model, perform semantic segmentation processing on the target RV image feature information to obtain a point cloud semantic segmentation result.

[0070] It should be noted that the first laser radar can be a main laser radar on the vehicle, and the second laser radar can be a blind spot laser radar on the vehicle. Here, the number of the first laser radar can be one, and the number of the second laser radar can be multiple.

[0071] It should be noted that the semantic segmentation model can be a model based on a two-dimensional (2D) convolutional network.

[0072] It should be noted that the point cloud data to be processed may include each frame of original three-dimensional point cloud data collected by the first laser radar and the second laser radar.

[0073] It should be noted that the point cloud coverage generated by a LiDAR typically includes both the horizontal field of view (FOV) and the vertical field of view. The vertical field of view of a LiDAR can be determined by the LiDAR's hardware parameters. For example, hardware parameters may include the number of beams.

[0074] Here, the field of view of the lidar can be an angular range. For example, a lidar with a 32-beam horizontal scanning range of 360 degrees can generate a point cloud with a horizontal field of view of [0, 360] degrees and a vertical field of view of [-16, 15] degrees.

[0075] It should be noted that part or all of the execution entities of steps 101 to 105 may be an application located in the local terminal, or may be a functional unit such as a plug-in or software development kit (SDK) set in the application located in the local terminal, or may be a processing engine located in a network-side server, or may be a distributed system located on the network side, for example, a processing engine or distributed system in a point cloud semantic segmentation platform on the network side, etc. This embodiment does not specifically limit this.

[0076] It is understandable that the application may be a native program (nativeApp) installed on the local terminal, or may be a webpage program (webApp) of a browser on the local terminal, which is not limited in this embodiment.

[0077] In this way, the second RV image feature information corresponding to the second lidar can be compressed, and then the third RV image feature information can be obtained based on the first RV image feature information and the compression processing to obtain the target RV image feature information for semantic segmentation model processing. This can effectively reduce the amount of information processed by the semantic segmentation model, and can achieve the goal of reducing the time consumption of the point cloud semantic segmentation task of the semantic segmentation model while ensuring the reliability of the point cloud semantic segmentation results in the multi-lidar scenario of autonomous driving, thereby ensuring the reliability and efficiency of the point cloud semantic segmentation processing.

[0078] Optionally, in a possible implementation of this embodiment, in step 103, image compression processing can be performed on the second RV image feature information based on a preset width, and then the third RV image feature information can be obtained based on the result of the image compression processing.

[0079] In this implementation, the preset width may be determined based on the resolution and clarity of the second RV image feature information. For example, the higher the resolution and clarity of the second RV image feature information, the smaller the preset width may be. Furthermore, the preset width may also be determined based on actual service requirements.

[0080] In a specific implementation of this method, the preset width can be half the width of the second RV image feature information. The second RV image feature information can be n×m two-dimensional vector information, where n can be vector information representing length, and m can be vector information representing width. First, with respect to the m vector information, the odd-numbered row vector information can be extracted. Second, the extracted odd-numbered row vector information is concatenated in a preset order, and the result of the concatenation is used as the result of the image compression process. Finally, the result of the image compression process is used as the third RV image feature information.

[0081] In another specific implementation of this method, the preset width can be two-thirds the width of the second RV image feature information. The second RV image feature information can be n×m two-dimensional vector information. First, for the m vector information, two rows of vector information can be extracted every other row to obtain extracted vector information. Second, the extracted vector information is concatenated in a preset order, and the concatenated result is used as the result of the image compression process. Finally, the result of the image compression process is used as the third RV image feature information.

[0082] It is understandable that, here, the third RV image feature information may be obtained by utilizing other existing image compression processing methods, and the specific implementation method may not be specifically limited.

[0083] In this way, the second RV image feature information can be compressed in the width direction according to a preset width to obtain compressed third RV image feature information, thereby reducing the amount of information of the RV image feature information in the width direction, so that the amount of information processed by the semantic segmentation model can be reduced subsequently.

[0084] Optionally, in a possible implementation of this embodiment, in step 104, the first RV image feature information and the third RV image feature information may be spliced ​​together, and then the target RV image feature information may be obtained based on the result of the splicing process.

[0085] In a specific implementation process of this implementation, the first RV image feature information and the third RV image feature information may be spliced ​​in the width direction, and then the target RV image feature information may be obtained based on the result of the splicing process.

[0086] Here, the width direction may be the width direction of the RV image feature information. The RV image feature information in the width direction may represent the RV image feature information within a vertical field of view.

[0087] In this way, by splicing the first RV image feature information and the third RV image feature information in the width direction, the RV image feature information corresponding to the compressed original point cloud data can be obtained. While ensuring the integrity of the image information represented by the RV image feature information, the amount of information of the RV image feature information in the width direction is reduced, so that the amount of information processed by the semantic segmentation model can be reduced subsequently.

[0088] It should be noted that the specific implementation process provided in this implementation can be combined with the various specific implementation processes provided in the aforementioned implementations to implement the method for point cloud semantic segmentation of this embodiment. For a detailed description, please refer to the relevant content in the aforementioned implementations and will not be repeated here.

[0089] Optionally, in a possible implementation of this embodiment, in step 102, the point cloud data can be projected to obtain RV image feature information of the point cloud data, and then the first RV image feature information corresponding to the point cloud data within the vertical field of view of the first laser radar and the second RV image feature information corresponding to the point cloud data within the vertical field of view of the second laser radar can be extracted from the RV image feature information.

[0090] In a specific implementation process of this implementation, spherical projection processing may be performed on the point cloud data to obtain RV image feature information of the point cloud data.

[0091] It is understandable that, here, other existing projection processing methods can also be used to obtain RV image feature information of point cloud data, and the specific implementation method is not specifically limited here.

[0092] It should be noted that the specific implementation process provided in this implementation can be combined with the various specific implementation processes provided in the aforementioned implementations to implement the method for point cloud semantic segmentation of this embodiment. For a detailed description, please refer to the relevant content in the aforementioned implementations and will not be repeated here.

[0093] Optionally, in a possible implementation of this embodiment, in step 102, the point cloud data within the vertical field of view of the first lidar may be projected to extract the first RV image feature information, and then the point cloud data within the vertical field of view of the second lidar may be projected to extract the second RV image feature information.

[0094] In a specific implementation process of this implementation, first, spherical projection processing can be performed on the point cloud data within the vertical field of view of the first laser radar to obtain first RV image feature information. Secondly, the first RV image feature information is extracted.

[0095] In another specific implementation of this embodiment, first, spherical projection processing can be performed on the point cloud data within the vertical field of view of the second laser radar to obtain the second RV image feature information. Then, the second RV image feature information is extracted.

[0096] It is understandable that, here, other existing projection processing methods can also be used to obtain the first RV image feature information and the second RV image feature information of the point cloud data, and the specific implementation method is not specifically limited here.

[0097] It should be noted that the specific implementation process provided in this implementation can be combined with the various specific implementation processes provided in the aforementioned implementations to implement the method for point cloud semantic segmentation of this embodiment. For a detailed description, please refer to the relevant content in the aforementioned implementations and will not be repeated here.

[0098] FIG2 is a flow chart of a model training method provided in another embodiment of the present application, as shown in FIG2 .

[0099] Step 201: Acquire sample point cloud data; the sample point cloud data is collected by a first laser radar and a second laser radar.

[0100] Step 202: extract first depth perspective RV image feature information corresponding to the sample point cloud data within the vertical field of view of the first laser radar, and second RV image feature information corresponding to the sample point cloud data within the vertical field of view of the second laser radar.

[0101] Step 203: Perform image compression processing on the second RV image feature information to obtain third RV image feature information.

[0102] Step 204: Obtain target RV image feature information according to the first RV image feature information and the third RV image feature information.

[0103] Step 205: Based on the target RV image feature information, the semantic segmentation model to be trained is trained to obtain a trained semantic segmentation model.

[0104] It should be noted that the semantic segmentation model can be a model based on a two-dimensional (2D) convolutional network.

[0105] It should be noted that by inputting the target RV image feature information obtained based on step 204 into the semantic segmentation model to be trained, a semantic segmentation result can be obtained. If the training termination condition is not met, the semantic segmentation model to be trained can be updated based on the semantic segmentation result and the new target RV image feature information obtained by re-executing steps 201 to 204 until the training termination condition is met to obtain a trained semantic segmentation model.

[0106] It should be noted that part or all of the execution entities of steps 201 to 205 may be an application located in the local terminal, or may also be a functional unit such as a plug-in or software development kit (SDK) set in the application located in the local terminal, or may also be a processing engine located in a network-side server, or may also be a distributed system located on the network side, for example, a processing engine or distributed system in a model training platform on the network side, etc. This embodiment does not specifically limit this.

[0107] It is understandable that the application may be a native program (nativeApp) installed on the local terminal, or may be a webpage program (webApp) of a browser on the local terminal, which is not limited in this embodiment.

[0108] In this way, the semantic segmentation model to be trained can be trained by utilizing the target RV image feature information obtained based on the first RV image feature information and the compressed third RV image feature information. This can achieve the goal of reducing the amount of feature information processed by the semantic segmentation model training while ensuring the performance of the trained semantic segmentation model in a multi-lidar scenario of autonomous driving, thereby reducing the time consumption of the semantic segmentation model training and ensuring the detection performance and efficiency of the semantic segmentation model.

[0109] Optionally, in a possible implementation of this embodiment, in step 202, the point cloud data can be specifically projected to obtain RV image feature information of the point cloud data, and then the first RV image feature information corresponding to the point cloud data within the vertical field of view of the first laser radar and the second RV image feature information corresponding to the point cloud data within the vertical field of view of the second laser radar can be extracted from the RV image feature information.

[0110] In a specific implementation process of this implementation, spherical projection processing may be performed on the point cloud data to obtain RV image feature information of the point cloud data.

[0111] It is understandable that, here, other existing projection processing methods can also be used to obtain RV image feature information of point cloud data, and the specific implementation method is not specifically limited here.

[0112] Optionally, in a possible implementation of this embodiment, in step 202, the point cloud data within the vertical field of view of the first lidar may be projected to extract the first RV image feature information, and then the point cloud data within the vertical field of view of the second lidar may be projected to extract the second RV image feature information.

[0113] In a specific implementation process of this implementation, first, spherical projection processing can be performed on the point cloud data within the vertical field of view of the first laser radar to obtain first RV image feature information. Secondly, the first RV image feature information is extracted.

[0114] In another specific implementation of this embodiment, first, spherical projection processing can be performed on the point cloud data within the vertical field of view of the second laser radar to obtain the second RV image feature information. Then, the second RV image feature information is extracted.

[0115] It is understandable that, here, other existing projection processing methods can also be used to obtain the first RV image feature information and the second RV image feature information of the point cloud data, and the specific implementation method is not specifically limited here.

[0116] Optionally, in a possible implementation of this embodiment, in step 203, image compression processing can be performed on the width of the second RV image feature information based on a preset width, and then the third RV image feature information can be obtained based on the result of the image compression processing.

[0117] In this implementation, the preset width may be determined according to the resolution of the second RV image feature information or according to actual business requirements.

[0118] In a specific implementation of this method, the preset width can be half the width of the second RV image feature information. The second RV image feature information can be n×m two-dimensional vector information, where n can be vector information representing length, and m can be vector information representing width. First, with respect to the m vector information, the odd-numbered row vector information can be extracted. Second, the extracted odd-numbered row vector information is concatenated in a preset order, and the result of the concatenation is used as the result of the image compression process. Finally, the result of the image compression process is used as the third RV image feature information.

[0119] In a specific implementation of this method, the preset width can be two-thirds the width of the second RV image feature information. The second RV image feature information can be n×m two-dimensional vector information. First, for the m vector information, two rows of vector information can be extracted every other row to obtain extracted vector information. Second, the extracted vector information is concatenated in a preset order, and the concatenated result is used as the result of the image compression process. Finally, the result of the image compression process is used as the third RV image feature information.

[0120] It is understandable that, here, the third RV image feature information may be obtained by utilizing other existing image compression processing methods, and the specific implementation method may not be specifically limited.

[0121] It should be noted that the specific implementation process provided in this implementation can be combined with the various specific implementation processes provided in the aforementioned implementation to implement the model training method of this embodiment. For a detailed description, please refer to the relevant content in the aforementioned implementation, which will not be repeated here.

[0122] Optionally, in a possible implementation of this embodiment, in step 104, the first RV image feature information and the third RV image feature information may be spliced ​​together, and then the target RV image feature information may be obtained based on the result of the splicing process.

[0123] In a specific implementation process of this implementation, the first RV image feature information and the third RV image feature information may be spliced ​​in the width direction, and then the target RV image feature information may be obtained based on the result of the splicing process.

[0124] Here, the width direction may be the width direction of the RV image feature information. The RV image feature information in the width direction may represent the RV image feature information in the vertical direction of the vertical field of view range.

[0125] It should be noted that the specific implementation process provided in this implementation can be combined with the various specific implementation processes provided in the aforementioned implementation to implement the model training method of this embodiment. For a detailed description, please refer to the relevant content in the aforementioned implementation, which will not be repeated here.

[0126] In order to better understand the method of the embodiment of the present application, the method of the embodiment of the present application is described below with reference to the accompanying drawings and specific application scenarios.

[0127] FIG3 is a flow chart of a method for semantic segmentation of point clouds provided by another embodiment of the present application. As shown in FIG3 , the method for semantic segmentation of point clouds may specifically include:

[0128] Step 301: Acquire sample point cloud data; the sample point cloud data is collected by a first laser radar and a second laser radar.

[0129] In the application scenario of this embodiment, FIG4 is a schematic diagram of an application scenario of a method for point cloud semantic segmentation provided by another embodiment of the present application, as shown in FIG4 . The first laser radar can be a vehicle-mounted main laser radar, and the second laser radar can be a vehicle-mounted blind spot laser radar. The main laser radar can be a 32-beam laser radar with a horizontal scanning range of 360 degrees, and the vertical field of view of the main laser radar can be [-16, 15] degrees. The blind spot laser radar can be a 64-beam laser radar with a horizontal scanning range of 360 degrees, and the vertical field of view of the blind spot laser radar can be [-73, -10] degrees.

[0130] Step 302: extract first RV image feature information corresponding to the sample point cloud data within the vertical field of view of the first laser radar, and second RV image feature information corresponding to the sample point cloud data within the vertical field of view of the second laser radar.

[0131] Step 303: Perform image compression processing on the second RV image feature information to obtain third RV image feature information.

[0132] Step 304: Obtain target RV image feature information according to the first RV image feature information and the third RV image feature information.

[0133] Step 305: Based on the target RV image feature information, the semantic segmentation model to be trained is trained to obtain a trained semantic segmentation model.

[0134] Step 306: Obtain the point cloud data to be processed.

[0135] In this embodiment, the point cloud data to be processed is collected in real time by the first laser radar and the second laser radar.

[0136] In this embodiment, the vertical field of view of the primary LiDAR can preferably be [-16, 15] degrees, and the vertical field of view of the blind spot LiDAR can preferably be [-73, -10] degrees. The vertical fields of view of the primary LiDAR and the blind spot LiDAR overlap. The primary LiDAR can be used to obtain environmental information at medium and long distances from the vehicle, while the blind spot main LiDAR can be used to obtain environmental information near the vehicle body.

[0137] Step 307: Perform spherical projection processing on the point cloud data within the vertical field of view of the first laser radar to obtain first RV image feature information.

[0138] In this embodiment, the vertical field of view of the first laser radar can be [-16, 15] degrees, and the point cloud data within the range of [-16, 15] degrees can be mapped into an n×32 RV image, that is, the first RV image feature information, through spherical projection.

[0139] It can be understood that the horizontal field of view range of both lidars can be [-16, 15] degrees. Here, n of the RV image can be determined according to actual business needs, for example, it can be 512, etc.

[0140] It can be understood that here, the point cloud data may include all point clouds collected by the first lidar radar and part of the point cloud collected by the second lidar radar.

[0141] Step 308: Perform spherical projection processing on the point cloud data within the vertical field of view of the second laser radar to obtain second RV image feature information.

[0142] Step 309: Perform image compression processing on the width of the second RV image characteristic information to obtain third RV image characteristic information.

[0143] In this embodiment, the vertical field of view of the second laser radar can be [-80, -17] degrees, and the point cloud data within the vertical field of view of [-80, -17] degrees can be mapped into an n×64 RV image, i.e., the second RV image feature information, through spherical projection.

[0144] It can be understood that here, the vertical field of view of the second laser radar selected can be slightly larger than the vertical field of view corresponding to the actual hardware parameters of the second laser radar, so as to avoid the problem of missing point cloud data as much as possible.

[0145] Furthermore, the width of the second RV image characteristic information is compressed to half of the original width to obtain an n×32 RV image, namely, the third RV image characteristic information.

[0146] It is understandable that since the point cloud data within the vertical field of view of the second lidar are point clouds close to the vehicle body, the horizontal area covered by these point cloud data is relatively small, and compression will not cause substantial loss of information in the original point cloud data.

[0147] Step 310: Splice the first RV image feature information and the third RV image feature information to obtain target RV image feature information.

[0148] In this embodiment, the first RV image feature information and the third RV image feature information are spliced ​​in width to obtain target RV image feature information. Here, the first RV image feature information may be an n×32 RV image, the third RV image feature information may be an n×32 RV image, and the obtained target RV image feature information may be an n×64 RV image.

[0149] Step 311: Use the pre-trained semantic segmentation model to perform semantic segmentation processing on the target RV image feature information to obtain the result of point cloud semantic segmentation.

[0150] In this embodiment, the target RV image feature information is input into a pre-trained semantic segmentation model, and the final point cloud semantic segmentation result is output.

[0151] Based on the technical solution of this embodiment, in the semantic segmentation task of the multi-lidar scenario of autonomous driving, the second RV image feature information corresponding to the blind lidar can be compressed, and then the third RV image feature information can be obtained based on the first RV image feature information and the compression processing to obtain the target RV image feature information for semantic segmentation model processing. This can not only ensure that the valid point cloud within the field of view of the multi-lidar is semantically segmented, but also greatly reduce the processing time of the semantic segmentation model. In actual applications, the time consumption can be reduced by 1 / 3.

[0152] It should be noted that for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

[0153] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0154] FIG5 shows a block diagram of a device for semantic segmentation of a point cloud provided by an embodiment of the present application, as shown in FIG5 . The device 500 for semantic segmentation of a point cloud in this embodiment may include an acquisition unit 501, an extraction unit 502, a compression unit 503, an acquisition unit 504, and a segmentation unit 505. The acquisition unit 501 is used to acquire point cloud data to be processed; the point cloud data is collected by a first laser radar and a second laser radar; the extraction unit 502 is used to extract the first depth perspective RV image feature information corresponding to the point cloud data within the vertical field of view of the first laser radar, and the second RV image feature information corresponding to the point cloud data within the vertical field of view of the second laser radar; the compression unit 503 is used to perform image compression processing on the second RV image feature information to obtain third RV image feature information; the acquisition unit 504 is used to obtain target RV image feature information based on the first RV image feature information and the third RV image feature information; the segmentation unit 505 is used to perform semantic segmentation processing on the target RV image feature information using a pre-trained semantic segmentation model to obtain a point cloud semantic segmentation result.

[0155] It should be noted that part or all of the point cloud semantic segmentation device of this embodiment can be an application located in the local terminal, or it can also be a functional unit such as a plug-in or software development kit (SDK) set in the application located in the local terminal, or it can also be a processing engine located in the network side server, or it can also be a distributed system located on the network side, for example, a processing engine or distributed system in the point cloud semantic segmentation platform on the network side, etc. This embodiment does not specifically limit this.

[0156] It is understandable that the application may be a native program (nativeApp) installed on the local terminal, or may be a webpage program (webApp) of a browser on the local terminal, which is not limited in this embodiment.

[0157] Optionally, in a possible implementation of this embodiment, the compression unit 503 can be specifically used to perform image compression processing on the second RV image feature information based on a preset width; and obtain third RV image feature information based on the result of the image compression processing.

[0158] Optionally, in a possible implementation of this embodiment, the obtaining unit 504 may be specifically configured to perform splicing processing on the first RV image feature information and the third RV image feature information; and obtain the target RV image feature information based on a result of the splicing processing.

[0159] Optionally, in a possible implementation of this embodiment, the extraction unit 502 can be specifically used to perform projection processing on the point cloud data to obtain RV image feature information of the point cloud data; and extract first RV image feature information corresponding to the point cloud data within the vertical field of view of the first laser radar and second RV image feature information corresponding to the point cloud data within the vertical field of view of the second laser radar from the RV image feature information.

[0160] Optionally, in a possible implementation of this embodiment, the extraction unit 502 can be specifically used to perform projection processing on the point cloud data within the vertical field of view of the first lidar to extract the first RV image feature information; and perform projection processing on the point cloud data within the vertical field of view of the second lidar to extract the second RV image feature information.

[0161] In this embodiment, the point cloud data to be processed can be obtained by the acquisition unit; the point cloud data is collected by the first laser radar and the second laser radar, and then the extraction unit can respectively extract the first depth perspective RV image feature information corresponding to the point cloud data within the vertical field of view of the first laser radar, and the second RV image feature information corresponding to the point cloud data within the vertical field of view of the second laser radar, and the compression unit performs image compression processing on the second RV image feature information to obtain the third RV image feature information, and the acquisition unit obtains the target RV image feature information according to the first RV image feature information and the third RV image feature information, so that the segmentation unit can use A pre-trained semantic segmentation model is used to perform semantic segmentation processing on the target RV image feature information to obtain the result of point cloud semantic segmentation. Since the second RV image feature information corresponding to the second lidar can be compressed, and then the third RV image feature information is obtained based on the first RV image feature information and the compression processing, the target RV image feature information for semantic segmentation model processing is obtained. This can effectively reduce the amount of information processed by the semantic segmentation model, and can achieve the goal of reducing the time consumption of the point cloud semantic segmentation task of the semantic segmentation model while ensuring the reliability of the point cloud semantic segmentation results in the multi-lidar scenario of autonomous driving, thereby ensuring the reliability and efficiency of the point cloud semantic segmentation processing.

[0162] FIG6 shows a structural block diagram of a model training device provided by an embodiment of the present application, as shown in FIG6 . The model training device 600 of this embodiment may include an acquisition unit 601, an extraction unit 602, a compression unit 603, an acquisition unit 604, and a training unit 605. The acquisition unit 601 is used to acquire sample point cloud data; the sample point cloud data is collected by a first laser radar and a second laser radar; the extraction unit 602 is used to respectively extract the first depth perspective RV image feature information corresponding to the sample point cloud data within the vertical field of view of the first laser radar, and the second RV image feature information corresponding to the sample point cloud data within the vertical field of view of the second laser radar; the compression unit 603 is used to perform image compression processing on the second RV image feature information to obtain third RV image feature information; the acquisition unit 604 is used to obtain target RV image feature information based on the first RV image feature information and the third RV image feature information; the training unit 605 is used to train the semantic segmentation model to be trained based on the target RV image feature information to obtain a trained semantic segmentation model.

[0163] It should be noted that part or all of the model training device of this embodiment can be an application located in the local terminal, or it can also be a functional unit such as a plug-in or software development kit (SDK) set in the application located in the local terminal, or it can also be a processing engine located in the network side server, or it can also be a distributed system located on the network side, for example, a processing engine or distributed system in the model training platform on the network side, etc. This embodiment does not specifically limit this.

[0164] It is understandable that the application may be a native program (nativeApp) installed on the local terminal, or may be a webpage program (webApp) of a browser on the local terminal, which is not limited in this embodiment.

[0165] In this embodiment, sample point cloud data can be obtained by an acquisition unit; the sample point cloud data is collected by a first laser radar and a second laser radar, and then the extraction unit can respectively extract the first depth perspective RV image feature information corresponding to the sample point cloud data within the vertical field of view of the first laser radar, and the second RV image feature information corresponding to the sample point cloud data within the vertical field of view of the second laser radar, and the compression unit performs image compression processing on the second RV image feature information to obtain third RV image feature information. The acquisition unit obtains target RV image feature information based on the first RV image feature information and the third RV image feature information, so that the training unit can train the semantic segmentation model to be trained based on the target RV image feature information to obtain a trained semantic segmentation model. Since the target RV image feature information obtained according to the first RV image feature information and the compressed third RV image feature information can be used to train the semantic segmentation model to be trained, it is possible to achieve in a multi-lidar scenario of autonomous driving, while ensuring the performance of the trained semantic segmentation model, reduce the amount of feature information processed by the semantic segmentation model training, reduce the time consumption of the semantic segmentation model training, and thus ensure the detection performance and efficiency of the semantic segmentation model.

[0166] In the technical solution of this application, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved, such as user images and attribute data, comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0167] According to an embodiment of the present application, the present application also provides an electronic device, a readable storage medium and a computer program product.

[0168] According to an embodiment of the present application, an autonomous driving vehicle including the provided electronic device is further provided. The autonomous driving vehicle may include an L2 or higher level unmanned vehicle, for example, an unmanned logistics vehicle, an autonomous driving logistics vehicle, etc.

[0169] FIG7 shows a schematic block diagram of an example electronic device 700 that can be used to implement an embodiment of the present application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.

[0170] As shown in Figure 7, electronic device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. In RAM 703, various programs and data required for the operation of electronic device 700 can also be stored. Computing unit 701, ROM 702, and RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to bus 704.

[0171] Multiple components in the electronic device 700 are connected to the I / O interface 705, including an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a magnetic disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the electronic device 700 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0172] The computing unit 701 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as the point cloud semantic segmentation method and the model training method. For example, in some embodiments, the point cloud semantic segmentation method and the model training method can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the point cloud semantic segmentation method and the model training method described above can be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform the point cloud semantic segmentation method and the model training method in any other appropriate manner (for example, by means of firmware).

[0173] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0174] The program code for implementing the methods of the present application can be written in any combination of one or more programming languages. Such program code can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the program code is executed by the processor or controller, the functions / operations specified in the flow charts and / or block diagrams are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0175] In the context of the present application, a machine-readable medium can be a tangible medium that can contain or store a program for use by an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0176] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0177] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0178] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0179] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this application can be achieved. This is not a limitation herein.

[0180] The above specific embodiments do not constitute a limitation on the scope of protection of this application. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application shall be included within the scope of protection of this application.

Claims

1. A method for point cloud semantic segmentation, characterized in that, The method includes: Obtaining point cloud data to be processed; the point cloud data is collected by a first lidar and a second lidar; Respectively extracting first depth perspective RV image feature information corresponding to the point cloud data within the vertical field of view of the first lidar, and second RV image feature information corresponding to the point cloud data within the vertical field of view of the second lidar; Performing image compression processing on the second RV image feature information to obtain third RV image feature information; Obtaining target RV image feature information according to the first RV image feature information and the third RV image feature information; Using a pre-trained semantic segmentation model to perform semantic segmentation processing on the target RV image feature information to obtain the result of point cloud semantic segmentation.

2. The method according to claim 1, wherein The performing image compression processing on the second RV image feature information to obtain third RV image feature information includes: Performing image compression processing on the second RV image feature information based on a preset width; Obtaining third RV image feature information based on the result of the image compression processing.

3. The method according to claim 1, wherein The obtaining target RV image feature information according to the first RV image feature information and the third RV image feature information includes: Performing splicing processing on the first RV image feature information and the third RV image feature information; Obtaining the target RV image feature information based on the result of the splicing processing.

4. The method according to claim 1, wherein The respectively extracting first depth perspective RV image feature information corresponding to the point cloud data within the vertical field of view of the first lidar, and second RV image feature information corresponding to the point cloud data within the vertical field of view of the second lidar includes: Performing projection processing on the point cloud data to obtain RV image feature information of the point cloud data; Respectively extracting first RV image feature information corresponding to the point cloud data within the vertical field of view of the first lidar and second RV image feature information corresponding to the point cloud data within the vertical field of view of the second lidar from the RV image feature information.

5. The method according to claim 1, wherein The respectively extracting first depth perspective RV image feature information corresponding to the point cloud data within the vertical field of view of the first lidar, and second RV image feature information corresponding to the point cloud data within the vertical field of view of the second lidar includes: Performing projection processing on the point cloud data within the vertical field of view of the first lidar to extract first RV image feature information; Performing projection processing on the point cloud data within the vertical field of view of the second lidar to extract second RV image feature information.

6. A method for model training, characterized in that, The method includes: Obtaining sample point cloud data; the sample point cloud data is collected by a first lidar and a second lidar; Respectively extracting first depth perspective RV image feature information corresponding to the sample point cloud data within the vertical field of view of the first lidar, and second RV image feature information corresponding to the sample point cloud data within the vertical field of view of the second lidar; Performing image compression processing on the second RV image feature information to obtain third RV image feature information; Obtain target RV image feature information according to the first RV image feature information and the third RV image feature information; Based on the target RV image feature information, train the semantic segmentation model to be trained to obtain a trained semantic segmentation model.

7. An apparatus for point cloud semantic segmentation, characterized in that, The device includes: An acquisition unit, configured to acquire point cloud data to be processed; the point cloud data is acquired by a first lidar and a second lidar; An extraction unit, configured to extract first depth-view RV image feature information corresponding to the point cloud data within the vertical field of view of the first lidar and second RV image feature information corresponding to the point cloud data within the vertical field of view of the second lidar, respectively; A compression unit, configured to perform image compression processing on the second RV image feature information to obtain third RV image feature information; An obtaining unit, configured to obtain target RV image feature information according to the first RV image feature information and the third RV image feature information; A segmentation unit, configured to perform semantic segmentation processing on the target RV image feature information by using a pre-trained semantic segmentation model to obtain a result of point cloud semantic segmentation.

8. An apparatus for model training, characterized in that, The device includes: An acquisition unit, configured to acquire sample point cloud data; the sample point cloud data is acquired by a first lidar and a second lidar; An extraction unit, configured to extract first depth-view RV image feature information corresponding to the sample point cloud data within the vertical field of view of the first lidar and second RV image feature information corresponding to the sample point cloud data within the vertical field of view of the second lidar, respectively; A compression unit, configured to perform image compression processing on the second RV image feature information to obtain third RV image feature information; An obtaining unit, configured to obtain target RV image feature information according to the first RV image feature information and the third RV image feature information; A training unit, configured to train the semantic segmentation model to be trained based on the target RV image feature information to obtain a trained semantic segmentation model.

9. An electronic device, including: At least one processor; And A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method according to any one of claims 1-6.

10. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to execute the method according to any one of claims 1-6.

11. A computer program product, including a computer program, where the computer program, when executed by a processor, implements the method according to any one of claims 1-6.

12. An autonomous vehicle, including the electronic device according to claim 9.

Images