Point Cloud Data Processing Method, Apparatus, Device and Medium

Abstract

The present invention discloses a method, apparatus, device and medium for processing point cloud data. The method for processing point cloud data includes: respectively dividing each point in the point cloud into a plurality of voxels, and obtaining the voxel features of each voxel according to the point features of the points included in each voxel; projecting each voxel onto a preset image, and obtaining the pixel features of each voxel in the preset image according to the projection result; performing feature splicing on the voxel features and the pixel features to form voxel splicing features corresponding to the voxels. This solution can improve the feature expression ability of voxels and enhance the data processing effect of point cloud data on the premise of low memory occupancy.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and particularly to a method, device, equipment and medium for processing point cloud data. Background Art

[0002] Radar has been widely used in intelligent perception tasks such as autonomous driving, such as point cloud semantic segmentation and target detection.

[0003] Currently, when performing classification perception tasks, in order to reduce the complexity of point cloud feature recognition, the point cloud is first voxelized, and then features are extracted from the voxels, such as using 2D networks and 3D networks to extract features of the voxels. Among them, 2D networks, such as Polar-Net (polar coordinate projection network), use 2D convolution to extract features. 3D networks, such as DS-Net (dense scale single-column neural network), directly use 3D convolution to extract features for each voxel.

[0004] Although 2D networks have the advantage of low latency, their accuracy is limited, and the discrimination of voxels in the height space based on the extracted features is poor. The accuracy of 3D networks is better than that of 2D networks, but they are time-consuming and have a large memory footprint. Summary of the Invention

[0005] The present invention provides a method, device, equipment and medium for processing point cloud data, which can improve the feature expression ability of voxels and the data processing effect of point cloud data on the premise of low memory occupancy.

[0006] According to one aspect of the present invention, there is provided a method for processing point cloud data, including:

[0007] Dividing each point in the point cloud into a plurality of voxels respectively, and obtaining the voxel feature of each voxel according to the point feature of the points included in each voxel;

[0008] Projecting each voxel onto a preset image, and obtaining the pixel feature of each voxel in the preset image according to the projection result;

[0009] Performing feature splicing on the voxel feature and the pixel feature to form a voxel splicing feature corresponding to the voxel.

[0010] According to another aspect of the present invention, there is provided a device for processing point cloud data, including:

[0011] A voxel feature acquisition module, configured to divide each point in the point cloud into a plurality of voxels respectively, and obtain the voxel feature of each voxel according to the point feature of the points included in each voxel;

[0012] A pixel feature acquisition module, configured to project each voxel onto a preset image, and obtain the pixel features of each voxel in the preset image according to the projection result;

[0013] A feature splicing module, configured to splice the voxel features and the pixel features to form voxel splicing features corresponding to the voxels.

[0014] According to another aspect of the present invention, there is provided an electronic device, where the electronic device includes:

[0015] At least one processor; and

[0016] A memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the method for processing point cloud data according to any embodiment of the present invention.

[0018] According to another aspect of the present invention, there is provided a computer-readable storage medium, where the computer-readable storage medium stores computer instructions, and the computer instructions are used to implement the method for processing point cloud data according to any embodiment of the present invention when executed by a processor.

[0019] The technical solution of this embodiment divides each point in the point cloud into multiple voxels respectively, and obtains the voxel features of each voxel according to the point features of the points included in each voxel. Furthermore, each voxel is projected onto a preset image, and according to the projection result, the pixel features of each voxel in the preset image are obtained, so as to splice the voxel features and the pixel features to form voxel splicing features corresponding to the voxels. Fusing the voxel features and the pixel features can improve the expression ability of the voxel features, and the voxel splicing features can be used to quickly identify and classify the targets in the point cloud, solving the problems in the prior art that the features extracted based on the 2D network have poor discrimination for voxels, and the 3D network has large memory occupation and serious time consumption when identifying the features of voxels, improving the expression ability of the voxel features and the data processing effect of the point cloud data.

[0020] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present invention, nor is it used to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. Description of the Drawings

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the accompanying drawings required for the description of the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can be obtained based on these drawings.

[0022] Figure 1 It is a flowchart of a method for processing point cloud data provided by an embodiment of the present invention;

[0023] Figure 2 It is a flowchart of another method for processing point cloud data provided by an embodiment of the present invention;

[0024] Figure 3 It is a flowchart of a panoramic segmentation method for three-dimensional point cloud data provided by an embodiment of the present invention;

[0025] Figure 4 It is a flowchart of another panoramic segmentation method for three-dimensional point cloud data provided by an embodiment of the present invention;

[0026] Figure 5 It is a schematic diagram of a field of view space provided by an embodiment of the present invention;

[0027] Figure 6 It is a schematic diagram of a semantic segmentation result provided by an embodiment of the present invention;

[0028] Figure 7 It is a point cloud schematic diagram of an offset point provided by an embodiment of the present invention;

[0029] Figure 8 It is a pseudo heat map provided by an embodiment of the present invention;

[0030] Figure 9 It is a schematic diagram of an instance ID prediction result provided by an embodiment of the present invention;

[0031] Figure 10 It is a result display diagram after central recombination provided by an embodiment of the present invention;

[0032] Figure 11 It is a flowchart of data processing of a panoramic segmentation system provided by an embodiment of the present invention;

[0033] Figure 12 It is a schematic diagram of a panoramic segmentation device for three-dimensional point cloud data provided by an embodiment of the present invention;

[0034] Figure 13 It is another panoramic segmentation system provided by the present invention;

[0035] Figure 14It is an analysis line chart of the target neighbor voxel values provided by an embodiment of the present invention;

[0036] Figure 15 It is a flowchart of panoramic segmentation provided by an embodiment of the present invention;

[0037] Figure 16 It is a schematic structural diagram of a point cloud data processing device provided by an embodiment of the present invention;

[0038] Figure 17 It shows a schematic structural diagram of an electronic device that can be used to implement the embodiments of the present invention. Detailed implementation manners

[0039] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0040] It should be noted that the terms "first", "second", "target", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and do not have to be used to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device including a series of steps or units does not have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0041] Figure 1 It is a flowchart of a point cloud data processing method provided by an embodiment of the present invention. This embodiment is applicable to the situation of accurately identifying the features of voxels and performing feature fusion. This method can be executed by a point cloud data processing device, which can be implemented in the form of hardware and / or software, and the point cloud data processing device can be configured in an electronic device.

[0042] As Figure 1 shown, the method includes:

[0043] S110. Divide each point in the point cloud into multiple voxels respectively, and obtain the voxel features of each voxel according to the point features of the points included in each voxel.

[0044] Point features can be features of points in a point cloud. Optionally, point features can include, but are not limited to, the point's position (e.g., x-axis, y-axis, and z-axis coordinates in a three-dimensional coordinate system) and intensity. Voxel features can be features of voxels determined based on point features and used for feature fusion. Point features and voxel features share at least one dimension.

[0045] In an embodiment of the present invention, the point features of each point in the point cloud can be first obtained, and then the space where the point cloud is located can be divided into voxels, so that each point in the point cloud is divided into multiple voxels, and the point features of the points divided into each voxel are further processed to obtain the voxel features corresponding to each voxel.

[0046] S120 , projecting each voxel into a preset image, and obtaining pixel features of each voxel in the preset image based on the projection result.

[0047] Among them, the preset image can be a pre-set two-dimensional image with a required number of pixels. Exemplarily, the preset image can be a two-dimensional image of H*W, where H represents the number of pixels in the horizontal direction and W represents the number of pixels in the vertical direction. The projection result can be a top-down projection image obtained after each voxel is projected from the preset image. The pixel features can be determined based on the projection results, and the features included in the pixels in the preset image are used for feature fusion. Optionally, the pixel features can include a first pixel feature and a second pixel feature. The first pixel feature and the second pixel feature have the same feature dimension, and the specific features included can be different or partially the same. Exemplarily, the first pixel feature can include feature 1, feature 2 and feature 3, and the second pixel feature can include feature 1, feature 3 and feature 4.

[0048] In an embodiment of the present invention, each voxel may be projected onto a preset image to obtain a projection result, and then the pixel features of each voxel in the preset image may be determined based on the voxels and voxel features of each pixel projected onto the preset image.

[0049] Optionally, the first pixel feature and the second pixel feature of each voxel in the preset image may be obtained according to application requirements and projection results.

[0050] S130 , performing feature splicing on the voxel feature and the pixel feature to form a voxel splicing feature corresponding to the voxel.

[0051] Among them, feature splicing can be achieved by fusing the pixel features and voxel features of the same voxel. The voxel splicing feature can be the result of feature splicing. Each voxel has a corresponding voxel splicing feature. Optionally, the voxel splicing feature can include a first voxel splicing feature and a second voxel splicing feature. The first voxel splicing feature can be the result of feature splicing between the voxel feature and the first pixel feature. The second voxel splicing feature can be the result of feature splicing between the voxel feature and the second pixel feature.

[0052] In the embodiments of the present invention, the pixel features included in each pixel of a preset image can be respectively assigned to the voxels projected onto each pixel, and then the voxel features and pixel features belonging to the same voxel are subjected to feature splicing to form voxel splicing features respectively corresponding to each voxel.

[0053] Optionally, the first pixel feature and the second pixel feature of the pixels of the preset image can be assigned to the voxels projected onto the corresponding pixels, and then the voxel features and the first pixel features belonging to the same voxel are subjected to feature splicing to form first voxel splicing features respectively corresponding to each voxel, and the voxel features and the second pixel features belonging to the same voxel can also be subjected to feature splicing to form second voxel splicing features respectively corresponding to each voxel.

[0054] In this solution, the pixel features can facilitate the distinction of voxels projected to different positions of the preset image, and the voxel features can distinguish different voxels in the height space because they carry the independent features of the voxels. By performing feature splicing on the voxel features and pixel features, the obtained voxel splicing features can characterize the features of the voxels from multiple dimensions, that is, obtain the features of the voxels with stronger distinguishability. When performing panoramic segmentation on the point cloud using the voxel splicing features with stronger distinguishability, the accuracy of panoramic segmentation can be guaranteed.

[0055] The technical solution of this embodiment divides each point in the point cloud into multiple voxels respectively, obtains the voxel features of each voxel according to the point features of the points included in each voxel, then projects each voxel onto the preset image, and obtains the pixel features of each voxel in the preset image according to the projection result, so as to perform feature splicing on the voxel features and pixel features to form voxel splicing features corresponding to the voxels. Fusing the voxel features and pixel features can improve the expression ability of the features of the voxels, and the voxel splicing features can be used to quickly identify and classify the targets in the point cloud, solving the problems in the prior art that the features extracted based on the 2D network have poor distinguishability for voxels, and the 3D network has large memory occupation and serious time consumption when identifying the features of voxels, improving the expression ability of the features of the voxels and the data processing effect of the point cloud data.

[0056] Figure 2The flowchart of another method for processing point cloud data provided by an embodiment of the present invention. The embodiment of the present invention is a specific implementation based on the above embodiment. In the embodiment of the present invention, a specific optional implementation manner for obtaining the voxel feature of each voxel according to the point feature of the points included in each voxel is given. As Figure 2 shown, the method includes:

[0057] S210. Respectively divide each point in the point cloud into a plurality of voxels, and obtain the voxel feature of each voxel according to the point feature of the points included in each voxel.

[0058] In an optional embodiment of the present invention, S210 may specifically include:

[0059] S211. Respectively divide each point in the point cloud into a plurality of voxels, and generate a first matrix diagram to be processed according to the point feature of the points included in each voxel and the number of voxels.

[0060] Among them, the first matrix diagram to be processed may be a matrix set determined according to the point feature of the points included in each voxel and the number of voxels.

[0061] In the embodiment of the present invention, after respectively dividing each point in the point cloud into a plurality of voxels, a first matrix diagram to be processed may be further generated according to the point feature of the points included in each voxel, the number of points included in each voxel, and the feature dimension of the point feature.

[0062] Optionally, the first matrix diagram to be processed may be a matrix set. N represents the number of voxels, represents the number of points included in the voxel, and K represents the feature dimension of the point feature.

[0063] S212. Use a multi-layer neural network and a max pooling layer to perform feature dimension processing on the first matrix diagram to be processed, and obtain the voxel feature of each voxel.

[0064] In the embodiment of the present invention, a multi-layer neural network may be used to perform dimension elevation processing on the feature dimension of the point feature in the first matrix diagram to be processed. Then, the number of points included in each voxel in the dimension elevation processing result is compressed through the max pooling layer, and further, the feature dimension of the point feature in the dimension elevation processing result after compression is compressed through the multi-layer neural network to obtain the voxel feature of each voxel.

[0065] S220. Project each voxel onto a preset image, and obtain the pixel feature of each voxel in the preset image according to the projection result.

[0066] In an alternative embodiment of the present invention, according to the projection result, the pixel features of each voxel in the preset image are obtained, including: obtaining a second matrix diagram to be processed according to the projection result; using a multi-layer neural network and a max pooling layer to process the feature dimension of the second matrix diagram to be processed to obtain a pixel feature diagram to be processed; obtaining a two-dimensional semantic segmentation network; and inputting the pixel feature diagram to be processed into the two-dimensional semantic segmentation network to obtain the pixel features of each voxel in the preset image.

[0067] Among them, the second matrix diagram to be processed may be a matrix set including the number of voxels falling into the pixels and voxel features determined according to the projection result. Optionally, the second matrix diagram to be processed may be matrix set, H represents the number of pixel points in the horizontal direction, W represents the number of pixel points in the vertical direction, represents the number of voxels falling into the pixel, and J represents the dimension of the voxel feature. The pixel feature diagram to be processed may be a two-dimensional feature diagram obtained by processing the feature dimension of the second matrix diagram to be processed. The two-dimensional semantic segmentation network may be a neural network for semantic segmentation, such as U-Net, etc.

[0068] In the embodiment of the present invention, a second matrix diagram to be processed may be generated according to the voxels projected onto each pixel in the projection result, the number of pixels in the preset image, and the voxel features. Then, a multi-layer neural network is used to perform dimension elevation processing on the dimension of the voxel features in the second matrix diagram to be processed. Then, the max pooling layer is used to compress the number of voxels projected onto each pixel in the result of the dimension elevation processing to obtain a two-dimensional feature diagram. Further, a two-dimensional semantic segmentation network is obtained, and the two-dimensional feature diagram is input into the two-dimensional semantic segmentation network. The features of the pixels under the top-down view after spatial interaction are extracted through the two-dimensional semantic segmentation network, so as to obtain the pixel features of each voxel in the preset image according to the output result of the two-dimensional semantic segmentation network.

[0069] Optionally, the features for semantic segmentation under the top-down view may be extracted through the two-dimensional semantic segmentation network, so as to determine the first pixel features of each voxel in the preset image according to the feature diagram output by the two-dimensional semantic segmentation network. The features for instance segmentation under the top-down view may also be extracted to determine the second pixel features of each voxel in the preset image according to the feature diagram output by the two-dimensional semantic segmentation network.

[0070] S230. Perform feature splicing on the voxel features and the pixel features to form voxel splicing features corresponding to the voxels.

[0071] In an alternative embodiment of the present invention, the method for processing point cloud data may further include: inputting the voxel splicing features corresponding to the voxels into a semantic classifier to obtain voxels with semantic types; and obtaining a point cloud with semantic types according to the voxels with semantic types and the points in the voxel point cloud.

[0072] Among them, the semantic classifier can be used for semantic segmentation.

[0073] In an embodiment of the present invention, the voxel splicing features corresponding to each voxel can be respectively input into the semantic classifier to obtain voxels with semantic types, and then the points in the point cloud within the same voxel are given the same semantic type as the voxel to which they belong, so that the points in the point cloud all have semantic types, that is, the point cloud can include the voxel splicing features of the voxels.

[0074] Optionally, the voxel splicing feature includes a first voxel splicing feature. The first voxel splicing features corresponding to each voxel can be input into the semantic classifier to obtain voxels with semantic types, and the points in the point cloud within the same voxel are given the same semantic type as the voxel to which they belong. At this time, the point cloud includes the first voxel splicing features of the voxels.

[0075] In an optional embodiment of the present invention, after obtaining the voxels with semantic types, it may further include: obtaining target voxels according to the voxels with semantic types; obtaining the target voxel splicing features of the target voxels based on the voxel splicing features and the target voxels; and performing instance segmentation on the target voxels according to the target voxel splicing features.

[0076] Among them, the target voxel can be a voxel that needs to be instance-segmented and is different from the background. Exemplarily, in the field of automotive autonomous driving, the target voxel can be a voxel representing a vehicle and / or a person. The specific entity type represented by the target voxel in the embodiment of the present invention is not limited. The target voxel splicing feature can be a voxel splicing feature corresponding to the target voxel. Optionally, the voxel splicing feature includes a second voxel splicing feature, and the target voxel splicing feature corresponding to the target voxel is formed based on the second voxel splicing feature and the target voxel.

[0077] In an embodiment of the present invention, the target voxels that need to be instance-segmented can be determined from all the voxels according to the semantic types of the voxels, and then the voxel splicing features corresponding to the target voxels are screened out from the voxel splicing features, that is, the target voxel splicing features are screened out, and further, instance segmentation is performed on the target voxels corresponding to the target voxel splicing features based on the target voxel splicing features.

[0078] Optionally, the target voxels can be determined according to the semantic types of the voxels, and then the features corresponding to the target voxels are screened out from the second voxel splicing features, so as to screen out the target voxel splicing features, and further, instance segmentation is performed on the target voxels corresponding to the target voxel splicing features based on the target voxel splicing features.

[0079] In an alternative embodiment of the present invention, performing instance segmentation on a target voxel according to the target voxel stitching feature may include: obtaining a target neighbor voxel that matches the target voxel based on the nearest neighbor node algorithm; obtaining a target neighbor voxel stitching feature of the target neighbor voxel based on the voxel stitching feature; creating a local attention matrix map according to the target voxel stitching feature and the target neighbor voxel stitching feature; obtaining an optimized voxel stitching feature of the target voxel according to the local attention matrix map; and performing instance segmentation on the target voxel according to the optimized voxel stitching feature.

[0080] Among them, the nearest neighbor node algorithm may be an algorithm for data classification based on neighboring samples, such as the K-NearestNeighbor algorithm, etc. The target neighbor voxel may be a target voxel within a certain spatial range centered on the target voxel. The target neighbor voxel stitching feature may be a voxel stitching feature corresponding to the target neighbor voxel. Optionally, the target neighbor voxel stitching feature may be a second voxel stitching feature corresponding to the target neighbor voxel. The local attention matrix map may be an attention matrix generated according to the target voxel stitching feature and the target neighbor voxel stitching feature. The optimized voxel stitching feature may be a voxel stitching feature determined according to the local attention matrix map. Optionally, the local attention matrix map may be an attention matrix generated according to the second voxel stitching feature corresponding to the target neighbor voxel and the second voxel stitching feature corresponding to the target voxel.

[0081] In an embodiment of the present invention, a target neighbor voxel adjacent to the target voxel may be obtained based on the nearest neighbor node algorithm, and then a target neighbor voxel stitching feature corresponding to the target neighbor voxel may be obtained from the voxel stitching features, so that the target neighbor voxel stitching feature and the target voxel stitching feature are input into a transform. The transform generates a local attention matrix map according to the target neighbor voxel stitching feature and the target voxel stitching feature, so as to establish a similarity relationship between the target voxel and the adjacent target neighbor voxels through the local attention matrix map, and then outputs an optimized voxel stitching feature corresponding to the target voxel. Further, instance segmentation is performed on the target voxel corresponding to the optimized voxel stitching feature based on the optimized voxel stitching feature.

[0082] Optionally, after obtaining the target neighbor voxel, a target neighbor voxel stitching feature corresponding to the target neighbor voxel may be obtained from the second voxel stitching features, so that the target neighbor voxel stitching feature and the second voxel stitching feature corresponding to the target voxel are input into a transform. The transform generates a local attention matrix map according to the second voxel stitching feature corresponding to the target neighbor voxel and the second voxel stitching feature corresponding to the target voxel.

[0083] In an alternative embodiment of the present invention, performing instance segmentation on a target voxel according to the target voxel stitching feature includes: inputting the target voxel stitching feature into a trained offset prediction neural network to obtain the offset of the target voxel; obtaining an offset voxel based on the offset and the target voxel; performing a top-down projection of the offset voxel onto a target preset image, and obtaining the pixel value of each pixel according to the number of offset voxels in each pixel of the target preset image; obtaining the pixel with the highest pixel value in a preset area according to the pixel value; determining a target center according to the pixel with the highest pixel value and assigning a matching instance identifier to the target center; obtaining the instance identifier of the target voxel belonging to the target center according to the instance identifier of the target center.

[0084] Among them, the offset prediction neural network can be used to determine the offset of each voxel when the voxels in the point cloud are centroided. The specific type of the offset prediction neural network is not limited in the embodiments of the present invention. The offset voxel can be a voxel obtained by offsetting the position of the target voxel according to the offset. The target preset image can be a pre-set two-dimensional image, and the size of the pre-set image can be set by itself according to the display needs. Optionally, the size and the number of pixels of the target preset image and the preset image can be the same or different. The preset area can be an area in the target preset image. Optionally, the target preset image can be divided into multiple preset areas as needed, or the entire target preset image can be used as a preset area. Exemplarily, assuming that the size of the target preset image is 3dm * 7dm, if the target preset image is divided into 100 preset areas with the same size, the size of the preset area is 0.3dm * 0.7dm. The target center can be the position of the pixel with the highest pixel value in the target preset image in the preset area, which is used to represent the instance center. Optionally, when the target preset image is divided into one preset area, the target center is unique, and when the target preset image is divided into multiple preset areas, there is a target center in each preset area. The instance identifier can be a mark used to distinguish different instances.

[0085] In an embodiment of the present invention, the target voxel stitching feature, or the voxel carrying the target voxel stitching feature, can be first input into the trained offset prediction neural network, and the center clustering of each target voxel is performed according to the offset prediction neural network to obtain the offset of each target voxel. Then, according to the offset matching each target voxel, position offset processing is respectively performed on each target voxel to obtain each offset voxel. After obtaining the offset voxels, a target preset image can be acquired as needed, and then the offset voxels are orthographically projected onto the target preset image. Thus, according to the number of offset voxels that each pixel in the target preset image falls into during the orthographic projection, the pixel value of each pixel in the target preset image is determined. Further, according to the pixel value of each pixel in the target preset image and the preset region of the target preset image, the pixel with the highest pixel value in the preset region can be determined. Specifically, when the target preset image is divided into one preset region, the pixel with the highest pixel value in the target preset image is used as the pixel with the highest pixel value in the preset region. When the target preset image is divided into multiple preset regions, the pixels with the highest pixel value in each preset region are respectively acquired. The position of the pixel with the highest pixel value in the target preset image is used as the target center, thereby establishing a one-to-one matching relationship between the target center and the preset region. Further, a matching instance identifier is assigned to the target center, and then the instance identifier of the target center is assigned to the target voxels orthographically projected onto the preset region to which the target center belongs, so that the target voxels belonging to the target center obtain the instance identifier.

[0086] Optionally, when the target preset image is divided into one preset region, only one target center needs to be assigned an instance identifier. When the target preset image is divided into multiple preset regions, matching instance identifiers need to be assigned to each target center corresponding to each preset region respectively.

[0087] In an optional embodiment of the present invention, instance segmentation of the target voxels according to the optimized voxel stitching feature may include: inputting the optimized voxel stitching feature into the trained offset prediction neural network to obtain the offset of the target voxels; obtaining offset voxels based on the offset and the target voxels; orthographically projecting the offset voxels onto the target preset image, and obtaining the pixel value of each pixel according to the number of offset voxels in each pixel of the target preset image; obtaining the pixel with the highest pixel value in the preset region according to the pixel value; determining the target center according to the pixel with the highest pixel value and assigning a matching instance identifier to the target center; obtaining the instance identifier of the target voxels belonging to the target center according to the instance identifier of the target center.

[0088] In an example of the present invention, the optimized voxel stitching feature can also be input into the trained offset prediction neural network to obtain the offset of the target voxels, and subsequent operations such as obtaining offset voxels based on the offset and the target voxels are performed to obtain the instance identifier of the target voxels belonging to the target center.

[0089] In an alternative embodiment of the present invention, performing a top-down projection of the offset voxel onto the target preset image may include: obtaining a preset spatial dimension of the offset voxel; obtaining the target preset image based on the preset spatial dimension; wherein the preset spatial dimension matches each pixel in the target preset image; and performing a top-down projection of the offset voxel onto the target preset image.

[0090] Wherein, the preset spatial dimension may be a preset two-dimensional spatial dimension. Optionally, the preset spatial dimension may be a two-dimensional spatial dimension of a*b, where the sizes of a and b may be the same or different, and the measurement units of a and b may be decimeters or meters, etc. The embodiments of the present invention do not limit the measurement units of a and b.

[0091] In the embodiments of the present invention, according to the resolution requirement of the target preset image, the top-down two-dimensional image of the offset voxel in the top-down view may be divided to determine the preset spatial dimension, and based on the preset spatial dimension, the target preset image with pixels matching the preset spatial dimension may be obtained, so as to perform a top-down projection of the offset voxel onto the target preset image.

[0092] In an alternative embodiment of the present invention, obtaining the pixel with the highest pixel value in the preset area according to the pixel value may include: comparing the pixel values of each pixel in the preset area to obtain the pixel with the highest pixel value in the preset area.

[0093] In the embodiments of the present invention, when the target preset image is divided into a preset area, compare the pixel values of each pixel in the target preset image, and use the pixel with the highest pixel value in the target preset image as the pixel with the highest pixel value in the preset area. When the target preset image is divided into multiple preset areas, obtain the pixel values of each pixel in each preset area respectively, and then compare the pixel values of each pixel in each preset area in units of the preset area (only compare the pixel values of pixels belonging to the same preset area), so as to obtain the pixel with the highest pixel value in each preset area.

[0094] In an alternative embodiment of the present invention, assigning a matching instance identifier to the target center may include: obtaining the current target center and prior size data matching the semantic type to be processed; obtaining a detection range to be detected that matches the current target center according to the current target center and the prior size data; and when it is determined that there is no other target center in the detection range to be detected, not updating the instance identifier of the current target center.

[0095] Among them, the semantic type to be processed can be the semantic type that matches the current target center, that is, the semantic type of the point cloud for which the entity identifier needs to be updated and discriminated. The prior size data can be used to represent the radius of the entity determined according to experience. The detection range can be a region range determined with the current target center as the origin and the prior size data as the radius.

[0096] In an embodiment of the present invention, the current target center and the prior size data matching the semantic type to be processed can be obtained first. Further, with the current target center as the center of the circle and the prior size data as the radius, the detection range matching the current target center is obtained. Then, the target centers within the detection range are traversed. When there are no other target centers within the detection range, the instance identifier of the current target center remains unchanged, and the instance identifier of the current target center is not used to update the instance identifiers of other target centers.

[0097] In an optional embodiment of the present invention, allocating a matching instance identifier to the target center may include: obtaining the current target center and the prior size data matching the semantic type to be processed; obtaining the detection range matching the current target center according to the current target center and the prior size data; when it is determined that there are other target centers within the detection range, performing a merging process on the instance identifiers of the target centers existing within the detection range.

[0098] In an embodiment of the present invention, after obtaining the detection range matching the current target center, the target centers within the detection range are further traversed. When there are other target centers within the detection range, the instance identifier of the current target center can be used as the instance identifier of the other target centers within the detection range.

[0099] In an optional embodiment of the present invention, allocating a matching instance identifier to the target center may include: obtaining the current target center and the prior size data matching the semantic type to be processed; obtaining the detection range matching the current target center according to the current target center and the prior size data; when it is determined that there are other target centers within the detection range and their semantic types are the same as that of the current target center, performing a merging process on the instance identifiers of the target centers existing within the detection range.

[0100] In an embodiment of the present invention, after obtaining the detection range matching the current target center, the target centers within the detection range are further traversed. When there are other target centers within the detection range, the semantic types of the other target centers within the detection range can be further obtained, so that the instance identifier of the current target center is used as the instance identifier of the other target centers within the detection range that have the same semantic type as the current target center.

[0101] In an alternative embodiment of the present invention, after obtaining the instance identifiers of the target voxels belonging to the target central voxel according to the instance identifier of the target central voxel, it may further include: obtaining the instance identifiers matching each point in the point cloud according to the instance identifiers of the target voxels and the points in the voxel midpoint cloud.

[0102] In an embodiment of the present invention, the voxel can be used as the allocation unit of the instance identifier, and the instance identifiers of the target voxels are respectively allocated to the points in the point cloud included in the target voxel, so that each point in the point cloud is allocated an instance identifier.

[0103] Exemplarily, assume that the instance identifier of voxel A is 1, and the instance identifiers of points a, b, and c in the point cloud included in voxel A are also 1.

[0104] In an embodiment of the present invention, each point in the point cloud is respectively included in multiple voxels, and a first matrix diagram to be processed is generated according to the point features of the points included in each voxel and the number of voxels. Then, the feature dimension of the first matrix diagram to be processed is processed by using a multi-layer neural network and a max pooling layer to obtain the voxel features of each voxel. Thereby, each voxel is projected onto a preset image, and according to the projection result, the pixel features of each voxel in the preset image are obtained. Furthermore, the voxel features and the pixel features are feature stitched to form voxel stitching features corresponding to the voxels. Fusing the voxel features and the pixel features can improve the expression ability of the voxel features, and using the voxel stitching features can quickly identify and classify the targets in the point cloud, solving the problems in the prior art that the discrimination of voxels based on the features extracted by the 2D network is poor, and the memory occupation is large and the time consumption is serious when the 3D network identifies the features of voxels, improving the expression ability of the voxel features and the data processing effect of the point cloud data.

[0105] The embodiment of the present invention further includes:

[0106] A1. A panoramic segmentation method for three-dimensional point cloud data, including: obtaining a point cloud with semantic types, inputting the target points in the point cloud into a trained offset prediction neural network to obtain the offsets of the target points; obtaining offset points based on the offsets and the target points; performing a top-down projection of the offset points onto a preset image, and obtaining the pixel value of each pixel according to the number of offset points in each pixel in the image; obtaining the pixel with the highest pixel value in a preset area according to the pixel value; determining a target center according to the pixel with the highest pixel value and allocating a matching instance identifier to the target center; obtaining the instance identifiers of the target points belonging to the target center according to the instance identifier of the target center.

[0107] A2. The method according to A1, wherein the obtaining of the point cloud with semantic types includes: obtaining the point cloud scanned by the lidar; inputting the point cloud into a pre-trained semantic segmentation neural network to obtain the point cloud with semantic types.

[0108] A3. The method according to A2, wherein the top-down projection of the offset point onto a preset image includes: obtaining the preset spatial dimension of the offset point; obtaining a preset image based on the preset spatial dimension, wherein the preset spatial dimension matches each pixel in the preset image; and projecting the offset point onto the preset image in a top-down manner.

[0109] A4. The method according to A1, wherein the obtaining of the pixel with the highest pixel value in a preset area according to the pixel value includes: comparing the pixel values of each pixel in the preset area to obtain the pixel with the highest pixel value in the preset area.

[0110] A5. The method according to any one of A1 - A4, wherein the assigning of a matching instance identifier to the target center includes: obtaining the current target center and prior size data matching the semantic type to be processed; obtaining a detection range matching the current target center according to the current target center and the prior size data; and when it is determined that there is no other target center within the detection range, not updating the instance identifier of the current target center.

[0111] A6. The method according to any one of A1 - A4, wherein the assigning of a matching instance identifier to the target center includes: obtaining the current target center and prior size data matching the semantic type to be processed; obtaining a detection range matching the current target center according to the current target center and the prior size data; and when it is determined that there is another target center within the detection range, performing a merging process on the instance identifiers of the target centers existing within the detection range.

[0112] A7. The method according to any one of A1 - A4, wherein the assigning of a matching instance identifier to the target center includes: obtaining the current target center and prior size data matching the semantic type to be processed; obtaining a detection range matching the current target center according to the current target center and the prior size data; and when it is determined that there is another target center within the detection range and its semantic type is the same as that of the current target center, performing a merging process on the instance identifiers of the target centers existing within the detection range.

[0113] B1. A panoramic segmentation device, comprising: an offset acquisition module, configured to acquire a point cloud with semantic types, input a target point in the point cloud into a trained offset prediction neural network, and obtain an offset of the target point; an offset point acquisition module, configured to obtain an offset point based on the offset and the target point; a pixel value acquisition module, configured to project the offset point onto a preset image in a top view manner, and obtain a pixel value of each pixel according to the number of offset points in each pixel of the image; a peak pixel acquisition module, configured to acquire a pixel with the highest pixel value in a preset area according to the pixel value; a first instance identification assignment module, configured to determine a target center according to the pixel with the highest pixel value, and assign a matching instance identification to the target center; a second instance identification assignment module, configured to obtain an instance identification of a target point belonging to the target center according to the instance identification of the target center.

[0114] C1. An electronic device, characterized in that the electronic device comprises: one or more processors; a storage device, configured to store one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the panoramic segmentation method of three-dimensional point cloud data as described in any one of claims A1 - A7.

[0115] D1. A computer storage medium, on which a computer program is stored, characterized in that when the program is executed by a processor, it implements the panoramic segmentation method of three-dimensional point cloud data as described in any one of claims 1 - 7.

[0116] In an optional embodiment of the present invention, before acquiring the point cloud with semantic types, it further comprises: respectively dividing each point in the point cloud into a plurality of voxels, and acquiring a voxel feature of each voxel according to the point features of the points included in each voxel; projecting each voxel onto a preset image, and acquiring a pixel feature of each voxel in the preset image according to the projection result; performing feature splicing on the voxel feature and the pixel feature to form a voxel splicing feature corresponding to the voxel; wherein, the point cloud includes the voxel splicing feature of the voxel.

[0117] In an embodiment of the present invention, after obtaining the voxel splicing feature corresponding to the voxel, it is further possible to assign the same voxel splicing feature as the voxel to which it belongs to the points in the point cloud within the same voxel, that is, the point cloud has the voxel splicing feature of the voxel, so as to acquire a point cloud with semantic types and having a voxel splicing feature, input a target point in the point cloud with semantic types and having a voxel splicing feature into a trained offset prediction neural network, and continue to execute subsequent steps such as obtaining the offset of the target point in A1.

[0118] Figure 3The figure is a flowchart of a panoramic segmentation method for 3D point cloud data provided by an embodiment of the present invention. This embodiment is applicable to the situation of efficient and accurate panoramic segmentation. This method can be executed by a panoramic segmentation device for 3D point cloud data. The device can be implemented in software and / or hardware, and is generally integrated in an electronic device. The electronic device can be a terminal device or a server device, etc. The embodiments of the present invention do not limit the type of the electronic device that executes the panoramic segmentation method for 3D point cloud data. Correspondingly, as Figure 3 shown, the method includes the following operations:

[0119] S310. Obtain point cloud with semantic types, input the target points in the point cloud into a trained offset prediction neural network, and obtain the offsets of the target points.

[0120] Among them, the target points can be points that need to be instance-segmented and are different from the background. Exemplarily, in the field of autonomous vehicle driving, the target points can be points in the point cloud that represent vehicles and / or people. The embodiments of the present invention do not limit the specific entity types represented by the target points. The offset prediction neural network can also be used to determine the offsets of each point when the points in the point cloud are clustered at the center. The embodiments of the present invention do not limit the specific type of the offset prediction neural network.

[0121] In the embodiments of the present invention, semantic segmentation can be first performed on the point cloud to obtain a point cloud with semantic types, and then each target point that needs to be instance-segmented is determined from the point cloud with semantic types. Thus, each target point in the point cloud is input into a trained offset prediction neural network, and each target point is clustered at the center according to the offset prediction neural network to obtain the offsets of each target point.

[0122] Exemplarily, the points in the point cloud with semantic types of vehicles and pedestrians can be used as target points. Then, the points in the point cloud of vehicles and pedestrians are input into a trained offset prediction neural network. The offset prediction neural network can cluster the points in each point cloud towards the center of the point cloud to obtain the offsets of the target points towards the centers of their respective point clouds.

[0123] S320. Obtain offset points based on the offsets and the target points.

[0124] Among them, the offset points can be points obtained by offsetting the positions of the target points according to the offsets.

[0125] In the embodiments of the present invention, position offset processing can be performed on each target point respectively according to the offset matching each target point to obtain each offset point.

[0126] S330. Project the offset points onto a preset image in a top view, and obtain the pixel value of each pixel according to the number of offset points in each pixel of the image.

[0127] Among them, the preset image can be a preset two-dimensional image, and the size of the preset image can be set by itself according to the display needs. The preset image is the target preset image in the above text.

[0128] In the embodiment of the present invention, the preset image can be obtained according to the needs first, and then the offset point is projected downward onto the preset image, so as to determine the pixel value of each pixel in the preset image according to the number of offset points that each pixel point in the preset image falls into during the downward projection.

[0129] S340. Obtain the pixel with the highest pixel value in the preset area according to the pixel value.

[0130] Among them, the preset area can be an area in the preset image. Optionally, the preset image can be divided as needed to obtain multiple preset areas, or the entire preset image can be used as a preset area. Exemplarily, assuming that the size of the preset image is 3dm * 7dm, if the preset image is divided into 100 preset areas with the same size, the size of the preset area is 0.3dm * 0.7dm.

[0131] In the embodiment of the present invention, the pixel with the highest pixel value in the preset area can be determined according to the pixel value of each pixel in the preset image and the preset area of the preset image. Specifically, when the preset image is divided into one preset area, the pixel with the highest pixel value in the preset image is used as the pixel with the highest pixel value in the preset area. When the preset image is divided into multiple preset areas, the pixels with the highest pixel values in each preset area are obtained respectively.

[0132] S350. Determine the target center according to the pixel with the highest pixel value, and assign a matching instance identifier to the target center.

[0133] Among them, the target center can be the position of the pixel with the highest pixel value in the preset area in the preset image, and is used to represent the instance center. Optionally, when the preset image is divided into one preset area, the target center is unique. When the preset image is divided into multiple preset areas, there is one target center in each preset area. The instance identifier can be a mark used to distinguish different instances.

[0134] In the embodiment of the present invention, the position of the pixel with the highest pixel value in the preset image can be used as the target center, so as to establish a one-to-one matching relationship between the target center and the preset area, and then assign a matching instance identifier to the target center. Specifically, when the preset image is divided into one preset area, only one instance identifier needs to be assigned to one target center. When the preset image is divided into multiple preset areas, matching instance identifiers need to be assigned to the respective target centers corresponding to each preset area.

[0135] S360. Obtain the instance identifier of the target points belonging to the target center according to the instance identifier of the target center.

[0136] In an embodiment of the present invention, the instance identifier of the target center can be assigned to other target points of the point cloud where the target center is located, that is, the instance identifier of the target center is assigned to the target points belonging to the target center, so that the target points belonging to the target center obtain the instance identifier.

[0137] The technical solution of this embodiment is to obtain the point cloud with semantic types, input the target points in the point cloud into the trained offset prediction neural network to obtain the offsets of the target points, and then obtain the offset points based on the offsets and the target points. Further, the offset points are orthographically projected onto a preset image, and according to the number of offset points in each pixel in the image, the pixel value of each pixel is obtained. Furthermore, according to the pixel value, the pixel with the highest pixel value in the preset area is obtained, and the target center is determined according to the pixel with the highest pixel value, and a matching instance identifier is assigned to the target center, so as to obtain the instance identifier of the target points belonging to the target center according to the instance identifier of the target center. In this solution, after the orthographic projection of the offset points, determining the target center according to the pixel with the highest pixel value in the preset area can make the target center have a good correspondence with the offset points, and can also make the point cloud where the target points are located determine a unique center. Obtaining the instance identifier of the target points belonging to the target center according to the instance identifier of the target center can efficiently and accurately determine the instance identifier of the target points belonging to the target center, solve the problems in the prior art that the number of target centers detected by the target detection network has errors, resulting in the inability to guarantee the correspondence between the target center and the offset points, increased system time consumption, and high complexity of data pre - preparation, and also solve the problems of serious time consumption and complex parameter adjustment caused by introducing the heuristic clustering algorithm, and can reduce the data configuration complexity and efficiently and accurately implement panoramic segmentation on the premise of ensuring the operation effect of the system.

[0138] Figure 4 It is a flowchart of another panoramic segmentation method for three - dimensional point cloud data provided by an embodiment of the present invention. This embodiment is a specific implementation based on the above - mentioned embodiment. In this embodiment, a specific and optional implementation manner of orthographically projecting the offset points onto a preset image is given. The specific process of orthographically projecting the offset points onto a preset image can be: obtain the preset spatial size of the offset points; based on the preset spatial size, obtain the preset image; where the preset spatial size matches each pixel in the preset image; orthographically project the offset points onto the preset image. Correspondingly, as Figure 4 shown, the method includes the following operations:

[0139] S410. Obtain the point cloud with semantic types, input the target points in the point cloud into the pre-trained offset prediction neural network, and obtain the offsets of the target points.

[0140] In an alternative embodiment of the present invention, obtaining the point cloud with semantic types may include: obtaining the point cloud scanned by the lidar; inputting the point cloud into a pre-trained semantic segmentation neural network to obtain the point cloud with semantic types.

[0141] Among them, the semantic segmentation neural network can be any type of neural network as long as it can perform semantic segmentation on the point cloud. The embodiments of the present invention do not limit the specific type of the semantic segmentation neural network.

[0142] In the embodiments of the present invention, the space to be scanned can be scanned by the lidar to obtain the point cloud, and then the point cloud is input into the pre-trained semantic segmentation neural network to identify the semantic types of the point cloud through the semantic segmentation neural network, and at least one point cloud with semantic types is obtained.

[0143] S420. Obtain the offset points based on the offsets and the target points.

[0144] S430. Obtain the preset spatial size of the offset points.

[0145] Among them, the preset spatial size can be a preset two-dimensional spatial size. Optionally, the preset spatial size can be a two-dimensional spatial size of a*b, where the sizes of a and b can be the same or different, and the measurement units of a and b can be decimeters or meters, etc. The embodiments of the present invention do not limit the measurement units of a and b.

[0146] In the embodiments of the present invention, according to the resolution requirements of the preset image, the top-down two-dimensional image of the offset points in the top-down view can be divided to determine the preset spatial size of the offset points.

[0147] S440. Obtain the preset image based on the preset spatial size.

[0148] Among them, the preset spatial size matches each pixel in the preset image, and one pixel of the preset image corresponds to the preset spatial size in the two-dimensional space.

[0149] In the embodiments of the present invention, the preset image with pixels matching the preset spatial size can be obtained based on the preset spatial size.

[0150] Exemplarily, assuming that the preset spatial size is 0.2m*0.3m, then each pixel of the preset image corresponds to a spatial size of 0.2m*0.3m.

[0151] S450. Perform a top-down projection of the offset points onto a preset image, and obtain the pixel value of each pixel based on the number of offset points in each pixel of the image.

[0152] In an embodiment of the present invention, the offset points can be projected onto a preset image from above. When performing the top-down projection, the number of projected offset points in each pixel of the preset image can be counted. Furthermore, the number of projected offset points in each pixel of the preset image can be used as the feature information of each pixel point, thereby determining the pixel value of each pixel.

[0153] S460. Obtain the pixel with the highest pixel value in a preset area based on the pixel value.

[0154] In an optional embodiment of the present invention, obtaining the pixel with the highest pixel value in a preset area based on the pixel value may include: comparing the pixel values of each pixel in the preset area to obtain the pixel with the highest pixel value in the preset area.

[0155] Specifically, when the preset image is divided into one preset area, compare the pixel values of each pixel in the preset image, and use the pixel with the highest pixel value in the preset image as the pixel with the highest pixel value in the preset area. When the preset image is divided into multiple preset areas, respectively obtain the pixel values of each pixel in each preset area, and then compare the pixel values of each pixel in each preset area in units of the preset area (only compare the pixel values of pixels belonging to the same preset area), so as to obtain the pixel with the highest pixel value in each preset area.

[0156] S470. Determine the target center based on the pixel with the highest pixel value, and assign a matching instance identifier to the target center.

[0157] In an optional embodiment of the present invention, assigning a matching instance identifier to the target center may include: obtaining the current target center and prior size data matching the semantic type to be processed; obtaining the detection range to be detected matching the current target center based on the current target center and the prior size data; when it is determined that there is no other target center within the detection range to be detected, do not update the instance identifier of the current target center.

[0158] Among them, the semantic type to be processed can be the semantic type matching the current target center, that is, the semantic type of the point cloud for which the entity identifier needs to be updated and discriminated. The prior size data can be used to represent the entity radius determined according to experience. The detection range to be detected can be a region range determined with the current target center as the origin and the prior size data as the radius.

[0159] In an embodiment of the present invention, the current target center can be obtained first, as well as the prior size data matching the semantic type to be processed. Further, with the current target center as the center of the circle and the prior size data as the radius, the detection range matching the current target center is obtained. Then, the target centers within the detection range are traversed. When there are no other target centers within the detection range, the instance identifier of the current target center remains unchanged, and the instance identifier of the current target center is not used to update the instance identifiers of other target centers.

[0160] In an alternative embodiment of the present invention, allocating a matching instance identifier to the target center may include: obtaining the current target center and the prior size data matching the semantic type to be processed; obtaining the detection range matching the current target center according to the current target center and the prior size data; when it is determined that there are other target centers within the detection range, performing a merging process on the instance identifiers of the target centers existing within the detection range.

[0161] In an embodiment of the present invention, after obtaining the detection range matching the current target center, the target centers within the detection range are further traversed. When there are other target centers within the detection range, the instance identifier of the current target center can be used as the instance identifier of the other target centers within the detection range.

[0162] In an alternative embodiment of the present invention, allocating a matching instance identifier to the target center may include: obtaining the current target center and the prior size data matching the semantic type to be processed; obtaining the detection range matching the current target center according to the current target center and the prior size data; when it is determined that there are other target centers within the detection range and their semantic types are the same as that of the current target center, performing a merging process on the instance identifiers of the target centers existing within the detection range.

[0163] In an embodiment of the present invention, after obtaining the detection range matching the current target center, the target centers within the detection range are further traversed. When there are other target centers within the detection range, the semantic types of the other target centers within the detection range can be further obtained, so that the instance identifier of the current target center is used as the instance identifier of the other target centers within the detection range that have the same semantic type as the current target center.

[0164] S480. Obtain the instance identifier of the target point belonging to the target center according to the instance identifier of the target center.

[0165] In a specific example, a panoramic segmentation system that executes a panoramic segmentation method for three-dimensional point cloud data can be used to determine the semantic types of the point cloud and the instance identifiers of different instances. The panoramic segmentation system may include a semantic segmentation module, a center regression module, a clustering pseudo heat map generation module, and a center recombination module. The point cloud of the current field of view space shown in Figure 5 scanned by the lidar of the driverless vehicle is input into the semantic segmentation module of the panoramic segmentation system. Based on the semantic segmentation neural network of the semantic segmentation module (such as existing PointNet, PolarNet, etc.), the semantic types of the point cloud are identified to obtain a point cloud with semantic types, and the point cloud with semantic types is input into the center regression module. The semantic segmentation result is referred to Figure 6 .

[0166] The center regression module is used to filter out the point cloud belonging to the foreground point (target point) type according to the per-point semantic type, such as the point cloud of pedestrians and vehicles (at this time, the points belonging to different instances cannot be distinguished, such as the point clouds of two different vehicles cannot be distinguished). Then, the foreground points are input into the trained offset prediction neural network to obtain the offset ΔP=(Δx, Δy, Δz) of the foreground points. Thus, based on the position coordinates P=(x, y, z) of the foreground points and the offset of the foreground points, the position of the foreground points is offset, that is, the center regression of the foreground points is performed to obtain the point cloud of the offset points (the schematic diagram of the point cloud of the offset points can be seen in Figure 7 ), and the offset points in the three-dimensional space are sent to the clustering pseudo heat map generation module. After the above processing, each foreground point has aggregated towards its respective instance, and the spatial distance between different instance points has also been increased. Among them, ΔP represents the offset of the foreground point, Δx represents the offset of the foreground point on the x-axis, Δy represents the offset of the foreground point on the y-axis, and Δz represents the offset of the foreground point on the z-axis. P represents the position coordinates of the foreground point, x represents the coordinate of the foreground point on the x-axis, y represents the coordinate of the foreground point on the y-axis, and z represents the coordinate of the foreground point on the z-axis.

[0167] The clustering pseudo-thermodynamic map generation module is used to project a 2D image of the offset points in the three-dimensional space from a top-down perspective onto a preset image, and draw a pseudo-thermodynamic map based on the number of offset points in each pixel of the preset image. This pseudo-thermodynamic map is similar to the function of predicting targets by the target detection network, but does not require training a neural network to obtain. In reality, different foreground points rarely overlap in the top-down perspective, so the top-down view is a relatively ideal space for processing information of different instances. After the information is reduced to 2D, some efficient computer vision operators can be utilized. During the projection process, it is defined that each pixel of the 2D photo (preset image) corresponds to a 0.2m * 0.2m spatial size in the 2D image of the offset points from the top-down perspective. All offset points can be projected onto a 2D photo with a size of (H, W). Record the number of offset points falling into each pixel. Simply taking the number of points as the feature information of the pixel, a clustered pseudo-thermodynamic map is generated, as shown in Figure 8 shown. Based on this pseudo-thermodynamic map, the pixel with the highest local peak (the pixel with the highest pixel value in the preset area) can represent the target center. Optionally, based on the sliding window-based 2D max pooling operation, the position of the local peak of the pixels in the 2D photo can be efficiently extracted to obtain the target center. After obtaining the target center, an instance ID can be assigned to each target center. All offset points will be assigned the instance ID of the target center closest to themselves based on the nearest neighbor principle (if an offset point is equidistant from multiple target centers, the instance ID of one of the target centers with the same distance as the offset point will be used as the instance of this offset point). All foreground points will obtain instance IDs and do not rely on heuristic clustering algorithms. Coupled with the semantic types of all point clouds already predicted in the semantic segmentation module, an efficient panoramic segmentation task can be completed.

[0168] After obtaining the offset points in the three-dimensional space, the foreground points can also be divided based on the target-level information by the target detection network. If several vehicles and pedestrians have been detected in the space, according to the target detection network, it can be judged which object the foreground point specifically belongs to to complete the instance segmentation task. However, predicting the target-level information through a target detection network, on the one hand, increases the system time consumption, and on the other hand, the targets predicted by the target detection network may have inconsistent results in terms of quantity and position with the clustered point clusters. The target center determined by the clustering pseudo-thermodynamic map generation module has a natural correspondence with the translated foreground point cluster (the cluster composed of offset points). As long as there is a place where foreground points gather, a target center will definitely be generated. In contrast, when determining the target center based on the target detection network, the number of detected target objects cannot be guaranteed, and there is no guarantee that there is a good correspondence between the detected target center and the information of the clustered point cluster in terms of position.

[0169] The generation of the target center is based on the aggregation of foreground points towards the instance center. In the driving scenario, the effect of point clouds belonging to small targets returning to the center is generally better than that of large targets. Large targets are more likely to only have partial information observed in the lidar sensor, such as Figure 9 In Figure 9 , a car predicts one instance ID, and a bus predicts multiple instance IDs. That is, the aggregation effect of points belonging to the car towards the center of the car is better. After the above operations, the correct instance ID is obtained. For the points belonging to the bus, when the bus is close to the lidar, only one side of the bus is scanned by the lidar, and the offset prediction neural network does not have enough information to accurately infer the offset of the points belonging to the bus. Referring to the segmentation result of the point cloud in the enlarged graph of the bus, it can be determined that the points belonging to the bus are predicted to have 4 target centers, thus generating 4 instance IDs instead of the expected single instance ID.

[0170] In order to segment large targets well even when the center regression is not ideal, through the center regrouping module, the target centers can be grouped once, and the target centers divided into the same group have the same instance ID. The specific approach is as follows:

[0171] 1. Based on the pseudo heat map, use the window 2D mean pooling operation to count the number of points of different semantic categories near each pixel (within the sliding window), and determine the semantic category of each pixel based on the principle of the maximum number of votes.

[0172] 2. Assign a prior size (prior size data) to each semantic category. This size is derived from the statistics of the object detection dataset. For example, for a car, the average size of the car can be statistically calculated based on the dataset, and based on this size, the minimum radius R = 0.5 * min(L, W) of each semantic category is established, where W and L represent the average width and length of the object.

[0173] 3. After the above steps, each target center has its own minimum radius, which can be regarded as its own bandwidth. Starting from a certain target center for traversal, in this traversal, it is regarded as the base center C b , with the instance ID being G b , and the minimum bandwidth being B b , if there is another target center C t within the bandwidth, with its ID being G t , then all centers with the ID G t will be re - assigned the instance ID of the base center, that is, G b . In addition to the bandwidth constraint, a constraint can be added that the base center C b and the target center C t must be of the same semantic category. This will have a better effect on different semantic categories in crowded scenarios, such as distinguishing pedestrians very close to vehicles.

[0174] Until the traversal of all target centers is completed, the target centers that meet the conditions of step 3 will be re-assigned instance IDs. As Figure 10 shown, after the center re-grouping, the buses originally divided into multiple IDs can be completely segmented, so that each bus has exactly one instance ID.

[0175] Figure 11 is a flowchart of data processing of a panoramic segmentation system provided by an embodiment of the present invention. As Figure 11 shown, the panoramic segmentation system receives the point cloud scanned by the lidar, and then performs semantic segmentation on the point cloud based on the semantic segmentation neural network to obtain the point cloud with semantic types, and filters the foreground points in the point cloud based on the semantic types, and performs center regression processing on the foreground points (that is, moves the foreground points according to the offset) to obtain the offset points, and further projects the offset points onto the 2D photo in a top-down manner to generate a pseudo heat map, so as to obtain multiple target centers based on the pixel values of the pixels in the pseudo heat map. After obtaining multiple target centers, perform center re-grouping on the target centers to obtain the instance IDs of each target center.

[0176] In the field of point cloud panoramic segmentation, the Panoptic-PolarNet method is currently leading in the industry. It obtains the heat map of target centers (target-level information) through a trained object detection network, and introduces the heat map into the panoramic segmentation system of this solution, that is, replaces the clustering pseudo heat map generation module, and obtains the panoramic segmentation method of the learned heat map. It can be seen from Table 1 that the panoramic segmentation method of the three-dimensional point cloud data of this solution significantly improves the panoramic segmentation accuracy (PQ: panoptic quality).

[0177] Table 1 Comparison table of the panoramic segmentation effect of the panoramic segmentation method of the learned heat map and the panoramic segmentation effect of this solution

[0178]

[0179] As shown in Table 1, the PQ after panoramic segmentation of all points in the point cloud according to the panoramic segmentation method of the learned heat map is 69.1, while the PQ after panoramic segmentation of all points in the point cloud using the panoramic segmentation method of this solution is 73.4 (an increase of 4.3 compared with the panoramic segmentation method of the learned heat map). It can be seen that the panoramic segmentation accuracy when performing panoramic segmentation on all points in the point cloud of this solution has been improved. The PQ after panoramic segmentation of the foreground points in the point cloud according to the panoramic segmentation method of the learned heat map is 65.7, while the PQ after panoramic segmentation of the foreground points in the point cloud using the panoramic segmentation method of this solution is 72.5 (an increase of 6.8 compared with the panoramic segmentation method of the learned heat map). It can be seen that the panoramic segmentation accuracy when performing panoramic segmentation on all points and foreground points in the point cloud of this solution has been improved.

[0180] The panoramic segmentation method of this solution ranks first in the panoramic segmentation tasks of the two most important datasets in the industry, semanticKITTI and nuScenes. For specific comparison, please refer to Tables 2 and 3.

[0181] Table 2 Comparison of segmentation effects of panoramic segmentation methods under three evaluation dimensions

[0182]

[0183] Among them, SQ represents the segmentation quality, and SQ(Th) represents the segmentation quality of foreground points. As shown in Table 2, the panoramic segmentation accuracy of this solution is the same as that of the Sxubj panoramic segmentation method, and the panoramic segmentation accuracy is higher than that of the AF2S3Net panoramic segmentation method. Compared with the Sxubj panoramic segmentation method and the AF2S3Net panoramic segmentation method, the segmentation quality of the panoramic segmentation method of this solution is higher. Moreover, compared with the Sxubj panoramic segmentation method and the AF2S3Net panoramic segmentation method, the segmentation quality of the panoramic segmentation method of this solution is higher when performing panoramic segmentation on foreground points. It can be seen that the segmentation quality of the panoramic segmentation method of this solution is the best.

[0184] Table 3 Comparison of segmentation effects of panoramic segmentation methods under four evaluation dimensions

[0185] Method Name PQ mIoU SQ RQ Panoptic-PHNet (This Solution) 0.815 0.815 0.919 0.884 SPVCNN++-Panoptic 0.791 0.803 0.909 0.868 Team_AX_Semantioc 0.731 0.730 0.888 0.818 PolarStrem-1 0.709 0.697 0.859 0.817

[0186] Among them, mIoU represents the evaluation index of semantic segmentation, which refers to the calculation of the intersection over union of the predicted results and the ground truth. The closer the predicted results are to the ground truth, the closer mIoU is to 1. RQ represents the recognition quality. As shown in Table 3, compared with the SPVCNN++-Panoptic, Team_AX_Semantioc, and PolarStrem-1 panoramic segmentation methods, the panoramic segmentation method of this solution has the best performance in the four indicators of panoramic segmentation accuracy, semantic segmentation, segmentation quality, and recognition quality, indicating that the panoramic segmentation effect of the panoramic segmentation method of this solution is the best.

[0187] In summary, the panoramic segmentation method for three-dimensional point cloud data proposed in this solution does not rely on the results of object detection, nor does it rely on heuristic clustering algorithms that have high requirements for empirical parameters and are time-consuming. Compared with methods that rely on object detection to segment different point cloud instances, this solution does not need to introduce an object detection network. Methods that rely on object detection are severely limited by the accuracy of object detection. The generation of pseudo heat maps does not rely on the introduction of object-level tasks that need to be learned. Through consistent projection operations, information output at the object level with an absolute high recall rate can be obtained. Together with the center regrouping module, the accuracy and efficiency of the entire instance segmentation part are greatly improved compared with existing methods, thereby improving the efficiency of fine-grained point cloud perception tasks. Moreover, this solution has strong scalability and can obtain fine-grained semantic information, instance information, and object center information under a unified framework.

[0188] In the technical solution of this embodiment, by obtaining point clouds with semantic types, inputting the target points in the point clouds into a trained offset prediction neural network to obtain the offsets of the target points, and then obtaining offset points based on the offsets and the target points, and obtaining a preset spatial size, based on the preset spatial size, a preset image is obtained. Further, the offset points are projected downward onto the preset image, and according to the number of offset points in each pixel in the image, the pixel value of each pixel is obtained, so as to obtain the pixel with the highest pixel value in the preset area according to the pixel value, thereby determining the object center according to the pixel with the highest pixel value, and assigning a matching instance identifier to the object center. Finally, according to the instance identifier of the object center, the instance identifier of the target points belonging to the object center is obtained. In this solution, after the offset points are projected downward, determining the object center according to the pixel with the highest pixel value in the preset area can make the object center have a good correspondence with the offset points, and can also make the point cloud where the target points are located determine a unique center. Obtaining the instance identifier of the target points belonging to the object center according to the instance identifier of the object center can efficiently and accurately determine the instance identifier of the target points belonging to the object center. And obtaining a preset image based on the preset spatial size can establish the correspondence between the preset spatial size and the pixels of the obtained preset image to flexibly meet the display requirements of the pixels, solving the problems in the prior art that the number of object centers detected by the object detection network has errors, resulting in the inability to guarantee the correspondence between the object center and the offset points, increased system time consumption, and high complexity of data pre-preparation. It also solves the problems of serious time consumption and complex parameter adjustment caused by introducing heuristic clustering algorithms, and can reduce the data configuration complexity while ensuring the operation effect of the system, flexibly meet the pixel display requirements of users, and efficiently and accurately achieve panoramic segmentation.

[0189] Figure 12 is a schematic diagram of a panoramic segmentation device for three-dimensional point cloud data provided by an embodiment of the present invention, as Figure 12As shown, the device includes: an offset acquisition module 510, an offset point acquisition module 520, a pixel value acquisition module 530, a peak pixel acquisition module 540, a first instance identifier assignment module 550, and a second instance identifier assignment module 560, where:

[0190] The offset acquisition module 510 is configured to acquire a point cloud with semantic types, input the target points in the point cloud into a trained offset prediction neural network, and obtain the offsets of the target points;

[0191] The offset point acquisition module 520 is configured to obtain offset points based on the offsets and the target points;

[0192] The pixel value acquisition module 530 is configured to project the offset points orthographically onto a preset image, and obtain the pixel value of each pixel according to the number of offset points in each pixel of the image;

[0193] The peak pixel acquisition module 540 is configured to obtain the pixel with the highest pixel value in a preset area according to the pixel values;

[0194] The first instance identifier assignment module 550 is configured to determine the target center according to the pixel with the highest pixel value, and assign a matching instance identifier to the target center;

[0195] The second instance identifier assignment module 560 is configured to obtain the instance identifiers of the target points belonging to the target center according to the instance identifier of the target center.

[0196] In the technical solution of this embodiment, by obtaining point clouds with semantic types, inputting the target points in the point clouds into a trained offset prediction neural network to obtain the offsets of the target points, and then obtaining offset points based on the offsets and the target points. Further, the offset points are orthographically projected onto a preset image, and according to the number of offset points in each pixel of the image, the pixel value of each pixel is obtained. Then, according to the pixel values, the pixel with the highest pixel value in the preset area is obtained, and the target center is determined based on the pixel with the highest pixel value. And a matching instance identifier is assigned to the target center, so as to obtain the instance identifier of the target points belonging to the target center according to the instance identifier of the target center. In this solution, after the orthographic projection of the offset points, determining the target center according to the pixel with the highest pixel value in the preset area can make the target center have a good correspondence with the offset points, and can also make the point cloud where the target points are located determine a unique center. Obtaining the instance identifier of the target points belonging to the target center according to the instance identifier of the target center can efficiently and accurately determine the instance identifier of the target points belonging to the target center, solving the problems in the prior art that the number of target centers detected by the target detection network has errors, resulting in the inability to guarantee the correspondence between the target center and the offset points, increased system time consumption, and high complexity of data pre - preparation. It also solves the problems of serious time consumption and complex parameter adjustment caused by introducing the heuristic clustering algorithm, and can reduce the data configuration complexity and efficiently and accurately achieve panoramic segmentation on the premise of ensuring the operation effect of the system.

[0197] Optionally, the offset obtaining module 510 is specifically configured to obtain the point cloud scanned by the lidar; input the point cloud into a pre - trained semantic segmentation neural network to obtain the point cloud with semantic types.

[0198] Optionally, the pixel value obtaining module 530 is specifically configured to obtain a preset spatial size; based on the preset spatial size of the offset points, obtain a preset image; wherein, the preset spatial size matches each pixel in the preset image; project the offset points orthographically onto the preset image.

[0199] Optionally, the peak pixel obtaining module 540 is specifically configured to compare the pixel values of each pixel in the preset area to obtain the pixel with the highest pixel value in the preset area.

[0200] Optionally, the first instance identifier assignment module 550 is specifically configured to obtain the current target center and the prior size data matching the semantic type to be processed; according to the current target center and the prior size data, obtain the detection range to be detected that matches the current target center; when it is determined that there is no other target center in the detection range to be detected, do not update the instance identifier of the current target center.

[0201] Optionally, the first instance identifier allocation module 550 is specifically configured to obtain the current target center and prior size data matching the semantic type to be processed; obtain a detection range to be detected that matches the current target center according to the current target center and the prior size data; when it is determined that there are other target centers within the detection range to be detected, perform a merging process on the instance identifiers of the target centers stored within the detection range to be detected.

[0202] Optionally, the first instance identifier allocation module 550 is specifically configured to obtain the current target center and prior size data matching the semantic type to be processed; obtain a detection range to be detected that matches the current target center according to the current target center and the prior size data; when it is determined that there are other target centers within the detection range to be detected and their semantic types are the same as that of the current target center, perform a merging process on the instance identifiers of the target centers stored within the detection range to be detected.

[0203] The above panoramic segmentation device for three-dimensional point cloud data can execute the panoramic segmentation method for three-dimensional point cloud data provided in any embodiment of the present invention, and has corresponding functional modules and beneficial effects for executing the method. Technical details not described in detail in this embodiment can be referred to the panoramic segmentation method for three-dimensional point cloud data provided in any embodiment of the present invention.

[0204] Since the above-introduced panoramic segmentation device for three-dimensional point cloud data is a device that can execute the panoramic segmentation method for three-dimensional point cloud data in the embodiments of the present invention, based on the panoramic segmentation method for three-dimensional point cloud data introduced in the embodiments of the present invention, those skilled in the art can understand the specific implementation manners and various variations of the panoramic segmentation device for three-dimensional point cloud data in this embodiment. Therefore, the implementation of how the panoramic segmentation device for three-dimensional point cloud data realizes the panoramic segmentation method for three-dimensional point cloud data in the embodiments of the present invention will not be described in detail here. As long as the device used by those skilled in the art to implement the panoramic segmentation method for three-dimensional point cloud data in the embodiments of the present invention belongs to the scope to be protected by this application.

[0205] An embodiment of the present invention also provides a computer storage medium storing a computer program, where the computer program, when executed by a computer processor, is used to execute the panoramic segmentation method for three-dimensional point cloud data according to any one of the above embodiments of the present invention, including: obtaining a point cloud with semantic types, inputting a target point in the point cloud into a trained offset prediction neural network to obtain an offset of the target point; obtaining an offset point based on the offset and the target point; performing a top-down projection of the offset point onto a preset image, and obtaining a pixel value of each pixel according to the number of offset points in each pixel of the image; obtaining a pixel with the highest pixel value in a preset area according to the pixel value; determining a target center according to the pixel with the highest pixel value, and assigning a matching instance identifier to the target center; obtaining an instance identifier of a target point belonging to the target center according to the instance identifier of the target center.

[0206] The computer storage medium according to an embodiment of the present invention may adopt any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples (non-exhaustive list) of the computer-readable storage medium include: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM) or a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, the computer-readable storage medium may be any tangible medium that contains or stores a program, and the program may be used by or in combination with an instruction execution system, apparatus, or device.

[0207] The computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, where the data signal carries computer-readable program code. Such a propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium may also be any computer-readable medium other than the computer-readable storage medium, and the computer-readable medium may send, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device.

[0208] The program code contained on the computer-readable medium can be transmitted by any suitable medium, including but not limited to wireless, wire, optical fiber cable, radio frequency (RF), etc., or any suitable combination of the above.

[0209] The computer program code for performing the operations of the present invention can be written in one or more programming languages or combinations thereof. The programming languages include object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, executed as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (for example, by using an Internet service provider to connect through the Internet).

[0210] Figure 13 Another panoramic segmentation system provided by the present invention is as Figure 13 shown. The panoramic segmentation system includes: a voxel encoder, a 2D top view encoder, a 2D basic network, a first feature splicing module, a second feature splicing module, a Knn-Transformer module, a semantic classifier, a center regression module, a clustering pseudo heat map generation module, and a center recombination module. Among them,

[0211] The voxel encoder can first obtain an N p ×K point cloud, and the K-dimensional point features of the points in the point cloud, and voxelize the point cloud to determine the points included in each voxel, and divide the point cloud into a first matrix graph to be processed, and for each point in the point cloud, use an MLP (Multilayer Perceptron) layer to lift the feature dimension of each point from K dimensions to 256 dimensions. Further, for the dimension, apply maxpooling (maximum pooling layer) to compress the first matrix graph to be processed after dimension elevation, output features of N×1×256 dimensions, and finally use an MLP layer to compress 256 dimensions to 16 to obtain an Nx16 feature matrix. This feature matrix represents the 16-dimensional features of N voxels. And send the Nx16 feature matrix to the 2D top view encoder.

[0212] A 2D top - view encoder is used to receive an Nx16 feature matrix and preset an HxW spatial top - view (preset image). According to the positions where the voxel centers are located, project N voxels onto the HxW spatial top - view, and then divide the spatial top - view into a second matrix diagram of H x W x x 16 to be processed, where represents the number of pixels falling into different voxels, so as to use the MLP layer to increase the feature dimension of the voxel features from 16 to 64. Then, for the dimension, use the maxpooling layer to compress it to obtain a 2D feature map of HxWx64 (feature map of pixels to be processed). Input the 2D feature map into a 2D basic network (two - dimensional semantic segmentation network).

[0213] A 2D basic network is used to extract features with a larger receptive field from the perspective of the top - view and output two different feature maps of HxWxL. One feature map of HxWxL can provide the first pixel features of L dimensions, and the other feature map of HxWxL can provide the second pixel features of L dimensions.

[0214] The first feature splicing module, in order to classify each voxel and extract features with high discriminability for each voxel. According to the situation of N voxels falling into each pixel of the spatial top - view, assign the first pixel features to each voxel to obtain NxL - dimensional information. Specifically, at least one voxel falling into the same pixel can be assigned the L - dimensional first pixel features corresponding to that pixel. Since the voxels in the same column in space have the same L - dimensional information and do not have distinguishability from each other, the NxL - dimensional information and the Nx16 - dimensional information of each voxel are spliced together to obtain the first voxel splicing features of NxC dimensions, where C = L + 16. The significance of this setting is that the L - dimensional information is the features extracted by the convolution operation with a large receptive field on the spatial top - view, which is convenient for quickly distinguishing voxels falling into different pixels in the spatial top - view, while the 16 - dimensional information is the features independently obtained by each voxel, which helps to distinguish voxels belonging to the same pixel in the top - view.

[0215] Since the voxel encoder takes very little time, the main time - consuming part of the point cloud data processing system is mainly in the 2D basic network. Fusing voxel features and pixel features can make the time - consumption of the point cloud data processing system similar to that of the 2D network and obtain more discriminative voxel features.

[0216] A semantic classifier is used to obtain the first voxel splicing features corresponding to the voxels and identify the semantic types of the voxels according to the first voxel splicing features, so as to send the indices of the target voxels to the second feature splicing module according to the semantic types of the voxels.

[0217] The second feature splicing module is used to assign the second pixel features to each voxel according to the situation of N voxels falling into each pixel in the spatial top view, obtaining NxL-dimensional information. Specifically, at least one voxel falling into the same pixel can be assigned the L-dimensional second pixel features corresponding to this pixel, and the NxL-dimensional information and the Nx16-dimensional information of each voxel are spliced together to obtain the second voxel splicing features of NxC dimensions. Assuming that the indexes of the target voxels sent by the semantic classifier to this module are M, the MxC-dimensional second voxel splicing features corresponding to the target voxels can be screened out. Further, the MxC-dimensional second voxel splicing features are sent to the Knn-Transformer module.

[0218] When the existing transformer calculates the attention matrix, it will calculate the similarity relationship between any one voxel and all other voxels, generating an MxM attention matrix, with high computational complexity and large memory occupancy. In the Knn-Transformer module of this solution, by inputting the MxC-dimensional second voxel splicing features (M is the number of target voxels, and C is the dimension of the target voxel splicing features), based on the position information of the target voxels, k target neighbor voxels are directly found for each target voxel, and then a local attention matrix graph of Mxk is generated, and the optimized voxel splicing features are output, and then the optimized voxel splicing features are sent to the center regression module. This solution can greatly reduce the computational complexity by using the Knn-Transformer module instead of the existing transformer model.

[0219] The center regression module is used to input the optimized voxel splicing features into the trained offset prediction neural network to obtain the offset ΔP=(Δx, Δy, Δz) of the target voxel, so as to perform position offset on the target voxel based on the position coordinates P=(x, y, z) of the target voxel center and the offset of the target voxel center, that is, perform center regression on the target voxel to obtain the offset voxel, and send the offset voxels in the three-dimensional space to the clustering pseudo heat map generation module. After the above processing, each foreground point has gathered towards its respective instance, and the spatial distance between different voxels has also been increased. Among them, ΔP represents the offset of the target voxel center, Δx represents the offset of the target voxel center on the x-axis, Δy represents the offset of the target voxel center on the y-axis, and Δz represents the offset of the target voxel center on the z-axis. P represents the position coordinates of the target voxel center, x represents the coordinate of the target voxel center on the x-axis, y represents the coordinate of the target voxel center on the y-axis, and z represents the coordinate of the foreground center on the z-axis.

[0220] The clustering pseudo-thermal map generation module is used to project a two-dimensional image from the top-down view of the offset voxels in three-dimensional space onto a target preset image, and draw a pseudo-thermal map according to the number of offset voxels in each pixel of the target preset image. This pseudo-thermal map is similar to the role of predicting targets by a target detection network, but does not require obtaining by training a neural network. In reality, different target voxels rarely overlap in the top-down view, so the top-down view is a relatively ideal space for processing information of different instances. After the information is reduced to 2D, some efficient computer vision operators can be utilized. During the projection process, it is defined that each pixel of the 2D photo (preset image) corresponds to a 0.2m * 0.2m spatial size in the two-dimensional image of the top-down view of the offset voxels. All offset voxels can be projected onto a 2D photo with a size of (H, W). Record the number of offset voxels falling into each pixel. Simply using the number of points as the feature information of the pixel, a clustered pseudo-thermal map is generated. Based on this pseudo-thermal map, the pixel with the highest local peak (the pixel with the highest pixel value in the preset area) can represent the target center. Optionally, based on the sliding window-based 2D max pooling operation, the position of the local peak of the pixels in the 2D photo can be efficiently extracted to obtain the target center. After obtaining the target center, an instance ID can be assigned to each target center. All offset voxels will be assigned the instance ID of the target center closest to themselves based on the nearest neighbor principle (if an offset voxel is equidistant from multiple target centers, the instance ID of one of the target centers with the same distance as the offset voxel will be used as the instance of the offset voxel). All target voxels thus obtain instance IDs and do not rely on heuristic clustering algorithms. Coupled with the semantic types of all voxels already predicted in the semantic classifier, an efficient panoramic segmentation task can be completed.

[0221] After obtaining the offset voxels in three-dimensional space, determining which object a target voxel specifically belongs to can complete the task of instance segmentation. However, predicting target-level information through a target detection network increases the system time consumption on the one hand, and on the other hand, the targets predicted by the target detection network may have inconsistent results with the aggregated voxel clusters in terms of quantity and position. The target center determined by the clustering pseudo-thermal map generation module has a natural correspondence with the translated target voxel cluster (the cluster composed of offset voxels). As long as there is a place where target voxels are aggregated, a target center will definitely be generated. In contrast, when determining the target center based on a target detection network, the number of detected target objects cannot be guaranteed, and there is no guarantee of a good correspondence in position between the detected target center and the information of the aggregated target voxel cluster.

[0222] The generation of the target center is based on the aggregation of target voxels towards the instance center. In the driving scenario, the voxels belonging to small targets generally have a better regression effect towards the center than large targets. Large targets are more likely to be observed only locally in the lidar sensor. For example, a car predicts one instance ID, while a bus predicts multiple instance IDs. That is, the voxels belonging to the car have a better aggregation effect towards the car center. After the above operations, the correct instance ID is obtained. For the voxels belonging to the bus, when the bus is close to the lidar, only one side of the bus is scanned by the lidar, and the offset prediction neural network does not have enough information to accurately infer the offset of the voxels belonging to the bus. Referring to the segmentation result in the enlarged graph of the bus, it can be determined that the voxels belonging to the bus are predicted to have 4 target centers, thus generating 4 instance IDs instead of the expected single instance ID.

[0223] In order to segment large targets well even when the center regression is not ideal, through the center regrouping module, the target centers can be grouped once, and the target centers divided into the same group have the same instance ID. The specific method is as follows:

[0224] 1. Based on the pseudo heat map, use the window 2D mean pooling operation to count the number of voxels of different semantic categories near each pixel (within the sliding window), and determine the semantic category of each pixel based on the principle of maximum number voting.

[0225] 2. Assign a prior size (prior size data) to each semantic category. This size is derived from the statistics of the object detection dataset. For example, for a car, the average size of the car can be statistically determined based on the dataset, and based on this size, the minimum radius R = 0.5 * min(L, W) of each semantic category is established, where W and L represent the average width and length of the object.

[0226] 3. After the above steps, each target center has its own minimum radius, which can be regarded as its own bandwidth. Starting from a certain target center for traversal, in this traversal, it is regarded as the basic center C b , with the instance ID G b , and the minimum bandwidth B b . If there is another target center C t within the bandwidth, whose ID is G t , then all centers with the ID G t will be re - assigned the instance ID of the basic center, that is, G b . In addition to the bandwidth constraint, a constraint that the basic center C b and the target center C t must be of the same semantic category can be added, so that the effect for different semantic categories in crowded scenarios will be better, such as the distinction of pedestrians very close to the vehicle.

[0227] Until the traversal of all target centers is completed, the target centers that meet the conditions of step 3 will be re-assigned instance IDs. After the center re-grouping, the buses originally divided into multiple IDs can be completely segmented, so that each bus has exactly one instance ID.

[0228] The panoramic segmentation effect of the panoramic segmentation system proposed in this scheme can be seen in Table 4 and Table 5.

[0229] Table 4 Comparison table of segmentation effects of panoramic segmentation using features extracted by 2D network and voxel stitching features

[0230] Method PQ mIoU Top-View Feature 60.6 64.5 Voxel Stitching Feature 61.7 65.7

[0231] As shown in Table 4, on the semanticKITTI dataset, where the top-down view feature represents the method of extracting features by the 2D network, compared with the top-down view feature method, the proposed scheme using voxel stitching features improves the PQ index by 1.1% and the mIoU index by 1.2% in panoramic segmentation.

[0232] Table 5 Comparison table of panoramic segmentation effects with and without Knn-Transformer module

[0233] Method PQ <![CDATA[PQ Th > Without Knn-Transformer Module 61.2 68.1 With Knn-Transformer Module 61.7 69.3

[0234] As shown in Table 5, for the panoramic segmentation task, the Knn-Transformer module is used to strengthen the target voxel stitching features to predict the instance center offset of each voxel, and then subsequent instance segmentation is performed. Compared with the method without the Knn-Transformer module, the PQ index is improved by 0.5%, and the PQ Th index is improved by 1.2%. PQ Th represents the panoramic segmentation accuracy of the target voxel.

[0235] As Figure 14 shows, in the Knn-Transformer model, the value of the number k of target neighbor voxels affects the segmentation accuracy. As k increases, the panoramic segmentation accuracy PQ of the target voxel Th increases, but it basically saturates after reaching 25, and even if k is increased to the maximum value N, the accuracy will not change. When k is increased to the maximum value, it means that the similarity relationship is calculated between any voxel and all other voxels, which is equivalent to the existing transformer network. When the Knn-Transformer module has a value of k = 25, it already achieves the same accuracy as the existing transformer in the point cloud panoramic segmentation task, but our computational complexity is much lower than that of the existing transformer. If the input has M elements, the computational complexity of the existing transformer is M 2, and the computational complexity of this solution is Mxk, where k takes the value of 25. Since the number of voxels in a frame of point cloud is often in the thousands or tens of thousands, the Knn-Transformer model based on this solution can greatly reduce the computational complexity of the attention matrix map.

[0236] Figure 15 is a flowchart of panoramic segmentation provided by an embodiment of the present invention. As Figure 15 shown, the radar point cloud can be input into the voxel encoder. The voxel encoder divides each point in the point cloud into multiple voxels respectively, and obtains the voxel features of each voxel according to the point features of the points included in each voxel, and inputs the voxel features into the 2D top view encoder to generate a pixel feature map to be processed through the 2D top view encoder, and sends the pixel feature map to be processed to the 2D basic network. The pixel features of each voxel in the preset image are obtained through the 2D basic network, and the voxel features and the pixel features are feature-stitched to obtain voxel stitching features. Thus, the voxel stitching features are respectively input into the semantic classifier and the Knn-Transformer module, and further, the semantic types of each voxel and the points in the point cloud are obtained according to the output result of the semantic classifier, and the optimized voxel stitching features are obtained based on the Knn-Transformer module using the target voxel stitching features in the voxel stitching features. Furthermore, the optimized voxel stitching features are input into the instance segmenter for instance segmentation.

[0237] Figure 16 is a schematic structural diagram of a point cloud data processing device provided by an embodiment of the present invention. As Figure 16 shown, the device includes: a voxel feature acquisition module 710, a pixel feature acquisition module 720, and a feature stitching module 730, where:

[0238] The voxel feature acquisition module 710 is configured to divide each point in the point cloud into multiple voxels respectively, and obtain the voxel features of each voxel according to the point features of the points included in each voxel;

[0239] The pixel feature acquisition module 720 is configured to project each voxel onto a preset image, and obtain the pixel features of each voxel in the preset image according to the projection result;

[0240] The feature stitching module 730 is configured to perform feature stitching on the voxel features and the pixel features to form voxel stitching features corresponding to the voxels.

[0241] The technical solution of this embodiment is to divide each point in the point cloud into multiple voxels respectively, and obtain the voxel features of each voxel according to the point features of the points included in each voxel. Then, project each voxel onto a preset image, and obtain the pixel features of each voxel in the preset image according to the projection result. Thus, the voxel features and the pixel features are feature - stitched to form voxel - stitched features corresponding to the voxels. Fusing the voxel features and the pixel features can improve the expression ability of the voxel features, and using the voxel - stitched features can quickly identify and classify the targets in the point cloud, solving the problems in the prior art that the features extracted based on the 2D network have poor discrimination for voxels, and the 3D network has large memory occupation and serious time - consumption when identifying the features of voxels, improving the expression ability of the voxel features and the data processing effect of the point cloud data.

[0242] Optionally, the voxel feature acquisition module 710 is specifically configured to obtain the voxel features of each voxel according to the point features, including: generating a first matrix graph to be processed according to the point features of the points included in each voxel and the number of voxels; using a multi - layer neural network and a max - pooling layer to process the feature dimension of the first matrix graph to be processed, and obtaining the voxel features of each voxel.

[0243] Optionally, the pixel feature acquisition module 720 is specifically configured to obtain a second matrix graph to be processed according to the projection result; using a multi - layer neural network and a max - pooling layer to process the feature dimension of the second matrix graph to be processed, and obtaining a pixel feature graph to be processed; obtaining a two - dimensional semantic segmentation network; inputting the pixel feature graph to be processed into the two - dimensional semantic segmentation network, and obtaining the pixel features of each voxel in the preset image.

[0244] Optionally, the point cloud data processing device further includes a semantic segmentation module, configured to input the voxel - stitched features corresponding to the voxels into a semantic classifier to obtain voxels with semantic types; and obtaining a point cloud with semantic types according to the voxels with semantic types and the points in the point cloud of the voxels.

[0245] Optionally, the point cloud data processing device further includes an instance segmentation module, configured to obtain target voxels according to the voxels with semantic types; obtaining the target voxel - stitched features of the target voxels based on the voxel - stitched features and the target voxels; and performing instance segmentation on the target voxels according to the target voxel - stitched features.

[0246] Optionally, the instance segmentation module is specifically configured to obtain target neighbor voxels that match the target voxel based on the nearest neighbor node algorithm; obtain the target neighbor voxel splicing features of the target neighbor voxels based on the voxel splicing features; create a local attention matrix map according to the target voxel splicing features and the target neighbor voxel splicing features; obtain the optimized voxel splicing features of the target voxel according to the local attention matrix map; and perform instance segmentation on the target voxel according to the optimized voxel splicing features.

[0247] Optionally, the instance segmentation module includes an offset acquisition unit, an offset voxel acquisition unit, a pixel value acquisition unit, a peak pixel acquisition unit, a first instance identification assignment unit, and a second instance identification assignment unit; wherein, the offset acquisition unit is configured to input the target voxel splicing features into a trained offset prediction neural network to obtain the offset of the target voxel; the offset voxel acquisition unit is configured to obtain an offset voxel based on the offset and the target voxel; the pixel value acquisition unit is configured to project the offset voxel onto a target preset image from above, and obtain the pixel value of each pixel according to the number of offset voxels in each pixel of the target preset image; the first instance identification assignment unit is configured to obtain the pixel with the highest pixel value in a preset area according to the pixel value; determine a target center according to the pixel with the highest pixel value, and assign a matching instance identification to the target center; the second instance identification assignment unit is configured to obtain the instance identification of the target voxels belonging to the target center according to the instance identification of the target center.

[0248] Optionally, the offset acquisition unit is further configured to input the optimized voxel splicing features into a trained offset prediction neural network to obtain the offset of the target voxel.

[0249] Optionally, the pixel value acquisition unit is specifically configured to obtain a preset spatial dimension of the offset voxel; obtain a target preset image based on the preset spatial dimension; wherein, the preset spatial dimension matches each pixel in the target preset image; and project the offset voxel onto the target preset image from above.

[0250] Optionally, the peak pixel acquisition unit is specifically configured to compare the pixel values of each pixel in the preset area to obtain the pixel with the highest pixel value in the preset area.

[0251] Optionally, the first instance identification assignment unit is specifically configured to obtain a current target center and prior size data matching the semantic type to be processed; obtain a detection range to be detected that matches the current target center according to the current target center and the prior size data; and when it is determined that there is no other target center within the detection range to be detected, do not update the instance identification of the current target center.

[0252] Optionally, the first instance identifier allocation unit is specifically configured to obtain the current target center and prior size data matching the semantic type to be processed; obtain a detection range to be detected that matches the current target center according to the current target center and the prior size data; when it is determined that there are other target centers within the detection range to be detected, perform a merging process on the instance identifiers of the target centers stored within the detection range to be detected.

[0253] Optionally, the first instance identifier allocation unit is specifically configured to obtain the current target center and prior size data matching the semantic type to be processed; obtain a detection range to be detected that matches the current target center according to the current target center and the prior size data; when it is determined that there are other target centers within the detection range to be detected and the semantic types of the other target centers are the same as that of the current target center, perform a merging process on the instance identifiers of the target centers stored within the detection range to be detected.

[0254] The point cloud data processing device provided by the embodiments of the present invention can execute the point cloud data processing method provided by any embodiment of the present invention, and has corresponding functional modules and beneficial effects for executing the method.

[0255] Figure 17 The structural schematic diagram of an electronic device that can be used to implement the embodiments of the present invention is shown. As Figure 17 shown, the electronic device 10 includes at least one processor 11, and a memory communicatively connected to at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc. Among them, the memory stores a computer program executable by at least one processor. The processor 11 can execute various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. The input / output (I / O) interface 15 is also connected to the bus 14.

[0256] Multiple components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disc, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0257] The processor 11 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the method for processing point cloud data.

[0258] In some embodiments, the method for processing cloud data can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the method for processing point cloud data described above can be executed. Alternatively, in other embodiments, the processor 11 can be configured to execute the method by any other suitable means (e.g., by means of firmware) to respectively divide each point in the point cloud into a plurality of voxels, and obtain the voxel features of each voxel according to the point features of the points included in each voxel; project each voxel onto a preset image, and obtain the pixel features of each voxel in the preset image according to the projection result; perform feature splicing on the voxel features and the pixel features to form voxel splicing features corresponding to the voxels. The processor 11 can also be configured to execute to obtain a point cloud with semantic types, input the target points in the point cloud into a trained offset prediction neural network to obtain the offsets of the target points; obtain offset points based on the offsets and the target points; perform a top-down projection of the offset points onto a preset image, and obtain the pixel value of each pixel according to the number of offset points in each pixel in the image; obtain the pixel with the highest pixel value in a preset area according to the pixel value; determine the target center according to the pixel with the highest pixel value, and assign a matching instance identifier to the target center; obtain the instance identifier of the target points belonging to the target center according to the instance identifier of the target center.

[0259] The various embodiments of the systems and techniques described above in this document can be implemented in digital electronic circuitry, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on a chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be a special-purpose or general-purpose programmable processor that receives data and instructions from a storage system, at least one input device, and at least one output device, and transmits the data and instructions to the storage system, the at least one input device, and the at least one output device.

[0260] The computer programs for implementing the methods of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, a special purpose computer, or other programmable data processing apparatus, such that the computer programs, when executed by the processor, cause the functions / operations specified in the flowchart and / or block diagram to be implemented. The computer programs can be executed entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine or entirely on the remote machine or server.

[0261] In the context of the present invention, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. The computer-readable storage medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, the computer-readable storage medium can be a machine-readable signal medium. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0262] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and a pointing device (e.g., a mouse or a trackball) through which the user can provide input to the electronic device. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0263] The systems and techniques described herein can be implemented in a computing system including backend components (e.g., as a data server), or a computing system including middleware components (e.g., an application server), or a computing system including frontend components (e.g., a user computer having a graphical user interface or a web browser through which the user can interact with an implementation of the systems and techniques described herein), or a computing system including any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected to each other by digital data communication in any form or medium (e.g., a communication network). Examples of communication networks include: local area network (LAN), wide area network (WAN), blockchain network, and the Internet.

[0264] The computing system can include a client and a server. The client and the server are generally far from each other and usually interact through a communication network. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system and solves the defects of difficult management and weak business scalability existing in traditional physical hosts and VPS services.

[0265] It should be understood that various forms of the processes shown above can be used, with steps reordered, added, or deleted. For example, the steps recited in the present invention can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved, and no limitation is made herein.

[0266] The above specific embodiments do not constitute a limitation on the protection scope of the present invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for processing point cloud data, characterized in that Including: Dividing each point in the point cloud into multiple voxels respectively, and obtaining the voxel features of each voxel according to the point features of the points included in each voxel; Projecting each voxel onto a preset image, and obtaining the pixel features of each voxel in the preset image according to the projection result; Performing feature splicing on the voxel features and the pixel features to form voxel splicing features corresponding to the voxels; Inputting the voxel splicing features corresponding to the voxels into a semantic classifier to obtain voxels with semantic types; obtaining target voxels according to the voxels with semantic types; obtaining the target voxel splicing features of the target voxels based on the voxel splicing features and the target voxels; performing instance segmentation on the target voxels according to the target voxel splicing features; Obtaining a point cloud with semantic types according to the voxels with semantic types and the points in the voxel in the point cloud.

2. The method according to claim 1, wherein The obtaining the voxel features of each voxel according to the point features of the points included in each voxel includes: Generating a first matrix graph to be processed according to the point features of the points included in each voxel and the number of the voxels; Performing feature dimension processing on the first matrix graph to be processed by using a multi-layer neural network and a max pooling layer to obtain the voxel features of each voxel.

3. The method according to claim 1, wherein The obtaining the pixel features of each voxel in the preset image according to the projection result includes: Obtaining a second matrix graph to be processed according to the projection result; Performing feature dimension processing on the second matrix graph to be processed by using a multi-layer neural network and a max pooling layer to obtain a pixel feature graph to be processed; Obtaining a two-dimensional semantic segmentation network; Inputting the pixel feature graph to be processed into the two-dimensional semantic segmentation network to obtain the pixel features of each voxel in the preset image.

4. The method according to claim 1, wherein The performing instance segmentation on the target voxels according to the target voxel splicing features includes: Obtaining target neighbor voxels matched with the target voxels based on the nearest neighbor node algorithm; Obtaining the target neighbor voxel splicing features of the target neighbor voxels based on the voxel splicing features; Creating a local attention matrix graph according to the target voxel splicing features and the target neighbor voxel splicing features; Obtaining optimized voxel splicing features of the target voxels according to the local attention matrix graph; Performing instance segmentation on the target voxels according to the optimized voxel splicing features.

5. The method according to claim 1, characterized in that, The performing instance segmentation on the target voxels according to the target voxel splicing features includes: Inputting the target voxel splicing features into a trained offset prediction neural network to obtain the offset of the target voxels; Obtaining offset voxels based on the offset and the target voxels; Performing a top-down projection of the offset voxels onto a target preset image, and obtaining the pixel value of each pixel according to the number of offset voxels in each pixel in the target preset image; Obtaining the pixel with the highest pixel value in a preset area according to the pixel value; Determining a target center according to the pixel with the highest pixel value and assigning a matching instance identifier to the target center; Obtaining the instance identifier of the target voxels belonging to the target center according to the instance identifier of the target center.

6. A processing device for point cloud data, characterized in that, Including: A voxel feature acquisition module, configured to divide each point in the point cloud into multiple voxels respectively, and acquire the voxel features of each voxel according to the point features of the points divided into each voxel; A pixel feature acquisition module, configured to project each voxel onto a preset image, and acquire the pixel features of each voxel in the preset image according to the projection result; A feature splicing module, configured to splice the voxel features and the pixel features to form voxel splicing features corresponding to the voxels; The point cloud data processing device further includes a semantic segmentation module, configured to input the voxel splicing features corresponding to the voxels into a semantic classifier to obtain voxels with semantic types; and acquire a point cloud with semantic types according to the voxels with semantic types and the points in the point cloud in the voxels; The point cloud data processing device further includes an instance segmentation module, configured to acquire target voxels according to the voxels with semantic types; acquire target voxel splicing features of the target voxels based on the voxel splicing features and the target voxels; and perform instance segmentation on the target voxels according to the target voxel splicing features.

7. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the point cloud data processing method according to any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions, and the computer instructions are used to implement the point cloud data processing method according to any one of claims 1-5 when executed by a processor.

Images