Semantic segmentation method and semantic segmentation device for point cloud data and electronic equipment

Abstract

The invention provides a semantic segmentation method and a semantic segmentation device for point cloud data and electronic equipment. The semantic segmentation method comprises the steps that voxels obtained through conversion based on input point cloud data are processed, and initial features are obtained; performing down-sampling processing on the initial feature for a preset number of times to obtain a down-sampling feature; global features are extracted from the down-sampling features, the global features and aerial view features are fused, fused features are obtained, and the aerial view features are obtained through conversion from the global features; performing up-sampling processing of a preset number of times on the fusion feature by combining a result of the down-sampling processing of the preset number of times to obtain an up-sampling feature; and semantic tag prediction is carried out based on the up-sampling features, so that a semantic segmentation result is obtained.

Description

Technical Field

[0001] The present disclosure generally relates to the field of computer vision, and more specifically, to a semantic segmentation method, a semantic segmentation device and an electronic device for point cloud data. Background Art

[0002] At present, three-dimensional semantic segmentation can be divided into point cloud-based, image-based and multimodal methods according to the input data modality. Point cloud data can be obtained through lidar sensors. As a set of coordinates of three-dimensional points, it can be used to reflect the geometric information of a scene or object. Existing semantic segmentation based on point cloud data can generally be divided into point-based, projection-based and voxel-based methods. Point-based methods can use point features and position features to learn the correlation of adjacent points to improve the recognition accuracy of semantic segmentation models. Projection-based methods can convert point cloud data into bird's-eye view (BEV) data, and then use two-dimensional neural networks to extract and process features of the bird's-eye view data, and reflect the results in three-dimensional space. Voxel-based methods divide the three-dimensional space into multiple uniform or non-uniform voxel blocks, and then use sparse convolution to extract features and predict them.

[0003] However, the current semantic segmentation based on point cloud data has several shortcomings: 1) The semantic segmentation model can often only process a small receptive field and has poor perception of sparse and large point cloud space; 2) The semantic segmentation model can only process limited context information and cannot fuse global information and local information to help achieve point cloud segmentation; 3) It ignores the possibility of migrating advanced technologies in the two-dimensional image field to the three-dimensional field. Summary of the invention

[0004] The embodiments of the present disclosure provide a semantic segmentation method, a semantic segmentation device and an electronic device for point cloud data. By fusing voxel features with bird's-eye view features, the ability of the semantic segmentation model to extract global features can be enhanced, and the semantic segmentation accuracy can be improved.

[0005] According to a first aspect of an embodiment of the present disclosure, a semantic segmentation method for point cloud data is provided, the semantic segmentation method comprising: processing voxels obtained based on the conversion of input point cloud data to obtain initial features; obtaining downsampled features by downsampling the initial features a preset number of times; extracting global features from the downsampled features, and fusing the global features with bird's-eye view features to obtain fused features, wherein the bird's-eye view features are converted from the global features; performing upsampling on the fused features a preset number of times in combination with the results of the downsampling processing a preset number of times to obtain upsampled features; performing semantic label prediction based on the upsampled features to obtain a semantic segmentation result.

[0006] Optionally, the semantic segmentation method further includes: converting the point cloud data into a plurality of voxels by performing voxelization processing on the point cloud data.

[0007] Optionally, the downsampling feature is obtained by downsampling the initial feature a preset number of times, including: extracting local features from the initial features through a first plurality of dynamic spatial aggregation convolutional layers, extracting global features from the local features through a first Transformer module, and downsampling the global features through a downsampling layer to obtain a first downsampling result; extracting the i-th local feature from the i-th downsampling result through the first plurality of dynamic spatial aggregation convolutional layers, extracting the i-th global feature from the i-th local feature through the first Transformer module, and downsampling the i-th global feature through the downsampling layer to obtain the i+1-th downsampling result, wherein 1≤i≤N-1, N is the preset number of times and N is a natural number greater than or equal to 2; and using the N-th downsampling result as the downsampling feature.

[0008] Optionally, each dynamic spatial aggregation convolution layer performs the following operations: convolving the input initial feature or the i-th downsampling result with a convolution kernel of a first preset size to obtain a first feature; convolving the first feature with multiple convolution kernels of different sizes to obtain multiple second features, wherein the multiple convolution kernels of different sizes are obtained by decomposing the convolution kernel of the second preset size in length, width and depth directions; determining the output of the corresponding dynamic spatial aggregation convolution layer based on the sum of the products of the first feature and each of the second features and the corresponding weights, wherein the weights corresponding to the multiple second features are calculated using a preset function.

[0009] Optionally, extracting global features from the downsampled features includes: extracting local features from the downsampled features through multiple dynamic spatial aggregation convolutional layers, and extracting global features from the local features through a Transformer module.

[0010] Optionally, the fusing of the global features and the bird's-eye view features to obtain the fused features includes: compressing the global features into two-dimensional bird's-eye view features by densifying the global features and taking the maximum value in the depth direction; obtaining updated bird's-eye view features by performing two-dimensional deformable convolution on the two-dimensional bird's-eye view features; and fusing the global features with the updated bird's-eye view features to obtain the fused features.

[0011] Optionally, the fusing the global feature with the updated bird's-eye view feature to obtain the fused feature includes: connecting the global feature with the updated bird's-eye view feature through a first fully connected layer to obtain the fused feature; or, processing the updated bird's-eye view feature through a second fully connected layer, multiplying the processing result with the global feature point by point, and adding the multiplication result to the global feature to obtain the fused feature; or, obtaining the fused feature by applying a preset deformation function to the global feature, the sampling points in the updated bird's-eye view feature corresponding to the global feature, and the updated bird's-eye view feature.

[0012] Optionally, the method combines the results of the preset number of downsampling processes and performs upsampling processes on the fused features a preset number of times to obtain upsampling features, including: upsampling the fused features through an upsampling layer, fusing the features obtained by the upsampling process with the Nth downsampling result to obtain a first fusion result, and convolving the first fusion result through a second plurality of dynamic spatial aggregation convolution layers to obtain a first upsampling result; upsampling the i-th upsampling result through the upsampling layer, fusing the features obtained by the i-th upsampling process with the Ni-th downsampling result to obtain an i+1-th fusion result, and convolving the i+1-th fusion result through the second plurality of dynamic spatial aggregation convolution layers to obtain an i+1-th upsampling result; using the N-th upsampling result as the upsampling feature.

[0013] Optionally, the semantic segmentation method is executed by a semantic segmentation model, wherein the semantic segmentation model is trained by the following steps: performing sample mixing enhancement on the cloud data serving as the initial sample, and performing point cloud structure enhancement on the point cloud data after the sample mixing enhancement to obtain enhanced point cloud data; performing voxelization on the enhanced point cloud data to convert the point cloud data into a plurality of voxel sample data for training; and training the semantic segmentation model based on the voxel sample data and a loss function constructed using a cross entropy function and a Lovasz-Softmax function.

[0014] Optionally, the sample mixing enhancement of the point cloud data used as the initial sample includes: performing the following processing for every two frames of point cloud data: reading the two frames of point cloud data and the label of each point in the two frames of point cloud data; calculating the polar coordinate azimuth of each point in each frame of point cloud data; exchanging the points in the first frame of point cloud data within a preset polar coordinate azimuth range with the points in the second frame of point cloud data; copying all points labeled with preset labels in the exchanged second frame of point cloud data to the first frame of point cloud data; updating the labels of each point in the first frame of point cloud data to obtain the point cloud data after the sample mixing enhancement.

[0015] Optionally, the sample mixing enhancement of the point cloud data used as the initial sample includes: performing the following processing for every two frames of point cloud data: reading the two frames of point cloud data and the label of each point in the two frames of point cloud data; calculating the elevation angle of each point in each frame of point cloud data; evenly dividing the preset elevation angle interval into a preset number of sub-elevation angle intervals, and dividing the sub-elevation angle intervals into odd elevation angle intervals and even elevation angle intervals according to the parity of the sequence numbers of the sub-elevation angle intervals; exchanging the points in the second frame of point cloud data whose elevation angles belong to the even elevation angle interval with the points in the first frame of point cloud data whose elevation angles belong to the even elevation angle interval, or exchanging the points in the second frame of point cloud data whose elevation angles belong to the odd elevation angle interval with the points in the first frame of point cloud data whose elevation angles belong to the odd elevation angle interval; updating the labels of each point in the first frame of point cloud data to obtain the point cloud data after sample mixing enhancement.

[0016] Optionally, the point cloud structure enhancement of the point cloud data after sample mixing enhancement includes: rotating the point cloud data after sample mixing enhancement by a preset angle around a first coordinate axis; flipping the rotated point cloud data around an arbitrary coordinate axis; scaling the flipped point cloud data; adding noise data to the scaled point cloud data to obtain point cloud structure enhanced point cloud data.

[0017] According to a first aspect of an embodiment of the present disclosure, a semantic segmentation device for point cloud data is provided, the semantic segmentation device comprising: a data processing module, configured to process voxels obtained by conversion based on input point cloud data to obtain initial features; a downsampling module, configured to obtain downsampled features by performing downsampling processing on the initial features a preset number of times; a feature fusion module, configured to extract global features from the downsampled features, and fuse the global features with bird's-eye view features to obtain fused features, wherein the bird's-eye view features are converted from the global features; an upsampling module, configured to perform upsampling processing on the fused features a preset number of times in combination with the results of the downsampling processing a preset number of times to obtain upsampled features; a prediction module, configured to perform semantic label prediction based on the upsampled features to obtain a semantic segmentation result.

[0018] Optionally, the semantic segmentation device further includes: a data conversion module, configured to convert the point cloud data into a plurality of voxels by performing voxelization processing on the point cloud data.

[0019] Optionally, the downsampling module is configured to: extract local features from the initial features through a first plurality of dynamic spatial aggregation convolutional layers, extract global features from the local features through a first Transformer module, and downsample the global features through a downsampling layer to obtain a first downsampling result; extract the i-th local feature from the i-th downsampling result through the first plurality of dynamic spatial aggregation convolutional layers, extract the i-th global feature from the i-th local feature through the first Transformer module, and downsample the i-th global feature through the downsampling layer to obtain the i+1-th downsampling result, wherein 1≤i≤N-1, N is the preset number of times and N is a natural number greater than or equal to 2; and use the N-th downsampling result as the downsampling feature.

[0020] Optionally, each dynamic spatial aggregation convolution layer performs the following operations: convolving the input initial feature or the i-th downsampling result with a convolution kernel of a first preset size to obtain a first feature; convolving the first feature with multiple convolution kernels of different sizes to obtain multiple second features, wherein the multiple convolution kernels of different sizes are obtained by decomposing the convolution kernel of the second preset size in length, width and depth directions; determining the output of the corresponding dynamic spatial aggregation convolution layer based on the sum of the products of the first feature and each of the second features and the corresponding weights, wherein the weights corresponding to the multiple second features are calculated using a preset function.

[0021] Optionally, the feature fusion module is configured to: extract local features from the downsampled features through multiple dynamic spatial aggregation convolutional layers, and extract global features from the local features through a Transformer module.

[0022] Optionally, the feature fusion module is further configured to: compress the global feature into a two-dimensional bird's-eye view feature by densifying the global feature and taking the maximum value in the depth direction; obtain an updated bird's-eye view feature by performing a two-dimensional deformable convolution process on the two-dimensional bird's-eye view feature; and fuse the global feature with the updated bird's-eye view feature to obtain the fused feature.

[0023] Optionally, the feature fusion module is further configured to: connect the global feature with the updated bird's-eye view feature through a first fully connected layer to obtain the fused feature; or, process the updated bird's-eye view feature through a second fully connected layer, multiply the processing result with the global feature point by point, and add the multiplication result to the global feature to obtain the fused feature; or, obtain the fused feature by applying a preset deformation function to the global feature, the sampling points in the updated bird's-eye view feature corresponding to the global feature, and the updated bird's-eye view feature.

[0024] Optionally, the upsampling module is configured to: upsample the fused features through the upsampling layer, fuse the features obtained by the upsampling process with the Nth downsampling result to obtain a first fusion result, and convolve the first fusion result through a second plurality of dynamic spatial aggregation convolution layers to obtain a first upsampling result; upsample the i-th upsampling result through the upsampling layer, fuse the features obtained by the i-th upsampling process with the Ni-th downsampling result to obtain an i+1-th fusion result, and convolve the i+1-th fusion result through the second plurality of dynamic spatial aggregation convolution layers to obtain an i+1-th upsampling result; and use the N-th upsampling result as the upsampling feature.

[0025] Optionally, the semantic segmentation device is implemented by a semantic segmentation model, wherein the semantic segmentation model is trained by the following operations: performing sample mixing enhancement on the cloud data serving as the initial sample, and performing point cloud structure enhancement on the point cloud data after the sample mixing enhancement to obtain enhanced point cloud data; performing voxelization on the enhanced point cloud data to convert the point cloud data into a plurality of voxel sample data for training; and training the semantic segmentation model based on the voxel sample data and a loss function constructed using a cross entropy function and a Lovasz-Softmax function.

[0026] Optionally, the sample mixing enhancement of the point cloud data used as the initial sample includes: performing the following processing for every two frames of point cloud data: reading the two frames of point cloud data and the label of each point in the two frames of point cloud data; calculating the polar coordinate azimuth of each point in each frame of point cloud data; exchanging the points in the first frame of point cloud data within a preset polar coordinate azimuth range with the points in the second frame of point cloud data; copying all points labeled with preset labels in the exchanged second frame of point cloud data to the first frame of point cloud data; updating the labels of each point in the first frame of point cloud data to obtain the point cloud data after the sample mixing enhancement.

[0027] Optionally, the sample mixing enhancement of the point cloud data used as the initial sample includes: performing the following processing for every two frames of point cloud data: reading the two frames of point cloud data and the label of each point in the two frames of point cloud data; calculating the elevation angle of each point in each frame of point cloud data; evenly dividing the preset elevation angle interval into a preset number of sub-elevation angle intervals, and dividing the sub-elevation angle intervals into odd elevation angle intervals and even elevation angle intervals according to the parity of the sequence numbers of the sub-elevation angle intervals; exchanging the points in the second frame of point cloud data whose elevation angles belong to the even elevation angle interval with the points in the first frame of point cloud data whose elevation angles belong to the even elevation angle interval, or exchanging the points in the second frame of point cloud data whose elevation angles belong to the odd elevation angle interval with the points in the first frame of point cloud data whose elevation angles belong to the odd elevation angle interval; updating the labels of each point in the first frame of point cloud data to obtain the point cloud data after sample mixing enhancement.

[0028] Optionally, the point cloud structure enhancement of the point cloud data after sample mixing enhancement includes: rotating the point cloud data after sample mixing enhancement by a preset angle around a first coordinate axis; flipping the rotated point cloud data around an arbitrary coordinate axis; scaling the flipped point cloud data; adding noise data to the scaled point cloud data to obtain point cloud structure enhanced point cloud data.

[0029] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, comprising at least one processor; and at least one memory storing computer executable instructions, wherein when the computer executable instructions are executed by the at least one processor, the at least one processor is prompted to execute the semantic segmentation method of point cloud data described in the first aspect.

[0030] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium storing instructions is provided, and when the instructions are executed by a processor, the semantic segmentation method of point cloud data as described above is implemented.

[0031] According to the semantic segmentation method, semantic segmentation device and electronic device of point cloud data disclosed in the present invention, it is possible to improve the perception ability of the semantic segmentation model for sparse and large point cloud space, realize the fusion of global information and local information, use 2D operators to enhance the ability to extract global information, and improve the accuracy of semantic segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 is a flowchart illustrating a semantic segmentation method performed by a semantic segmentation model according to an embodiment of the present disclosure.

[0033] Figure 2A is a flow chart illustrating operations performed by a DSAC layer according to an embodiment of the present disclosure.

[0034] Figure 2Bis a diagram illustrating an example of operations performed by a DSAC layer according to an embodiment of the present disclosure.

[0035] Figure 3A is a flowchart illustrating an operation of fusing global features with bird's-eye view features according to an embodiment of the present disclosure.

[0036] Figure 3B is a diagram illustrating an example of an operation of fusing a global feature with a bird's-eye view feature according to an embodiment of the present disclosure.

[0037] Figure 4 is a diagram illustrating an example of a semantic segmentation model according to an embodiment of the present disclosure.

[0038] Figure 5 is a flowchart illustrating a method for training a semantic segmentation model according to an embodiment of the present disclosure.

[0039] Figure 6 is a block diagram showing a semantic segmentation apparatus for point cloud data according to an embodiment of the present disclosure.

[0040] Figure 7 is a block diagram illustrating an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0041] The following description with reference to the accompanying drawings is provided to facilitate a comprehensive understanding of the various embodiments of the present disclosure as defined by the claims and their equivalents. This description includes various specific details to facilitate understanding but should be considered as exemplary only. Therefore, one of ordinary skill in the art will recognize that various changes and modifications can be made to the various embodiments described herein without departing from the scope and spirit of the present disclosure. In addition, for the sake of clarity and conciseness, descriptions of well-known functions and structures may be omitted.

[0042] The terms and expressions used in the following specification and claims are not limited to their dictionary meanings, but are merely used by the inventor to enable a clear and consistent understanding of the present disclosure. Therefore, it should be apparent to those skilled in the art that the following description of various embodiments of the present disclosure is provided for illustration purposes only and not for the purpose of limiting the present disclosure as defined in the appended claims and their equivalents.

[0043] It should be understood that the singular forms "a", "an", and "the" may also include plural references unless the context clearly indicates otherwise. Thus, for example, reference to "a component surface" includes reference to one or more such surfaces. When we refer to an element as being "connected" or "coupled" to another element, the one element may be directly connected or coupled to the other element, or the one element and the other element may establish a connection relationship through an intermediate element. In addition, "connected" or "coupled" as used herein may include wireless connection or wireless coupling.

[0044] The term "include" or "may include" refers to the presence of the corresponding disclosed functions, operations or components that can be used in various embodiments of the present disclosure, rather than limiting the presence of one or more additional functions, operations or features. In addition, the term "include" or "have" may be interpreted as indicating certain characteristics, numbers, steps, operations, constituent elements, components or combinations thereof, but should not be interpreted as excluding the possibility of the presence of one or more other characteristics, numbers, steps, operations, constituent elements, components or combinations thereof.

[0045] The term "or" used in various embodiments of the present disclosure includes any of the listed terms and all combinations thereof. For example, "A or B" may include A, may include B, or may include both A and B. When describing multiple (two or more) items, if the relationship between the multiple items is not clearly defined, the multiple items may refer to one, multiple, or all of the multiple items. For example, the description of "parameter A includes A1, A2, A3" may be implemented as parameter A including A1 or A2 or A3, or may be implemented as parameter A including at least two of the three items A1, A2, and A3.

[0046] Unless defined differently, all terms (including technical terms or scientific terms) used in the present disclosure have the same meanings as understood by those skilled in the art described in the present disclosure. Common terms as defined in dictionaries are interpreted as having meanings consistent with the context in the relevant technical field, and should not be interpreted ideally or overly formally unless clearly defined in the present disclosure.

[0047] At least some functions of the device or electronic device provided in the embodiments of the present disclosure can be implemented by an AI model, such as at least one module among multiple modules of the device or electronic device can be implemented by an AI model. Functions associated with AI can be performed by non-volatile memory, volatile memory and processor.

[0048] The processor may include one or more processors. In this case, the one or more processors may be general-purpose processors, such as a central processing unit (CPU), an application processor (AP), etc., or pure graphics processing units, such as a graphics processing unit (GPU), a visual processing unit (VPU), and / or an AI-specific processor, such as a neural processing unit (NPU).

[0049] The one or more processors control the processing of input data according to predefined operating rules or artificial intelligence (AI) models stored in non-volatile memory and volatile memory. The predefined operating rules or artificial intelligence models are provided by training or learning.

[0050] Here, providing by learning means obtaining a predefined operating rule or an AI model with desired characteristics by applying a learning algorithm to a plurality of learning data. The learning can be performed in the device or electronic device itself in which the AI ​​according to the embodiment is executed, and / or can be implemented by a separate server / system.

[0051] The AI ​​model may include multiple neural network layers. Each layer has multiple weight values, and each layer performs neural network calculations by calculating between the input data of the layer (such as the calculation results of the previous layer and / or the input data of the AI ​​model) and the multiple weight values ​​of the current layer. Examples of neural networks include, but are not limited to, convolutional neural networks (CNNs), deep neural networks (DNNs), recurrent neural networks (RNNs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), generative adversarial networks (GANs), and deep Q networks.

[0052] A learning algorithm is a method of using a plurality of learning data to train a predetermined target device (e.g., a robot) to enable, allow, or control the target device to make a determination or prediction. Examples of the learning algorithm include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0053] In the present application, an artificial intelligence model can be obtained by training. Here, "obtained by training" means obtaining a predefined operating rule or artificial intelligence model configured to perform a desired feature (or purpose) by training a basic artificial intelligence model with multiple training data through a training algorithm. The artificial intelligence model may include multiple neural network layers. Each of the multiple neural network layers includes multiple weight values, and the neural network calculation is performed by calculating between the calculation result of the previous layer and the multiple weight values.

[0054] The following describes several optional embodiments to illustrate the technical solutions of the embodiments of the present disclosure and the technical effects produced by the technical solutions of the present disclosure. It should be noted that the following embodiments can refer to, draw on or combine with each other, and the same terms, similar features and similar implementation steps in different embodiments will not be described repeatedly.

[0055] Figure 1 is a flowchart illustrating a semantic segmentation method performed by a semantic segmentation model according to an embodiment of the present disclosure.

[0056] Reference Figure 1 In step S101, the voxels obtained by conversion based on the input point cloud data are processed to obtain initial features. Here, the voxels can be initially extracted, and the initial dimensions can be mapped to a high-dimensional space to obtain initial features. For example, the initial features can be extracted from the voxels by a Stem layer including 2 convolutional layers, 2 normalization layers, and 1 activation function layer. The Stem layer is well known to those skilled in the art and will not be described in detail here.

[0057] Optionally, in order to obtain voxels, the point cloud data can be converted into multiple voxels by voxelizing the point cloud data. Here, the point cloud data can be obtained by a 3D scanning device (such as but not limited to a laser radar, a stereo camera, a time-of-flight camera, etc.). The point cloud data is a set of vectors in a three-dimensional coordinate system, which are usually expressed in the form of X, Y, and Z three-dimensional coordinates to represent the outer surface shape of an object. When converting the point cloud data into multiple voxels, the continuous space containing the point cloud data can be divided into multiple regular cuboids, each cuboid being a voxel. Each point is assigned to a corresponding cuboid according to the three-dimensional coordinates of each point in the point cloud data. In any cuboid, a label of a point is randomly selected as the label of the voxel, and the labels of other points in the cuboid are deleted, that is, the labels of the points located in the same voxel are the same. Finally, the voxels are input into a pre-trained semantic segmentation model to obtain a semantic segmentation result. Here, any method other than the above method can be used to convert points and data into multiple voxels, and the present disclosure does not impose any restrictions on this.

[0058] In step S102, the initial feature is downsampled a preset number of times to obtain a downsampled feature. Here, the preset number of times may be greater than or equal to 2.

[0059] According to an embodiment of the present disclosure, first, the initial feature may be downsampled to obtain a first downsampled result. Then, the i-th downsampled result may be downsampled to obtain an i+1-th downsampled result, where 1≤i≤N-1, N is a preset number and N is a natural number greater than or equal to 2. Finally, the N-th downsampled result may be used as the downsampled feature.

[0060] As an example, the downsampling module may be used to downsample the initial features. Specifically, the downsampling module may include a plurality of downsampling blocks connected in series. Each downsampling block may perform downsampling processing of a preset scale. The following is described by taking the need to perform 4 downsampling processes as an example, but the present disclosure is not limited thereto, and the preset number of times may be more or less than 4 times.

[0061] As described above, the initial feature may first be downsampled at a preset scale to obtain a first downsampled result, then the first downsampled result may be downsampled at a preset scale to obtain a second downsampled result, then the second downsampled result may be downsampled at a preset scale to obtain a third downsampled result, and finally the third downsampled result may be downsampled at a preset scale to obtain a fourth downsampled result, and the fourth downsampled result may be used as the final downsampled feature. Here, the preset scale may be a 1 / 2 scale. In this case, the scale of the first downsampled result is 1 / 2 of the original spatial scale, the scale of the second downsampled result is 1 / 4 of the original spatial scale, the scale of the third downsampled result is 1 / 8 of the original spatial scale, and the scale of the fourth downsampled result is 1 / 16 of the original spatial scale. In other words, the spatial scale of the latter downsampled process is 1 / 2 of the spatial scale of the previous downsampled process. The channel dimensions corresponding to the four downsampled results may be 64, 128, 256, and 256. According to an embodiment of the present disclosure, N may be adjusted according to actual needs and network structure to achieve different times of downsampled processing.

[0062] On the other hand, according to an embodiment of the present disclosure, the downsampling process can be performed by a first plurality of dynamic spatial aggregation convolution (DSAC) layers, a first Transformer module and a downsampling layer. Specifically, local features can be extracted from the initial features by the first plurality of DSAC layers, global features can be extracted from the local features by the first Transformer module, and the global features can be downsampled by the downsampling layer to obtain the first downsampling result. Then, the i-th local feature can be extracted from the i-th downsampling result by the first plurality of DSAC layers, the i-th global feature can be extracted from the i-th local feature by the first Transformer module, and the i-th global feature can be downsampled by the downsampling layer to obtain the i+1-th downsampling result. As described above, 1≤i≤N-1, N is a preset number of times and N is a natural number greater than or equal to 2. Finally, the N-th downsampling result can be used as a downsampling feature.

[0063] As an example, the first plurality of DSAC layers may include two DSAC layers. Thus, for the initial features, the first DSAC layer may be used to extract local features of the initial features, and the local features of the initial features may be input to the second DSAC layer to extract local features again. Next, the Transformer module is used to extract global features of the local features output from the second DSAC layer. Finally, the global features are downsampled using the downsampling layer to obtain the first downsampling result.

[0064] Figure 2A is a flow chart illustrating operations performed by a DSAC layer according to an embodiment of the present disclosure. Figure 2B 1 is a diagram showing an example of operations performed by the DSAC layer according to an embodiment of the present disclosure. Here, each DSAC layer may perform the following operations: Figure 2A and Figure 2B The operation shown.

[0065] Reference Figure 2A and Figure 2B In step S201, a convolution kernel of a first preset size (in Figure 2B In step S202, a convolution kernel 1 is used to convolve the input initial feature or the i-th downsampling result to obtain the first feature. Next, in step S202, a convolution kernel 1 is used to convolve the input initial feature or the i-th downsampling result to obtain the first feature. Figure 2B The first feature is convolved by a plurality of second features (shown as "convolution kernel 2-1", "convolution kernel 2-2", and "convolution kernel 2-3"). Here, the sizes of convolution kernel 2-1, convolution kernel 2-2, and convolution kernel 2-3 are different from each other, and they are different from the size of convolution kernel 1. Further, the length and width of convolution kernel 2-1 are greater than the length and width of convolution kernel 1, the length and depth of convolution kernel 2-2 are greater than the length and depth of convolution kernel 1, and the width and depth of convolution kernel 2-3 are greater than the width and depth of convolution kernel 1. However, the present disclosure is not limited to this. For example, the length and width of convolution kernel 2-2 are greater than the length and width of convolution kernel 1, the length and depth of convolution kernel 2-3 are greater than the length and depth of convolution kernel 1, and the width and depth of convolution kernel 2-1 are greater than the width and depth of convolution kernel 1. For another example, the length and width of convolution kernel 2-3 are greater than the length and width of convolution kernel 1, the length and depth of convolution kernel 2-1 are greater than the length and depth of convolution kernel 1, and the width and depth of convolution kernel 2-2 are greater than the width and depth of convolution kernel 1. Subsequently, in step S203, the output of the DSAC layer is determined according to the first feature and the sum of the products of each second feature and the corresponding weight. Here, the weights corresponding to the multiple second features can be calculated using a preset function.

[0066] As an example, the first feature may be extracted by the following equation (1).

[0067] X=xinp +conv 3*3*3 (x inp ) (1)

[0068] Among them, X is the first feature, x inp is the initial feature or the i-th downsampling result, conv 3*3*3 A convolution kernel of a first preset size, wherein the length, width, and depth of the convolution kernel of the first preset size are, for example, 3. However, the present disclosure is not limited thereto, and the length, width, and depth of the convolution kernel of the first preset size can be set to other values ​​as needed. Optionally, after performing the 3×3×3 convolution operation, a LayerNorm (LN) normalization operation and a GeLU activation function need to be applied to output the first feature from the result of the convolution operation. The LN normalization operation and the GeLU activation function are well known to those skilled in the art and will not be described in detail here.

[0069] Next, a plurality of second features may be extracted by the following equation (2).

[0070] X i =conv i (X),i=1,2,3 (2)

[0071] Among them, X i is the second feature of the i-th feature, conv i , i=1,2,3 represent convolution kernels of sizes 5×5×1, 5×1×5, and 1×5×5, respectively. According to equation (2), the length, width, and depth of the convolution kernel of the second preset size are all, for example, 5. However, the present disclosure is not limited thereto, and the length, width, and depth of the convolution kernel of the second preset size can be set to other values ​​as needed. A plurality of convolution kernels of different sizes are obtained by decomposing the convolution kernel of the second preset size in the length, width, and depth directions, respectively. i , that is, a 5×5×1 convolution kernel, a 5×1×5 convolution kernel, and a 1×5×5 convolution kernel. In this way, multiple second features can be obtained by convolving the first feature with multiple convolution kernels of different sizes. As described above, after performing 5×5×1, 5×1×5, and 1×5×5 convolution operations, it is also necessary to apply the LN normalization operation and the GeLU activation function to output the second feature from the result of the convolution operation.

[0072] Next, a plurality of weights corresponding to the plurality of second features may be calculated by the following equation (3).

[0073] W=softmax(Linear(X))∈R n×3 (3)

[0074] Among them, W is the weight corresponding to the i-th second feature, R is a rational number, n represents non-empty point cloud data, softmax is the activation function, and Linear is the linear function. Figure 2B The “mapping” in represents the operation corresponding to equation (3).

[0075] Finally, the output of the DSAC layer is determined according to the sum of the first feature and the product of each second feature and the corresponding weight. The output of the DSAC layer can be calculated by the following equation (4):

[0076]

[0077] Among them, X out is the output of the DSAC layer. Figure 2B The “∑” in denoted the operation corresponding to equation (4).

[0078] By decomposing the large convolution kernel into three convolution kernels according to length, width and depth as described above, the processing capability of the semantic segmentation model for sparse and large point cloud space can be improved, and the number of parameters and calculation amount can be reduced during the calculation process, thereby improving the processing capability.

[0079] Return to reference Figure 1 In step S103, a global feature is extracted from the downsampled feature, and the global feature and the bird's-eye view feature are fused to obtain a fused feature. According to an embodiment of the present disclosure, the bird's-eye view feature can be converted from the global feature.

[0080] According to an embodiment of the present disclosure, local features can be extracted from downsampled features through multiple DSAC layers, and global features can be extracted from local features through a Transformer module. Here, the number of DSAC layers can be 2, but the present disclosure is not limited thereto. The first DSAC layer can be used to extract local features of the downsampled features, the second DSAC layer can be used to extract local features of the local features output by the first DSAC layer, and then the Transformer module can be used to extract global features from the local features output by the second DSAC layer.

[0081] Figure 3A is a flowchart showing an operation of fusing global features with bird's-eye view features according to an embodiment of the present disclosure, Figure 3B is a diagram illustrating an example of an operation of fusing a global feature with a bird's-eye view feature according to an embodiment of the present disclosure.

[0082] Reference Figure 3A and Figure 3BIn step S301, the global features can be compressed into two-dimensional BEV features by performing densification processing on the global features and taking the maximum value in the depth direction. Then, in step S302, the two-dimensional BEV features can be processed by two-dimensional deformable convolution to obtain updated BEV features. Finally, in step S303, the global features can be fused with the updated BEV features to obtain fused features.

[0083] As an example, the global features can be compressed into two-dimensional BEV features by the following equation (5).

[0084] F bev =Max(Dense(F voxel )) (5)

[0085] Among them, F bev is the BEV feature, Max is the process of taking the maximum value in the depth direction, Dense is the densification process, and F voxel It is a global feature. Figure 3B The “compression” in represents the operation corresponding to equation (5).

[0086] Then, the updated BEV characteristics can be obtained by the following equation (6).

[0087] F′ bev =O(conv(F bev )) (6)

[0088] Among them, F' bev is the updated BEV feature, conv represents a 3×3 convolution operation, and O represents a two-dimensional operation operator, that is, a two-dimensional deformable convolution DCN v3 operator. Here, the 3×3 convolution operation is only an example, and convolutions and operations of other dimensions may also be used. Figure 3B The “convolution” and “two-dimensional operation operator” in represent the operations corresponding to equation (6).

[0089] Finally, the fusion feature can be obtained by the following equation (7).

[0090] F′ voxel =Fusion(F voxel ,F′ bev ) (7)

[0091] Among them, F' voxel It is a fusion feature, and fusion is a fusion operation, for example, a multi-mode dynamic fusion operation. Figure 3B The “fusion” in represents the operation corresponding to equation (7).

[0092] According to an embodiment of the present disclosure, the global feature and the updated BEV feature can be connected through the first fully connected layer to obtain a fusion feature. Specifically, the above operation can be implemented by the following equation (8).

[0093]

[0094] Among them, MLP is the first fully connected layer, Here, the first fully connected layer may be a double-layer fully connected layer, but the present disclosure is not limited thereto.

[0095] Alternatively, the updated BEV features may be processed by a second fully connected layer different from the first fully connected layer, the processing result may be point-wise multiplied with the global feature, and the multiplication result may be added to the global feature to obtain a fused feature. Specifically, the above operation may be implemented by the following equation (9).

[0096] F′ voxel =F voxel +σ(FC(F′ bev ))⊙F voxel (9)

[0097] Where FC represents the fully connected layer, which is used to learn the representation of BEV features, σ represents the Sigmod activation function, and ⊙ represents point-by-point multiplication.

[0098] Alternatively, the fused feature can be obtained by applying a preset deformation function to the global feature, the sampling points corresponding to the global feature in the updated BEV feature, and the updated BEV feature. Specifically, the above operation can be implemented by the following equation (10).

[0099]

[0100] Where ref represents the number of sampling points, P(p+Δ j ) represents the sampling point of each global feature on the corresponding BEV feature map, Δ j Indicates the offset.

[0101] According to an embodiment of the present disclosure, the ability of the semantic segmentation model to extract global features can be enhanced by converting sparse voxel features into dense BEV features and then fusing the BEV features with the sparse voxel features using a two-dimensional operator.

[0102] Return to reference Figure 1 In step S104, by combining the results of the preset number of downsampling processes, the fusion features are upsampled a preset number of times to obtain upsampled features.

[0103] According to an embodiment of the present disclosure, first, the fusion feature can be upsampled through an upsampling layer, and the feature obtained by the upsampling process is fused with the Nth downsampling result to obtain a first fusion result, and the first fusion result is convolved through a second plurality of DSAC layers to obtain a first upsampling result. Then, the i-th upsampling result can be upsampled through an upsampling layer, and the feature obtained by the i-th upsampling process is fused with the Ni-th downsampling result to obtain the i+1-th fusion result, and the i+1-th fusion result is convolved through a second plurality of DSAC layers to obtain the i+1-th upsampling result. As described above, 1≤i≤N-1, N is the preset number of times and N is a natural number greater than or equal to 2. Finally, the Nth upsampling result can be used as an upsampling feature. Optionally, the fusion of the feature obtained by the upsampling process and the corresponding downsampling result (for example, the feature obtained by the i-th upsampling process and the Ni-th downsampling result) can be achieved by, for example, skip connection fusion.

[0104] As an example, the upsampling process can be implemented using an upsampling module, and the upsampling module may include a plurality of upsampling blocks connected in series. As an example, each upsampling block may perform an upsampling process of a preset scale, and each upsampling block includes an upsampling layer as described above and a second plurality of DSAC layers. The preset scale may be, for example, 1 / 2, and may be implemented by transposed convolution. The number of upsampling processes may be the same as the number of downsampling processes previously performed. For example, the preset number of times may be 4 times, but the present disclosure is not limited thereto. When the preset number of times is 4 times, the fused feature may first be subjected to an upsampling process of a preset scale, the result of the upsampling process may be fused with the fourth downsampling result, and the fused result may be convolved using a plurality of DSAC layers to obtain a first upsampling result. Then, the first upsampling result may be subjected to an upsampling process of a preset scale, the result of the upsampling process may be fused with the third downsampling result, and the fused result may be convolved using a plurality of DSAC layers to obtain a second upsampling result. The third upsampling result and the fourth upsampling result may be obtained in the same manner as the above-described acquisition method, and will not be described in detail herein.

[0105] According to the embodiments of the present disclosure, by performing jump connection fusion on the features obtained by upsampling processing and the corresponding downsampling results, the learning ability and generalization ability of the semantic segmentation model can be improved, and the recognition accuracy can be improved.

[0106] In step S105, semantic label prediction is performed based on the upsampled features, thereby obtaining a semantic segmentation result. Here, a prediction head can be used to perform semantic label prediction. The prediction head is well known to those skilled in the art and will not be described in detail here.

[0107] Figure 4is a diagram showing an example of a semantic segmentation model according to an embodiment of the present disclosure. The semantic segmentation model may perform the semantic segmentation method as described above. Figure 4 The semantic segmentation model 400 includes a data processing module 401, a downsampling module 402, a feature aggregation module 403, an upsampling module 404 and a prediction head 405.

[0108] The semantic segmentation model 400 may receive voxels converted from the point cloud data and may output a semantic segmentation result of the point cloud data. To this end, the point cloud data may be converted into voxels through a preset voxelization process and then the voxels may be input into the semantic segmentation model 400 .

[0109] The data processing module 401 may extract initial features from the input voxels and input the initial features to the down-sampling module 402 .

[0110] The downsampling module 402 may include multiple (for example, but not limited to 4) downsampling blocks, each of which is connected in series. Each downsampling block may include multiple (for example, but not limited to 2) DSAC layers, Transform layers, and downsampling layers. The first downsampling block performs a downsampling process on the initial feature with a scale of 1 / 2 to obtain a first downsampling result with a scale of 1 / 2 of the initial spatial scale. The second downsampling block performs a downsampling process on the first downsampling result with a scale of 1 / 2 to obtain a second downsampling result with a scale of 1 / 4 of the initial spatial scale. The third downsampling block performs a downsampling process on the second downsampling result with a scale of 1 / 2 to obtain a third downsampling result with a scale of 1 / 8 of the initial spatial scale. The fourth downsampling block performs a downsampling process on the third downsampling result with a scale of 1 / 2 to obtain a fourth downsampling result with a scale of 1 / 16 of the initial spatial scale. For each downsampling block, the input initial feature or the local features of the previous downsampling result can be extracted by the first DSAC layer, and the local features can be extracted again by inputting the extracted local features into the second DSAC layer. Then, the global features can be extracted from the local features output by the second DSAC layer through the Transformer module. Finally, the global features can be downsampled through the downsampling layer to obtain the corresponding downsampled features. The downsampling results of the fourth downsampling block can be input into the feature aggregation module as downsampled features.

[0111] The feature aggregation module 403 includes multiple (for example but not limited to 2) DSAC layers, Transform layers and fusion layers connected in series. First, local features can be extracted from the downsampled features through multiple DSAC layers, and then global features can be extracted from the local features output by the Transformer module. Finally, the global features and BEV features can be fused through the fusion layer to obtain fused features, and the fused features can be input to the upsampling module. Here, the BEV features can be converted from the global features.

[0112] The upsampling module 404 may include multiple (for example, but not limited to 4) upsampling blocks, each of which is connected in series. Each upsampling block may include an upsampling layer and multiple (for example, but not limited to 2) DSAC layers. For the first upsampling block, the fused feature may be upsampled by the upsampling layer, and the result of the upsampling process may be fused with the fourth downsampling result, and then the local features of the fused result may be extracted by the first DSAC layer, and the local features may be extracted again by inputting the extracted local features into the second DSAC layer. The local features output by the second DSAC layer may be used as the first upsampling result. Then, the first upsampling result may be similarly processed by the second upsampling block to obtain a second upsampling result, the second upsampling result may be similarly processed by the third upsampling block to obtain a third upsampling result, and the third upsampling result may be similarly processed by the fourth upsampling block to obtain a fourth upsampling result as the upsampling feature.

[0113] The prediction head 405 may perform semantic label prediction based on the upsampled features, thereby obtaining a semantic segmentation result.

[0114] Refer to the following Figure 5 A method for training a semantic segmentation model according to an embodiment of the present disclosure is described.

[0115] Figure 5 is a flowchart illustrating a method for training a semantic segmentation model according to an embodiment of the present disclosure.

[0116] Reference Figure 5 In step S501, sample mixing enhancement is performed on the point cloud data as the initial sample, and point cloud structure enhancement is performed on the point cloud data after sample mixing enhancement to obtain enhanced point cloud data. Here, by performing sample mixing enhancement and point cloud structure enhancement, the diversity of the point cloud data as the initial sample can be increased to improve the training accuracy.

[0117] Specifically, sample mixing enhancement can be achieved by performing the following processing on every two frames of point cloud data. First, the two frames of point cloud data and the labels of each point in the two frames of point cloud data can be read; then, the polar coordinate azimuth of each point in each frame of point cloud data can be calculated; then, the points in the first frame of point cloud data within the preset polar coordinate azimuth range can be exchanged with the points in the second frame of point cloud data; then, all points in the exchanged second frame of point cloud data with preset labels can be copied to the first frame of point cloud data; finally, the labels of each point in the first frame of point cloud data can be updated to obtain point cloud data after sample mixing enhancement.

[0118] Here, the preset polar coordinate azimuth range can be (α, β). When the point cloud data comes from the Semantic-KITTI dataset, α is The random value in , β=α+π. When the point cloud data comes from the nuScenes dataset,

[0119] On the other hand, sample mixing enhancement can also be achieved by performing the following processing on every two frames of point cloud data. First, the two frames of point cloud data and the labels of each point in the two frames of point cloud data can be read; then, the elevation angle of each point in each frame of point cloud data can be calculated; then, the preset elevation angle interval can be evenly divided into a preset number of sub-elevation angle intervals, and the sub-elevation angle intervals can be divided into odd elevation angle intervals and even elevation angle intervals according to the parity of the sequence numbers of the sub-elevation angle intervals; then, the points in the second frame of point cloud data whose elevation angles belong to the even elevation angle interval can be exchanged with the points in the first frame of point cloud data whose elevation angles belong to the even elevation angle interval, or, the points in the second frame of point cloud data whose elevation angles belong to the odd elevation angle interval can be exchanged with the points in the first frame of point cloud data whose elevation angles belong to the odd elevation angle interval; finally, the labels of each point in the first frame of point cloud data can be updated to obtain the point cloud data after sample mixing enhancement.

[0120] Here, the elevation angle can be determined by the following equation (11).

[0121]

[0122] Among them, x, y, z refer to the three-dimensional coordinates of the midpoint of the point cloud data. The preset elevation angle range can be expressed as The preset number can be any value among 3, 4, 5, and 6. When the point cloud data comes from the Semantic-KITTI dataset, When the point cloud data comes from the nuScenes dataset,

[0123] Optionally, point cloud structure enhancement can be achieved through the following processing. First, the point cloud data after sample mixing enhancement can be rotated around the first coordinate axis by a preset angle. Then, the rotated point cloud data can be flipped around any coordinate axis. Next, the flipped point cloud data can be scaled. Finally, noise data can be added to the scaled point cloud data to obtain point cloud data with point cloud structure enhancement. As a non-limiting example, the first coordinate axis can be the Z axis in the three-dimensional coordinate axis, and the preset angle can be any angle value in (-π,π). The scaling scale can be obtained by random uniform sampling of [0.95,1.05]. As another non-limiting example, when adding noise data, it is necessary to use noise data with a normal distribution with a mean of 0 and a standard deviation of 0.1.

[0124] Return to reference Figure 5 In step S502, the enhanced point cloud data is voxelized to convert the point cloud data into a plurality of voxel sample data for training. Here, the point cloud data may be converted into cuboid voxel data, but the present disclosure is not limited thereto.

[0125] In step S503, the semantic segmentation model is trained based on the voxel sample data and the loss function constructed using the cross entropy function and the Lovasz-Softmax function. Here, the various parameters of the semantic segmentation model are updated by minimizing the value of the loss function constructed using the cross entropy function and the Lovasz-Softmax function to achieve the training of the semantic segmentation model.

[0126] Figure 6 is a block diagram showing a semantic segmentation apparatus for point cloud data according to an embodiment of the present disclosure. According to an embodiment of the present disclosure, the semantic segmentation apparatus may be implemented by the semantic segmentation model as described above.

[0127] refer to Figure 6 , the semantic segmentation device 600 includes a data processing module 601, a downsampling module 602, a feature fusion module 603, an upsampling module 604 and a prediction module 605. The data processing module 601 can process the voxels obtained based on the input point cloud data conversion to obtain the initial features. The downsampling module 602 can obtain the downsampled features by performing a downsampling process on the initial features for a preset number of times. The feature fusion module 603 can extract global features from the downsampled features, and fuse the global features and the BEV features to obtain the fused features. As described above, the BEV features can be obtained from the global feature conversion. The upsampling module 604 can perform an upsampling process on the fused features for a preset number of times by combining the results of the downsampling process for a preset number of times to obtain the upsampled features. The prediction module 605 can perform semantic label prediction based on the upsampled features to obtain the semantic segmentation results.

[0128] According to an embodiment of the present disclosure, the semantic segmentation apparatus 600 may further include a data conversion module (not shown). The data conversion module may convert the point cloud data into a plurality of voxels by performing voxelization processing on the point cloud data.

[0129] Specifically, the downsampling module 602 may extract local features from the initial features through the first plurality of DSAC layers, extract global features from the local features through the first Transformer module, and downsample the global features through the downsampling layer to obtain a first downsampling result. Subsequently, the downsampling module 602 may extract the i-th local feature from the i-th downsampling result through the first plurality of DSAC layers, extract the i-th global feature from the i-th local feature through the first Transformer module, and downsample the i-th global feature through the downsampling layer to obtain the i+1-th downsampling result. As described above, 1≤i≤N-1, N is a preset number of times and N is a natural number greater than or equal to 2. Finally, the downsampling module 602 may use the N-th downsampling result as a downsampling feature.

[0130] According to an embodiment of the present disclosure, each DSAC layer may perform the following operations: convolve an initial feature of an input or an i-th downsampling result through a convolution kernel of a first preset size to obtain a first feature; convolve the first feature through a plurality of convolution kernels of different sizes to obtain a plurality of second features, wherein the plurality of convolution kernels of different sizes are obtained by decomposing the convolution kernel of a second preset size in the length, width and depth directions; determine the output of the corresponding DSAC layer according to the sum of the products of the first feature and each second feature and the corresponding weight, wherein the weights corresponding to the plurality of second features are calculated using a preset function.

[0131] The feature fusion module 603 can extract local features from the downsampled features through multiple DSAC layers, and extract global features from the local features through the Transformer module. Afterwards, the feature fusion module 603 can compress the global features into two-dimensional bird's-eye view features by performing densification processing on the global features and taking the maximum value in the depth direction; obtain updated bird's-eye view features by performing two-dimensional deformable convolution processing on the two-dimensional bird's-eye view features; and fuse the global features with the updated bird's-eye view features to obtain fused features.

[0132] More specifically, the feature fusion module 603 may connect the global features with the updated bird's-eye view features through the first fully connected layer to obtain fused features; or the feature fusion module 603 may process the updated bird's-eye view features through the second fully connected layer, multiply the processing results with the global features point by point, and add the multiplication results to the global features to obtain fused features; or the feature fusion module 603 may obtain fused features by applying a preset deformation function to the global features, the sampling points in the updated bird's-eye view features corresponding to the global features, and the updated bird's-eye view features.

[0133] The upsampling module 604 may perform upsampling processing on the fused features through the upsampling layer, fuse the features obtained by the upsampling processing with the Nth downsampling result to obtain a first fusion result, and perform convolution processing on the first fusion result through the second plurality of DSAC layers to obtain a first upsampling result. Subsequently, the upsampling module 604 may perform upsampling processing on the i-th upsampling result through the upsampling layer, fuse the features obtained by the i-th upsampling processing with the Ni-th downsampling result to obtain an i+1-th fusion result, and perform convolution processing on the i+1-th fusion result through the second plurality of DSAC layers to obtain an i+1-th upsampling result. Finally, the upsampling module 604 may use the Nth upsampling result as the upsampling feature.

[0134] According to an embodiment of the present disclosure, the semantic segmentation model can be trained by the following operations. First, the cloud data as the initial sample can be sample mixed and enhanced, and the point cloud data after the sample mixed enhancement can be point cloud structure enhanced to obtain enhanced point cloud data. Then, the enhanced point cloud data can be voxelized to convert the point cloud data into a plurality of voxel sample data for training. Finally, the semantic segmentation model can be trained based on the voxel sample data and the loss function constructed using the cross entropy function and the Lovasz-Softmax function.

[0135] Optionally, sample mixing enhancement can be achieved by performing the following operations for every two frames of point cloud data: reading the two frames of point cloud data and the label of each point in the two frames of point cloud data; calculating the polar coordinate azimuth of each point in each frame of point cloud data; exchanging the points in the first frame of point cloud data within a preset polar coordinate azimuth range with the points in the second frame of point cloud data; copying all points labeled with preset labels in the exchanged second frame of point cloud data to the first frame of point cloud data; updating the labels of each point in the first frame of point cloud data to obtain point cloud data after sample mixing enhancement.

[0136] On the other hand, sample mixing enhancement can be achieved by performing the following operations for every two frames of point cloud data: reading the two frames of point cloud data and the label of each point in the two frames of point cloud data; calculating the elevation angle of each point in each frame of point cloud data; evenly dividing the preset elevation angle interval into a preset number of sub-elevation angle intervals, and dividing the sub-elevation angle intervals into odd elevation angle intervals and even elevation angle intervals according to the parity of the serial numbers of the sub-elevation angle intervals; exchanging the points in the second frame of point cloud data whose elevation angles belong to the even elevation angle interval with the points in the first frame of point cloud data whose elevation angles belong to the even elevation angle interval, or exchanging the points in the second frame of point cloud data whose elevation angles belong to the odd elevation angle interval with the points in the first frame of point cloud data whose elevation angles belong to the odd elevation angle interval; updating the labels of each point in the first frame of point cloud data to obtain point cloud data after sample mixing enhancement.

[0137] Optionally, point cloud structure enhancement can be achieved through the following processing: rotating the point cloud data after sample mixing enhancement by a preset angle around the first coordinate axis; flipping the rotated point cloud data around an arbitrary coordinate axis; scaling the flipped point cloud data; adding noise data to the scaled point cloud data to obtain point cloud data with point cloud structure enhancement.

[0138] Figure 7 is a block diagram illustrating an electronic device according to an example embodiment of the present disclosure.

[0139] Reference Figure 7 , the electronic device 700 may include at least one processor 701 and at least one memory 702. The at least one memory 702 may store computer executable instructions, and when the computer executable instructions are executed by the at least one processor 701, the at least one processor 701 is prompted to perform the semantic segmentation method of point cloud data according to an embodiment of the present disclosure.

[0140] As an example, the electronic device 700 may be a PC, a tablet device, a personal digital assistant, a smart phone, or other device capable of executing the above-mentioned instruction set. For example, the electronic device 700 may be connected to a wireless LAN. Here, the electronic device 700 is not necessarily a single electronic device, but may also be a collection of devices or circuits capable of executing the above-mentioned instructions (or instruction sets) individually or in combination. The electronic device may also be part of an integrated control system or a system manager, or may be configured as a portable electronic device interconnected with a local or remote (e.g., via wireless transmission) interface.

[0141] In the electronic device 700, the processor 701 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller or a microprocessor. As an example, the processor may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0142] The processor 701 can execute instructions or codes stored in a memory 702, wherein the memory 702 can also store data. Instructions and data can also be sent and received over a network via a network interface device, wherein the network interface device can use any known transmission protocol.

[0143] The memory 702 may be integrated with the processor 701. For example, the memory 702 may be a RAM or flash memory disposed in an integrated circuit microprocessor. In addition, the memory 702 may include a separate device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The memory 702 and the processor 701 may be operatively coupled or may communicate with each other through, for example, an I / O port or a network connection, so that the processor 701 can read files stored in the memory 702.

[0144] In addition, the electronic device 700 may also include a video display (eg, a liquid crystal display) and a user interaction interface (eg, a keyboard, a mouse, or a touch input device). All components of the electronic device may be connected to each other via a bus and / or a network.

[0145] According to some example embodiments of the present disclosure, a computer-readable storage medium storing computer-executable instructions may also be provided, wherein when the instructions are executed by at least one processor, the at least one processor is prompted to perform the semantic segmentation method of point cloud data according to some example embodiments of the present disclosure. Examples of computer-readable storage media here include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-transitory manner and provide the computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The instructions or computer programs in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0146] According to the semantic segmentation method, semantic segmentation device and electronic device of point cloud data disclosed in the present invention, it is possible to improve the perception ability of the semantic segmentation model for sparse and large point cloud space, realize the fusion of global information and local information, use 2D operators to enhance the ability to extract global information, and improve the accuracy of semantic segmentation.

[0147] Although the present disclosure includes specific examples, it will be understood by those skilled in the art that various changes in form and detail may be made without departing from the spirit and scope of the claims and their equivalents. The examples disclosed herein are to be considered in a descriptive sense and not for purposes of limitation. The description of features or aspects in each example will be considered applicable to similar features or aspects in other examples. Suitable results may be obtained if the described techniques are performed in a different order, and / or if the described systems, structures, devices, or circuits are combined in a different manner and / or replaced or supplemented by other components or their equivalents. Therefore, the scope of the disclosure is not limited by the detailed description, but by the claims and their equivalents, and all changes within the scope of the claims and their equivalents will be considered to be included in the present disclosure.

Claims

1. A semantic segmentation method for point cloud data, characterized in that: The semantic segmentation method comprises: Process the voxels obtained by converting the input point cloud data to obtain initial features; Obtaining downsampled features by downsampling the initial features a preset number of times; Extracting a global feature from the downsampled feature, and fusing the global feature with a bird's-eye view feature to obtain a fused feature, wherein the bird's-eye view feature is converted from the global feature; By combining the results of the preset number of downsampling processes, upsampling the fused features for the preset number of times is performed to obtain upsampling features; Semantic label prediction is performed based on the upsampled features to obtain a semantic segmentation result.

2. The semantic segmentation method according to claim 1, characterized in that: The semantic segmentation method further includes: converting the point cloud data into a plurality of voxels by performing voxelization processing on the point cloud data.

3. The semantic segmentation method according to claim 1, characterized in that: The downsampling process is performed on the initial feature for a preset number of times to obtain the downsampled feature, including: Extracting local features from the initial features through a first plurality of dynamic spatial aggregation convolutional layers, extracting global features from the local features through a first Transformer module, and downsampling the global features through a downsampling layer to obtain a first downsampling result; Extracting an i-th local feature from the i-th downsampling result through the first plurality of dynamic spatial aggregation convolutional layers, extracting an i-th global feature from the i-th local feature through the first Transformer module, and downsampling the i-th global feature through the downsampling layer to obtain an i+1-th downsampling result, wherein 1≤i≤N-1, N is the preset number of times and N is a natural number greater than or equal to 2; The Nth down-sampling result is used as the down-sampling feature.

4. The semantic segmentation method according to claim 3, characterized in that: Each dynamic spatial aggregation convolutional layer performs the following operations: Performing convolution processing on the input initial feature or the i-th downsampling result through a convolution kernel of a first preset size to obtain a first feature; The first feature is convolved by a plurality of convolution kernels of different sizes to obtain a plurality of second features, wherein the plurality of convolution kernels of different sizes are obtained by decomposing a convolution kernel of a second preset size in length, width, and depth directions; The output of the corresponding dynamic spatial aggregation convolution layer is determined according to the first feature and the sum of the products of each of the second features and the corresponding weights, wherein the weights corresponding to the multiple second features are calculated using a preset function.

5. The semantic segmentation method according to claim 1, characterized in that: The extracting global features from the down-sampled features comprises: Local features are extracted from the downsampled features through a plurality of dynamic spatial aggregation convolutional layers, and global features are extracted from the local features through a Transformer module.

6. The semantic segmentation method according to claim 1 or 5, characterized in that: The fusing of the global features and the bird's-eye view features to obtain fused features includes: Compressing the global features into two-dimensional bird's-eye view features by performing densification processing on the global features and taking the maximum value in the depth direction; By performing two-dimensional deformable convolution processing on the two-dimensional bird's-eye view features, an updated bird's-eye view feature is obtained; The global feature is fused with the updated bird's-eye view feature to obtain the fused feature.

7. The semantic segmentation method according to claim 6, characterized in that: The step of fusing the global feature with the updated bird's-eye view feature to obtain the fused feature includes: Connecting the global feature with the updated bird's-eye view feature through a first fully connected layer to obtain the fused feature; Alternatively, the updated bird's-eye view feature is processed by a second fully connected layer, the processing result is point-by-point multiplied with the global feature, and the multiplication result is added to the global feature to obtain the fused feature; Alternatively, the fused feature is obtained by applying a preset deformation function to the global feature, the sampling points corresponding to the global feature in the updated bird's-eye view feature, and the updated bird's-eye view feature.

8. The semantic segmentation method according to claim 3, characterized in that: The step of performing upsampling processing on the fusion feature for a preset number of times in combination with the result of the downsampling processing for the preset number of times to obtain the upsampling feature comprises: Performing upsampling processing on the fused features through an upsampling layer, fusing the features obtained through the upsampling processing with the Nth downsampling result to obtain a first fusion result, and performing convolution processing on the first fusion result through a second plurality of dynamic spatial aggregation convolution layers to obtain a first upsampling result; Performing upsampling processing on the i-th upsampling result through the upsampling layer, fusing the feature obtained by the i-th upsampling processing with the Ni-th downsampling result to obtain an i+1-th fused result, and performing convolution processing on the i+1-th fused result through the second plurality of dynamic spatial aggregation convolution layers to obtain an i+1-th upsampling result; The Nth upsampling result is used as the upsampling feature.

9. The semantic segmentation method according to claim 1, characterized in that: The semantic segmentation method is performed by a semantic segmentation model. The semantic segmentation model is trained by the following steps: Performing sample mixing enhancement on the cloud data used as the initial sample, and performing point cloud structure enhancement on the point cloud data after the sample mixing enhancement, to obtain enhanced point cloud data; Performing voxelization processing on the enhanced point cloud data to convert the point cloud data into a plurality of voxel sample data for training; The semantic segmentation model is trained based on the voxel sample data and a loss function constructed using a cross entropy function and a Lovasz-Softmax function.

10. The semantic segmentation method according to claim 9, characterized in that: The step of performing sample mixing enhancement on the point cloud data as the initial sample includes: For every two frames of point cloud data, the following processing is performed: Read the two frames of point cloud data and the label of each point in the two frames of point cloud data; Calculate the polar coordinate azimuth of each point in each frame of point cloud data; Exchanging points in the first frame of point cloud data with points in the second frame of point cloud data within a preset polar coordinate azimuth range; Copying all points labeled with a preset label in the second frame of point cloud data after the exchange to the first frame of point cloud data; Update the labels of each point in the first frame of point cloud data to obtain the point cloud data after sample mixing and enhancement.

11. The semantic segmentation method according to claim 9, characterized in that: The step of performing sample mixing enhancement on the point cloud data as the initial sample includes: For every two frames of point cloud data, the following processing is performed: Read the two frames of point cloud data and the label of each point in the two frames of point cloud data; Calculate the elevation angle of each point in each frame of point cloud data; Evenly dividing a preset elevation angle interval into a preset number of sub-elevation angle intervals, and dividing the sub-elevation angle intervals into odd elevation angle intervals and even elevation angle intervals according to the parity of the sequence numbers of the sub-elevation angle intervals; Swap the points whose elevation angles belong to the even elevation angle interval in the second frame of point cloud data with the points whose elevation angles belong to the even elevation angle interval in the first frame of point cloud data, or swap the points whose elevation angles belong to the odd elevation angle interval in the second frame of point cloud data with the points whose elevation angles belong to the odd elevation angle interval in the first frame of point cloud data; Update the labels of each point in the first frame of point cloud data to obtain the point cloud data after sample mixing and enhancement.

12. The semantic segmentation method according to claim 9, characterized in that: The step of performing point cloud structure enhancement on the point cloud data after sample mixing enhancement includes: Rotate the sample mixed enhanced point cloud data around the first coordinate axis by a preset angle; Flip the rotated point cloud data around any coordinate axis; Scale the flipped point cloud data; Noise data is added to the scaled point cloud data to obtain point cloud data with enhanced point cloud structure.

13. A semantic segmentation device for point cloud data, characterized in that: The semantic segmentation device comprises: A data processing module is configured to process voxels obtained by conversion based on input point cloud data to obtain initial features; A downsampling module is configured to obtain a downsampled feature by performing a downsampling process on the initial feature for a preset number of times; a feature fusion module configured to extract a global feature from the downsampled feature, and fuse the global feature with a bird's-eye view feature to obtain a fused feature, wherein the bird's-eye view feature is converted from the global feature; An upsampling module is configured to perform upsampling processing on the fusion feature for the preset number of times by combining the results of the downsampling processing for the preset number of times to obtain an upsampling feature; The prediction module is configured to perform semantic label prediction based on the upsampled features to obtain a semantic segmentation result.

14. An electronic device, characterized in that: include: at least one processor; at least one memory storing computer executable instructions, Wherein, when the computer executable instructions are executed by the at least one processor, the at least one processor is prompted to execute the semantic segmentation method of point cloud data as claimed in any one of claims 1 to 12.

15. A computer-readable storage medium storing instructions, characterized in that: When the instructions are executed by a processor, the semantic segmentation method of point cloud data as described in any one of claims 1 to 12 is implemented.

Images