3D point cloud semantic segmentation method based on point cloud information coding enhancement

Abstract

The invention discloses a 3D (three-dimensional) point cloud semantic segmentation method based on point cloud information coding enhancement, which is characterized in that various codes of point cloud original information are introduced and fused in each stage of a neural network, original information fuzziness and loss caused by network deepening are reduced, and high-precision 3D point cloud semantic segmentation is realized. Firstly, deviation standardization features of point cloud information are introduced through point cloud information coding, various feature codes are effectively fused into position coding, original information of point cloud loss is supplemented in a self-adaptive mode, and therefore the environment sensing capacity of a network is enhanced; and then, aiming at the problem of coding interference possibly occurring when various kinds of information coexist through a coding information distribution strategy, the coding information distribution strategy distributes the various kinds of point information in each stage of the network hierarchically, so that the interference of the various kinds of information is greatly reduced, and more accurate point cloud semantic segmentation is realized. The modular design can be easily inserted into an existing point-based point cloud segmentation model, and the model performance is remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of point cloud data processing, and in particular relates to a 3D point cloud semantic segmentation method based on point cloud information coding enhancement. Background Art

[0002] Understanding complex 3D real-world environments is crucial for the perception of service robots, the use of VR / AR devices, autonomous driving and other technologies. In this context, semantic segmentation technology of point clouds is the focus of 3D environment perception. By classifying 3D point clouds point by point, semantic segmentation algorithms can identify objects in a 3D environment and achieve intelligent perception of the 3D real world.

[0003] In order to achieve high-precision 3D point cloud semantic segmentation, existing algorithms often have deeper neural network layers. However, in the process of deepening the network, the original information of the point cloud will inevitably become blurred or even lost. Therefore, as a representation of the spatial geometric structure of the point cloud, existing algorithms often introduce the encoding of point cloud coordinate information in the modules of each local point set feature aggregation to supplement the lost geometric features. However, the original information contained in the point cloud not only contains basic coordinate information, but also typical additional information such as RGB color and normal direction information. However, the blurring and loss of these original information in deep neural networks have not received enough attention. Information such as color plays an important role in the semantic segmentation task of complex indoor environments. For example, paintings on the wall require detailed color representation to distinguish the picture from the wall. At the same time, after introducing more information encoding, the interference problem of multiple encodings will also arise. Therefore, it is necessary to propose an efficient point cloud original information encoding method to improve the performance of existing point cloud semantic segmentation algorithms in 3D environments with complex colors. Summary of the invention

[0004] In order to solve the shortcomings of the prior art and achieve the purpose of high-precision 3D scene perception, the present invention adopts the following technical solutions:

[0005] A 3D point cloud semantic segmentation method based on point cloud information coding enhancement includes the following steps:

[0006] Step 1: Encoding the point cloud location information; constructing a neighborhood subset of the point cloud, and encoding the location information of the point cloud and other points in the neighborhood subset to obtain the location information encoding;

[0007] Step 2: Encode the original feature information of the point cloud, and combine the point cloud position information encoding and the point cloud original feature information encoding;

[0008] Step 3: Code fusion: Combining the combined code, point cloud position information code, and point cloud original feature information code to perform feature enhanced code fusion;

[0009] Step 4: Point cloud coding allocation strategy; based on the different local feature aggregation modules at each stage of the point cloud semantic segmentation neural network, the point cloud information coding is alternately enhanced through the coding fusion, so that different local feature aggregation modules can obtain different supplements of the original feature information of the point cloud, alleviate the redundancy of information coding and make each feature aggregation module focus on each other, improve the effect of point cloud information coding, and finally complete the semantic segmentation of the point cloud.

[0010] Furthermore, in step 1, the scene Any point x containing point cloud features i =(f i ,p i ,c i ,...), Represents the semantic features of the point cloud, Represents the location characteristics of the point cloud, represents the color feature of the point cloud, and ... represents other possible features of the point cloud, such as the normal direction, etc. i Build a neighborhood subset of the point cloud for the center Other points in the subset The position information encoding process is as follows:

[0011]

[0012] in, Represents the encoded point cloud position information, θ pos (·) represents the position encoding mapping function, p i Represents point x i location information, Represents point x i The location information of the jth other point in the neighborhood subset.

[0013] Furthermore, in step 2, the relative deviation between the original feature information of the point cloud and the original feature information of other points in its neighborhood subset is calculated, and the original feature information of the point cloud is standardized based on the relative deviation and encoded to obtain the original feature information encoding of the point cloud.

[0014] Furthermore, in step S2, the feature deviation mean of all subsets is calculated based on the feature mean of the original feature of the point cloud and its neighborhood subset:

[0015]

[0016] in, Represents the center point x of the i-th point cloud i The original features of the jth other point cloud in the domain subset of Represents the i-th point cloud x iThe feature mean of the domain subset, Δ s represents the mean of the feature deviations of all neighborhood subsets in s scenes, Represents point cloud x i The scene you are in;

[0017] The standardized calculation formula for the deviation is as follows:

[0018]

[0019] Furthermore, the calculation formula for encoding the original feature information of the point cloud is as follows:

[0020]

[0021] in, Represents the original feature information of the encoded point cloud, θ info (·) represents the feature mapping function, and C(·) represents the concatenation of features. Based on the average deviation value, the color, normal direction and other features in each local point cloud subset are standardized and calculated, which can improve the sensitivity to color mutation and normal direction mutation.

[0022] Furthermore, in step 3, the original feature information and position information of the point cloud are combined and encoded, and the combined encoding formula is as follows:

[0023]

[0024] in, represents the combined encoding of the original feature z and the position coordinate p, θ z-p (·) represents the feature mapping function of information, Δ r Represents the coordinate radius of the point cloud subset, used to normalize the coordinate deviation.

[0025] Furthermore, in step 3, taking into account the possible associations between various features of the point cloud, a combined code containing different features is further designed on the basis of the above. Specifically, the combined code is based on various different feature information, and the combined code of each feature is spliced ​​with the original feature information code of the point cloud, and the lengths are aligned. The spliced ​​code is attached to the position information code for fusion, so as to obtain an enhanced feature code that fuses the original feature information of various point clouds.

[0026] Furthermore, in step 3, the fused codes are copied and concatenated to match the overall feature enhancement code with the point cloud feature dimension, so as to use vector addition to fuse the coding information with the point cloud features and supplement the basic features of the point cloud loss in the deep network.

[0027] Furthermore, the position information is encoded The feature vector dimension is other feature information encoding The feature vector dimension is twice that of , c represents the feature dimension, k represents the encoding replication factor, represents a real number set; the encoding fusion process formula of step 3 is as follows:

[0028]

[0029] Among them, δ z represents the original feature information encoding, δ pos represents the position information encoding, δ z-p represents the combined coding, δ sub-fus is an intermediate variable, which represents the feature coding at the channel level of the combined feature of different information codings. C(·) represents the concatenation of features. represents the replication of the encoding, δ fus It indicates the feature code with the same length as the point cloud feature after the copying and splicing process, that is, the fused code.

[0030] Furthermore, the original feature information encoding of the point cloud includes color information encoding and normal direction information encoding of the point cloud, and the combined encoding includes combined encoding of color information and position information of the point cloud, and combined encoding of normal direction information and position information; in the step S4, comprehensive encoding and targeted encoding are set, and the comprehensive encoding is a fusion of color information encoding, normal direction information encoding, and position information encoding; the targeted encoding includes color position fusion encoding, normal direction position fusion encoding, and position encoding, wherein color position fusion encoding is a fusion of color information encoding, combined encoding of color information and position information, and position information encoding, and normal direction position fusion encoding is a fusion of normal direction information encoding, combined encoding of normal direction information and position information, and position information encoding; in the deepest and shallowest layers of the point cloud semantic segmentation neural network, only basic position information encoding is introduced, and in the feature aggregation layers of other levels, it is allocated according to comprehensive encoding-color position fusion encoding-normal direction position fusion encoding-position encoding.

[0031] The advantages and beneficial effects of the present invention are:

[0032] The present invention provides a 3D point cloud semantic segmentation method based on point cloud information coding enhancement, which solves the problem of blurring original information in deep networks by comprehensively and efficiently introducing feature coding of various original information of point clouds in various feature aggregation modules of neural networks. In addition, an alternating coding strategy is used to alleviate the interference problem when multiple information codes coexist, and improve the effectiveness of original information supplementation. Experiments show that this plug-and-play method can effectively improve the effects of various point cloud semantic segmentation algorithms and achieve high-precision 3D scene perception. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 Flow chart of a 3D point cloud semantic segmentation method based on point cloud information coding enhancement in an embodiment of the present invention.

[0034] Figure 2a Schematic diagram of the method in the embodiment of the present invention.

[0035] Figure 2b Schematic diagram of the structure of the encoding / decoding module in an embodiment of the present invention.

[0036] Figure 3 Schematic diagram of the point cloud information encoding and fusion process in an embodiment of the present invention.

[0037] Figure 4 Schematic diagram of an alternate encoding strategy in an embodiment of the present invention.

[0038] Figure 5 This is a diagram showing the improvement effect of the method in an embodiment of the present invention compared with the baseline model in the S3DIS public dataset.

[0039] Figure 6 This is a diagram showing the improvement effect of the method in an embodiment of the present invention compared with the baseline model in the ScanNet public dataset. DETAILED DESCRIPTION

[0040] The specific implementation of the present invention is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the present invention, and is not used to limit the present invention.

[0041] like Figure 1 , Figure 2a , Figure 2b As shown, a 3D point cloud semantic segmentation method based on point cloud information coding enhancement includes the following steps:

[0042] Step 1: Encoding point cloud coordinate information: For each point cloud in the 3D scene, first sample a subset of the point cloud neighborhood, and calculate the relative coordinate difference between each point in the point cloud subset and the sampling center point to form a coordinate code.

[0043] Specifically, in each local point cloud subset at each feature aggregation stage of the neural network, point cloud coordinate information encoding is introduced to supplement the geometric features of the key point at the center of the subset and its surrounding neighborhood point cloud. The information encoding method used is a multi-layer perceptron. The point cloud subset is a point set obtained by neighborhood sampling with each point as the center in the scene point cloud, which represents the contextual information around the point. Preferably, the point cloud neighborhood sampling method is k-nearest neighbor sampling, and the sampling neighborhood is preferably 16.

[0044] Consider the scenario Any point x in i =(f i,p i ,c i ,...), Represents the semantic features of the point cloud, Represents the coordinate features of the point cloud, represents the color feature of the point cloud, and ... represents other possible features of the point cloud, such as normal direction, etc. At the same time, let the neighborhood point cloud subset centered at any point be Subset point The mathematical expression of the coordinate information encoding process is as follows:

[0045]

[0046] Among them, θ pos (·) is the mapping function used for coordinate encoding. Preferably, the mapping function is a multi-layer perceptron.

[0047] Step 2: Encoding the original feature information of the point cloud;

[0048] Calculate the deviation of the original feature of the point cloud; in each point cloud subset, further calculate the relative deviation of each point compared with other original information of the sampling center;

[0049] Specifically, in each local point cloud subset, the deviation between the color and normal direction feature averages of the point cloud center point and its neighboring point clouds is calculated to obtain the original feature deviation value of a single subset. The point cloud color, normal direction and other features must be normalized; the deviation calculation uses the numerical difference.

[0050] For each point cloud subset in the scene, its original feature deviation value needs to be calculated, and the average deviation value of each information in the scene needs to be calculated. Taking color information as an example, the calculation formula of this process is as follows:

[0051]

[0052] in, Represents the color characteristics of the point cloud center point and its neighboring point clouds, Representation Subset The mean color feature value in s Represents the mean of the color deviation values ​​of all subsets in the scene.

[0053] Standardization of original feature deviation of point cloud: according to the relative deviation of original information of each point cloud subset, the original feature deviation values ​​of all point cloud subsets in the scene are counted, and the color, normal direction and other features in each local point cloud subset are standardized based on the average deviation value, so as to improve the sensitivity to sudden changes in color and normal direction.

[0054] The standardized calculation formula for the deviation is as follows:

[0055]

[0056] Encoding of original feature information of point cloud: Based on the standardized point cloud information, all original feature information of point cloud is encoded.

[0057] Specifically, feature encoding of information is performed on each standardized point cloud subset to obtain information encoding of color and normal features of the point cloud, and to supplement the features of color change and normal direction change of the key point at the center of the subset and the point cloud in its surrounding area.

[0058] Taking color information as an example, the feature encoding calculation formula of point cloud color information is as follows:

[0059]

[0060] Among them, θ info (·) represents the feature mapping function of information, and C(·) represents the concatenation of features.

[0061] Combination coding design: Taking into account the possible associations between various features of point clouds, a combination coding containing different features is further designed based on the above.

[0062] Specifically, taking the color information of point cloud as an example, combining the color information and coordinate information of point cloud, we can get the combined encoding of mixed color and coordinate representation, and its mathematical expression is as follows:

[0063]

[0064] Among them, θ c-p (·) represents the feature mapping function of information, Δ r Represents the coordinate radius of the point cloud subset, used to normalize the coordinate deviation.

[0065] Encoding length alignment; e.g. Figure 3 As shown, after obtaining various codes, the color codes and mixed codes except the coordinate codes are vector-concatenated to align the feature vector lengths.

[0066] Specifically, after obtaining the coordinate coding, color coding, normal feature coding and various combination coding of each subset of the point cloud, the various coding are fused. In this process, the color coding and mixed coding except the coordinate coding are first vectorized and the feature vector lengths are aligned.

[0067] Step 3: Encoding fusion; Figure 3 As shown in FIG. 1 , the coordinate code is used as the basic feature, and the concatenated color, normal and other vectors are attached to it to obtain the feature code that integrates the original information of various point clouds.

[0068] Specifically, the distribution of the relative deviation of features such as color in 3D space is obtained through coding fusion, so as to characterize the texture and other properties of the point cloud subset. For the feature fusion process of coordinate coding and other coding, preferably, vector addition is used.

[0069] Code splicing combination; such as Figure 3 As shown in the figure, on the basis of obtaining the fusion code, the fused vector is copied and concatenated to match the overall feature enhancement code with the point cloud feature dimension, so as to use vector addition to fuse the coding information and point cloud features, and supplement the basic features of point cloud loss in the deep network. The vector copying and concatenation operations on the fused feature enhancement code do not change the features of the code, but only extend the vector length. Preferably, the point cloud feature dimensions of each level of the neural network are [32, 64, 128, 256, 512], and the coding dimensions of the fused point cloud information in each stage are [32, 64, 64, 64, 64], so the number of copies of the coding features at each stage is [1, 1, 2, 4, 8].

[0070] Furthermore, in the various information encoding mapping functions described in steps 1 and 2, the feature vector dimension of the coordinate information encoding is twice the length of other feature encodings. Specifically, assuming that the dimension of the point cloud feature at this stage is c, and the encoding replication factor is k, the coordinate encoding of the point cloud at this stage is Other codes are

[0071] On this basis, taking color information encoding, combined information encoding of color information and coordinate information, and coordinate information encoding as examples, the mathematical expression of the encoding fusion process described in step 3 is as follows:

[0072]

[0073] Among them, δ sub-fus Represents the feature coding at the channel level that combines the combined features of different information coding. fus It indicates the feature code with the same length as the point cloud feature after the copying and splicing process. Indicates a copy of the encoding.

[0074] Step 4: Original information encoding allocation strategy; Figure 4 As shown, the point cloud information encoding enhancement method of the above steps is alternately introduced into different local feature aggregation modules at each stage of the neural network, so that different local feature aggregation modules can obtain different original point cloud information supplements, alleviate the redundancy of information encoding and make each feature aggregation module focus on each other.

[0075] Specifically, in order to solve the conflict problem between feature encodings, a feature allocation strategy is introduced. Different original information enhancement encodings are alternately introduced into multiple local feature aggregation modules at each stage of the point cloud semantic segmentation network to alleviate the redundancy of information encoding and make each feature aggregation module focus on each other, thereby improving the effect of point cloud information encoding.

[0076] Furthermore, a variety of different feature coding combinations are involved, as shown in the following table:

[0077]

[0078] Among them, the comprehensive coding means that the coding fully supplements the coordinate, color, and normal direction information, δ col Indicates color information encoding, δ norm represents the normal direction information encoding, δ pos Indicates the location information encoding. Targeted encoding only contains two or one of the three, aiming to supplement specific point cloud information. c-p Represents the combined encoding of color information and position information, δ n-p Represents the combined encoding of normal direction information and position information.

[0079] At the same time, the feature allocation strategy involved in step 4 is preferably to introduce only basic coordinate information encoding in the deepest and shallowest layers of the neural network, and in the feature aggregation layers of other levels, the comprehensive encoding IEFM is used. mix -Color Position Fusion Coding IEFM col - Normal direction position fusion coding IEFM norm - Position Encoding IEFM pos Make an allocation.

[0080] In order to verify the effectiveness of this method, the present invention uses the public large-scale point cloud semantic segmentation datasets S3DIS and ScanNet to illustrate the method. Figure 5 and Figure 6 The experimental results show that this method demonstrates outstanding performance in large-scale 3D point cloud semantic segmentation datasets and achieves better results than the baseline models.

[0081] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some or all of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A 3D point cloud semantic segmentation method based on point cloud information coding enhancement, characterized in that The steps include: Step 1: Encoding the point cloud location information; constructing a neighborhood subset of the point cloud, and encoding the location information of the point cloud and other points in the neighborhood subset to obtain the location information encoding; Step 2: Encode the original feature information of the point cloud, and combine the point cloud position information encoding and the point cloud original feature information encoding; Step 3: Code fusion: Combining the combined code, point cloud position information code, and point cloud original feature information code to perform feature enhanced code fusion; Step 4: Point cloud coding allocation strategy; based on the different local feature aggregation modules at each stage of the point cloud semantic segmentation neural network, the point cloud information coding is alternately enhanced through the coding fusion, so that different local feature aggregation modules can obtain different supplements of the original feature information of the point cloud, and finally the semantic segmentation of the point cloud is completed.

2. The 3D point cloud semantic segmentation method based on point cloud information coding enhancement according to claim 1, characterized in that: In step 1, take any point x i Build a neighborhood subset of the point cloud for the center Other points in the subset The position information encoding process is as follows: in, Represents the encoded point cloud position information, θ pos (·) represents the position encoding mapping function, p i Represents point x i location information, Represents point x i The location information of the jth other point in the neighborhood subset.

3. The 3D point cloud semantic segmentation method based on point cloud information coding enhancement according to claim 1, characterized in that: In step 2, the relative deviation between the original feature information of the point cloud and the original feature information of other points in its neighborhood subset is calculated, and the original feature information of the point cloud is standardized based on the relative deviation and encoded to obtain the original feature information encoding of the point cloud.

4. The 3D point cloud semantic segmentation method based on point cloud information coding enhancement according to claim 3, characterized in that: In step S2, the feature deviation mean of all subsets is calculated based on the feature mean of the original feature of the point cloud and its neighborhood subset: in, Represents the i-th point cloud x i The original features of the jth other point cloud in the domain subset of Represents the i-th point cloud x i The feature mean of the domain subset, Δ s represents the mean of the feature deviations of all neighborhood subsets in s scenes, Represents point cloud x i The scene you are in; The standardized calculation formula for the deviation is as follows:

5. The 3D point cloud semantic segmentation method based on point cloud information coding enhancement according to claim 4, characterized in that: The calculation formula for encoding the original feature information of the point cloud is as follows: in, Represents the original feature information of the encoded point cloud, θ info (·) represents the feature mapping function, and C(·) represents the concatenation of features.

6. The 3D point cloud semantic segmentation method based on point cloud information coding enhancement according to claim 5, characterized in that: In step 2, the original feature information and position information of the point cloud are combined and encoded, and the combined encoding formula is as follows: in, represents the combined encoding of the original feature z and the position coordinate p, θ z-p (·) represents the feature mapping function of information, Δ r Indicates the coordinate radius of the point cloud subset.

7. The 3D point cloud semantic segmentation method based on point cloud information coding enhancement according to claim 1, characterized in that: In the step 2, combined coding is performed based on various types of feature information, and the combined coding of various types of features is spliced ​​with the original feature information coding of the point cloud, and the lengths are aligned. The spliced ​​coding is attached to the position information coding for fusion, so as to obtain an enhanced feature coding that fuses the original feature information of various types of point clouds.

8. The 3D point cloud semantic segmentation method based on point cloud information coding enhancement according to claim 7, characterized in that: In step 3, the fused codes are copied and concatenated to make the overall feature enhancement code match the feature dimension of the point cloud.

9. The 3D point cloud semantic segmentation method based on point cloud information coding enhancement according to claim 8, characterized in that: The position information encoding The feature vector dimension is other feature information encoding The feature vector dimension is twice that of , c represents the feature dimension, k represents the encoding replication factor, represents a real number set; the encoding fusion process formula of step 3 is as follows: Among them, δ z represents the original feature information encoding, δ pos represents the position information encoding, δ z-p represents the combined coding, δ sub-fus is an intermediate variable, which represents the feature coding at the channel level of the combined feature of different information codings. C(·) represents the concatenation of features. represents the replication of the encoding, δ fus It indicates the feature code with the same length as the point cloud feature after the copying and splicing process, that is, the fused code.

10. The 3D point cloud semantic segmentation method based on point cloud information coding enhancement according to claim 1, characterized in that: The original feature information encoding of the point cloud includes the color information encoding and the normal direction information encoding of the point cloud, and the combined encoding includes the combined encoding of the color information and the position information of the point cloud, and the combined encoding of the normal direction information and the position information; In the step S4, comprehensive coding and targeted coding are set, and comprehensive coding is a fusion of color information coding, normal direction information coding, and position information coding. Targeted coding includes color position fusion coding, normal direction position fusion coding, and position coding, wherein color position fusion coding is a fusion of color information coding, a combination coding of color information and position information, and position information coding, and normal direction position fusion coding is a fusion of normal direction information coding, a combination coding of normal direction information and position information, and position information coding; only basic position information coding is introduced in the deepest and shallowest layers of the point cloud semantic segmentation neural network, and in other levels of feature aggregation layers, it is allocated according to comprehensive coding-color position fusion coding-normal direction position fusion coding-position coding.

Images