A point cloud data processing method, device and medium based on random sampling

Abstract

The present invention provides a point cloud data processing method, device and medium based on random sampling, which relates to the field of point cloud analysis technology. The method includes obtaining original point cloud information; performing feature encoding processing according to the original point cloud information, obtaining deep-level features based on random sampling downsampling operations to obtain encoding information; performing feature decoding processing on the encoding information, obtaining decoding information based on multi-dimensional interpolation splicing and transposed convolution operations; constructing a point cloud semantic segmentation model based on the decoded information, and using the point cloud semantic segmentation model to perform data processing on the original point cloud information to obtain a point cloud semantic segmentation result. The present invention solves the problems of slow processing time, large number of parameters and low segmentation performance of existing point cloud semantic segmentation models in point cloud data in large scenes, as well as the problem of losing the semantic segmentation accuracy of the model for a small number of labels when the number of labels has significant differences.

Description

Technical Field

[0001] The present invention relates to the technical field of point cloud analysis, and in particular to a point cloud data processing method, device and medium based on random sampling. Background Art

[0002] 3D point clouds are widely used in important visual tasks such as autonomous driving, augmented reality, and architectural model reconstruction. With the rapid development of deep learning, semantic segmentation has achieved remarkable results in the field of images, and semantic segmentation on 3D point clouds has received more and more attention. With the advancement and development of 3D scanning equipment such as 3D LiDAR and sensor technology, it has become more reliable to obtain high-quality 3D point cloud data. Compared with the image field, which has been deeply cultivated for many years, 3D point cloud semantic segmentation still has a lot of room for improvement in terms of accuracy and performance. At the same time, due to the commonalities between images and point clouds, the methods in the image field can be improved to meet the needs of 3D point clouds.

[0003] Most of the models proposed in point cloud semantic segmentation adopt an encoding-decoding structure. Deep learning models with various different strategies have achieved good results in point cloud semantic segmentation. However, existing models often find it difficult to balance the segmentation accuracy and point cloud labels in large segmentation scenes. At the same time, there is a significant decline in segmentation performance when the number of labels is extremely unbalanced. Summary of the invention

[0004] The purpose of the present invention is to provide a point cloud data processing method, device and medium based on random sampling to improve the above problems. In order to achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0005] In a first aspect, the present application provides a point cloud data processing method based on random sampling, comprising:

[0006] Get original point cloud information;

[0007] Perform feature encoding processing according to the original point cloud information, and obtain deep features based on random sampling downsampling operations to obtain encoding information;

[0008] Performing feature decoding processing on the encoded information, and obtaining decoded information based on multi-dimensional interpolation, splicing and transposed convolution operations;

[0009] A point cloud semantic segmentation model is constructed based on the decoded information, and the point cloud semantic segmentation model is used to perform data processing on the original point cloud information to obtain a point cloud semantic segmentation result.

[0010] In a second aspect, the present application also provides a point cloud data processing device based on random sampling, comprising:

[0011] An acquisition unit, used for acquiring original point cloud information;

[0012] An encoding unit, used for performing feature encoding processing according to the original point cloud information, obtaining deep-level features based on a downsampling operation of random sampling to obtain encoding information;

[0013] A decoding unit, used to perform feature decoding processing on the encoded information, and obtain decoded information based on multi-dimensional interpolation, splicing and transposed convolution operations;

[0014] A processing unit is used to construct a point cloud semantic segmentation model based on the decoded information, and use the point cloud semantic segmentation model to perform data processing on the original point cloud information to obtain a point cloud semantic segmentation result.

[0015] In a third aspect, the present application also provides a point cloud data processing device based on random sampling, comprising:

[0016] Memory for storing computer programs;

[0017] A processor is used to implement the steps of the point cloud data processing method based on random sampling when executing the computer program.

[0018] In a fourth aspect, the present application further provides a readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the above-mentioned point cloud data processing method based on random sampling are implemented.

[0019] The beneficial effects of the present invention are as follows: the present invention uses a point cloud semantic segmentation model based on random sampling, adds local coordinate attention and position encoding to better pay attention to the neighborhood information of points and the relative information between neighboring points, captures deep-level feature information through random downsampling, combines multi-level information encoding and decoding, and thus fuses point cloud feature information of different scales. It can better handle data with highly unbalanced labels, smooth label boundaries, and improve the accuracy of point cloud semantic segmentation.

[0020] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or be understood by implementing the embodiments of the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 Schematic diagram of the process of point cloud data processing based on random sampling in an embodiment of the present invention;

[0023] Figure 2 Schematic diagram of the structure of the point cloud semantic segmentation model in an embodiment of the present invention;

[0024] Figure 3 Schematic diagram of the structure of the encoding module in an embodiment of the present invention;

[0025] Figure 4 A schematic diagram of the structure of the local coordinate attention in an embodiment of the present invention;

[0026] Figure 5 Schematic diagram of the structure of the self-attention mechanism in an embodiment of the present invention;

[0027] Figure 6 This is a schematic diagram of the structure of a location binding module in an embodiment of the present invention;

[0028] Figure 7 This is a schematic diagram of the structure of attention pooling in an embodiment of the present invention;

[0029] Figure 8 Schematic diagram of the structure of a point cloud data processing device based on random sampling described in an embodiment of the present invention.

[0030] Markings in the figure: 800, point cloud data processing device based on random sampling; 801, processor; 802, memory; 803, multimedia component; 804, I / O interface; 805, communication component. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0032] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings. At the same time, in the description of the present invention, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0033] Embodiment 1:

[0034] This embodiment provides a point cloud data processing method based on random sampling.

[0035] See also Figure 1 , the figure shows that the method includes step S100, step S200, step S300, and step S400.

[0036] Step S100: obtaining original point cloud information;

[0037] Step S200: performing feature encoding processing according to the original point cloud information, obtaining deep-level features based on a downsampling operation of random sampling to obtain encoding information;

[0038] Step S300: performing feature decoding processing on the encoded information, and obtaining decoded information based on multi-dimensional interpolation, splicing and transposed convolution operations;

[0039] Step S400: constructing a point cloud semantic segmentation model based on the decoded information, and using the point cloud semantic segmentation model to perform data processing on the original point cloud information to obtain a point cloud semantic segmentation result.

[0040] In order to better balance accuracy and performance, this embodiment improves the average intersection-over-union ratio of model segmentation when the number of labels is unbalanced. Figure 2Specifically, the point cloud semantic segmentation model adopts an encoder-decoder structure, captures deep feature information through random downsampling, and combines multi-level information encoding and decoding to achieve the purpose of point cloud semantic segmentation. Figure 2 In the above, the feature block represents the feature information output by each module. Indicates the number of points, represents the feature dimension, Indicates the dimension of the output data.

[0041] The step S200 includes:

[0042] Step S201: constructing an encoder according to the original point cloud information, wherein the encoder comprises a plurality of sequentially connected encoding layers, and the encoding layers comprise sequentially connected encoding modules and sampling modules;

[0043] Step S202: encoding and downsampling the input feature information through each coding layer to obtain coded feature information output by each coding layer;

[0044] Step S203: The coding feature information output by each coding layer is taken together as the coding information output by the encoder.

[0045] In this embodiment, the encoder is used to perform multi-layer feature extraction. Encoding is performed through encoding modules of multiple encoding layers to generate point cloud encoding information of the feature dimension corresponding to the layer, and then random downsampling is performed with a ratio of 4 to obtain encoded feature information. In each encoding layer, the sampling points of the layer are taken as the input of the next encoding module to mine deep features and form new encoded feature information.

[0046] The step S202 includes:

[0047] Step A1: The feature information inputted from each coding layer is transformed through two shared multi-layer perceptrons to obtain first transformed information and second transformed information;

[0048] Step A2: Position encoding the original point cloud information and the first transformation information through local coordinate attention to obtain attention weights, thereby obtaining first enhanced information;

[0049] Step A3: performing feature transformation on the first enhanced information through a shared multi-layer perceptron to obtain third transformed information;

[0050] Step A4: The first enhanced information is sequentially subjected to a position binding module and attention pooling to enhance the position information, thereby obtaining second enhanced information;

[0051] Step A5: splicing the third transformation information and the second enhancement information to obtain first splicing information, and performing activation function calculation and normalization processing on the first splicing information in sequence to obtain first feature information;

[0052] Step A6: The first feature information is sequentially subjected to a position binding module and attention pooling to enhance the position information, thereby obtaining third enhanced information;

[0053] Step A7: performing feature transformation on the first feature information through a shared multi-layer perceptron to obtain fourth transformed information;

[0054] Step A8: After feature concatenation of the fourth transformed information and the third enhanced information, feature transformation is performed in sequence through activation function calculation, normalization processing and shared multi-layer perceptron to obtain fifth transformed information;

[0055] Step A9: After feature concatenation of the fifth transformation information and the second transformation information, point cloud coding information is obtained through activation function calculation and normalization processing, wherein the point cloud coding information is used as output feature information of the coding module corresponding to the coding layer;

[0056] Step A10: Randomly downsample the point cloud coding information through the sampling module of the coding layer to obtain the coding feature information of the coding layer.

[0057] In this embodiment, the coding layer includes a coding module and a sampling module, and the specific structure of the coding module is as follows: Figure 3 As shown, local coordinate attention, referred to as LCA, (+) represents vector addition, Indicates the number of points, and Both represent feature dimensions.

[0058] The encoding module consists of local coordinate attention, position binding module and attention pooling, which aims to expand the feature dimension of points and strengthen information association. At the same time, multiple shared multi-layer perceptrons are added to the encoding module. The shared multi-layer perceptrons are used to perform feature transformation so that the model can better fit the data set, and the residual connection is used to add a direct channel between the encoding module and the sampling module, which effectively solves the problem of gradient disappearance and gradient explosion, thereby promoting the rapid convergence of the model.

[0059] At each stage of downsampling through random sampling, the encoding module can deeply extract the feature information of the point, improve the problem of local point cloud information loss caused by downsampling, gradually expand the receptive field of the downsampled point, and expand the feature dimension to maintain the surrounding information of the point, so as to capture deep information.

[0060] In the encoding module, after the feature input, only one self-attention calculation with position encoding is performed, namely local coordinate attention, and two layers of position binding modules and attention pooling are connected after the local coordinate attention. This approach achieves a good balance between efficiency and performance.

[0061] The specific steps of performing position encoding through local coordinate attention to obtain attention weights are:

[0062] Step B1: obtaining first input feature information and second input feature information, wherein the first input feature information and the second input feature information are both feature information of local coordinate attention input, and the second input feature information is original point cloud information;

[0063] In this embodiment, in order to enable the model to maximize the relationship between local information, local coordinate attention is proposed. The specific structure of local coordinate attention is as follows: Figure 4 As shown in the figure, local coordinate attention is referred to as LCA, local coordinate encoding block is referred to as LCE, (+) represents vector addition, Indicates the point Neighbors Dimensional features, represents the neighboring point index, Represents the feature dimension.

[0064] In this embodiment, the first input feature information is the first transformation information of the coding layer, and the KNN algorithm is used to obtain the neighboring point information near the point, that is, to obtain the first neighboring point information of each point in the first transformation information and the second neighboring point information of the original point cloud information.

[0065] Step B2: Obtain the first neighboring point information of each point in the first input feature information;

[0066] Step B3: Calculate the relative coordinate information and Euclidean distance between each point and its neighboring point based on the first neighboring point information;

[0067] Step B4: Calculate the position code of each point in multiple dimensions according to the trigonometric function and the first neighbor point information;

[0068] In this embodiment, in the local coordinate encoding block, the trigonometric function PE is used to construct the position encoding, wherein the position encoding formula is:

[0069] ;

[0070] ;

[0071] In the formula, Indicates when When it is an even number, Points With point No. The relative coordinate information of the neighboring points is in The position encoding of the dimension, Indicates when When it is an odd number, Points With point No. The relative coordinate information of the neighboring points is in The position encoding of the dimension, represents the dimension of the local vector, Indicates Points No. Neighboring points, , Indicate point The first neighbor information.

[0072] Step B5: Obtaining coordinate information and second neighboring point information of each point in the second input feature information;

[0073] Step B6: constructing local coordinate codes of multiple local blocks based on the first neighbor point information, relative coordinate information, Euclidean distance and position code of each point, each of which consists of a point and corresponding neighbor points;

[0074] In this embodiment, the three-dimensional point cloud information naturally carries coordinate information compared to language information, that is, it contains position information in each dimension. The coordinate information itself helps the model understand the context. In position encoding, the low-frequency dimension position encoding changes quickly, which enables the model to learn the subtle relationship between neighboring points. The high-frequency dimension position encoding changes slowly, which enables points to capture long-distance dependencies.

[0075] Therefore, using relative coordinate information will enable self-attention to focus on local structural changes, thereby strengthening the capture of local relationships. Using absolute coordinate information allows self-attention to consider the coordinate relationship of the entire point cloud while paying attention to local position changes. Usually, the coordinate range of a large point cloud may be huge, so parameters are set to scale the coordinate scale to match the position changes of multiple frequency dimensions in position encoding.

[0076] In this embodiment, in LCE, for the Points Points in the constructed local block , that is, point Neighbors , whose local coordinates are encoded as:

[0077] ;

[0078] In the formula, Indicate point The local coordinate encoding of represents a linear transformation, Indicates Points, Indicate point No. Neighboring points, Indicate point With point The relative coordinate information of Indicate point With point The Euclidean distance of Indicate point Positional encoding in all dimensions, represents feature concatenation, , Indicate point The first neighbor information.

[0079] Step B7: performing point-by-point feature transformation on each point based on the local coordinate code, the second neighboring point information and the coordinate information to obtain first transformed feature information, where the first transformed feature information includes an attention weight of a local block corresponding to each point;

[0080] In this embodiment, the structure of the self-attention mechanism is as follows Figure 5 As shown, represents point multiplication, Tile represents copying, i.e. copying the point to a structure that matches the number of its neighbors, M represents vector multiplication, S represents the SoftMax function, (+) and (-) represent vector addition and vector subtraction respectively, Q represents query, K represents key value, and V represents weight. represents the neighboring point index, represents the feature dimension, Indicates sum.

[0081] In the self-attention mechanism, the feedforward layer extracts the complex features of the input data through linear transformation and activation function, thereby enhancing the model's nonlinear expression ability and ability to understand data. The point-by-point feature transformation is a linear transformation that enables the model to enhance its data understanding ability.

[0082] Specifically, through the vector attention in the self-attention mechanism, the attention weight is obtained for the local block composed of each point, and the formula for calculating the attention weight is:

[0083] ;

[0084] ;

[0085] In the formula, Indicate point The attention weight corresponding to the local block, Indicates Points, Indicate point No. Neighboring points, Indicate point No. Neighboring points, Indicate point The first neighbor information of Indicate point The second neighbor information, represents the SoftMax function, , , , and Both represent point-wise feature transformation, represents the intermediate vector, Indicate point The local coordinate encoding of represents the dot product, where the point-by-point feature transformation It is the feedforward layer, which consists of two linear layers and an activation layer. It is input into the activation function through the residual connection to obtain vector attention.

[0086] Therefore, the self-attention mechanism unifies different features into the same dimension through multiple point-by-point feature transformations.

[0087] Step B8: After concatenating the first transformed feature information and the first input feature information, activation function calculation and normalization processing are performed to obtain feature information output by local coordinate attention.

[0088] The specific steps of enhancing the position information through the position binding module and attention pooling are:

[0089] Step C1: obtaining third neighboring point information of each point in the third input feature information, where the third input feature information is feature information input by the position binding module;

[0090] Step C2: converting the feature dimension of the local coordinate encoding through a shared multi-layer perceptron to obtain feature conversion information;

[0091] Step C3: splicing the third neighbor point information and the feature conversion information to obtain spliced ​​neighbor point information, where the spliced ​​neighbor point information is feature information output by the position binding module;

[0092] Step C4: calculating the splicing neighbor point information by linear transformation and normalized exponential function in sequence to obtain second transformation feature information;

[0093] Step C5: performing a dot product operation and a summation operation on the second transformed feature information and the spliced ​​neighbor point information in sequence to obtain feature fusion information;

[0094] Step C6: Expand the feature dimension of the feature fusion information through a shared multi-layer perceptron to obtain enhanced feature information, where the enhanced feature information is feature information output by attention pooling.

[0095] In this embodiment, the position binding module is used to strengthen the position information of the point. The local coordinate attention LCA can well aggregate the feature information of the local points, capture the association between the local points, and capture multi-scale information under random sampling to expand the receptive field. By simply stacking LCA, the association between points can be extremely well strengthened.

[0096] However, the computational performance loss it brings is huge, and this performance loss is particularly obvious when processing large point clouds. Compared with the Transformer in natural language processing, the point cloud information in space focuses more on the relationship in position structure rather than the multi-dimensional relationship between words in natural language processing. For example, when natural language processing captures the subject-predicate, cause-effect and other relationships between sentences, it uses a multi-head attention mechanism to capture the multiple relationships of word vectors. This means that point cloud information can use position encoding with fewer dimensions to capture the relationship between points at different distances.

[0097] Therefore, after obtaining self-attention, we hope to strengthen the position information in the feature in a simpler way. For this purpose, a position binding module is designed to splice the obtained local coordinate encoding into the local feature containing self-attention information, and expand it to the output dimension of the model through a shared multi-layer perceptron.

[0098] like Figure 6 As shown, it is the structural diagram of the location binding module, where: represents feature concatenation, represents the neighboring point index, Represents the feature dimension.

[0099] The position binding module uses the KNN algorithm to obtain the neighboring point information of the midpoint of the current feature information, that is, to obtain the third neighboring point information of each point. At the same time, the local coordinate encoding is converted into feature dimensions through a shared multi-layer perceptron to obtain feature conversion information. Finally, the third neighboring point information and the feature conversion information are spliced ​​to obtain the spliced ​​neighboring point information.

[0100] In this embodiment, the structure of attention pooling is as follows Figure 7 As shown, where S represents the SoftMax function, represents dot product, represents the neighboring point index, and Both represent feature dimensions.

[0101] Attention pooling assigns a weight to each element in the sequence, which reflects the relative importance of the element in the entire sequence. Using these weights to perform a weighted summation of sequence elements helps remove redundant information, retain key features, strengthen context associations, and improve overall efficiency.

[0102] The attention pooling method can be understood as the re-weighting and integration of the local splicing neighboring point information formed after each point passes through the position binding module. The use of the position binding module and attention pooling simplifies the model complexity of directly using LCE stacking, while enhancing the connection between the features after self-attention and the position, and strengthening the position information.

[0103] In this embodiment, the expression of attention pooling is:

[0104] ;

[0105] ;

[0106] In the formula, Represents the feature information of the attention pooling output, Indicates the midpoint of the splicing neighbor point information No. Neighboring points, Indicates Points, and Both represent weight parameters, and Both represent offset terms, represents dot product, represents a linear transformation, represents the SoftMax function, represents the hyperbolic tangent function, represents a shared multilayer perceptron, Represents the number of neighboring points.

[0107] The step S300 includes:

[0108] Step S301: constructing a decoder based on the encoder, the decoder comprising a plurality of decoding layers connected in sequence, each decoding layer corresponding to a coding layer one by one;

[0109] Step S302: up-sampling the fourth input feature information of each decoding layer by nearest neighbor interpolation to obtain first decoding feature information of the decoding layer, wherein the fourth input feature information is feature information output by a previous decoding layer corresponding to the decoding layer;

[0110] Step S303: After concatenating the first decoding feature information of each decoding layer with the fifth input feature information of the decoding layer, a transposed convolution operation is performed to obtain the second decoding feature information of the decoding layer, where the second decoding feature information is the feature information output by the decoding layer, and the fifth input feature information is the encoding feature information output by the encoding layer corresponding to the decoding layer;

[0111] Step S304: Use the second decoding feature information of the last decoding layer as the decoding information output by the decoder.

[0112] In this embodiment, the decoder is designed as multiple decoding layers, each decoding layer is a splicing and transposed convolution module, which is used to perform multi-dimensional post-interpolation splicing and transposed convolution operations, encode the feature coding information of the corresponding encoding layer and the feature information output by the previous decoding layer, and obtain the feature channel output of the encoding layer, so as to fuse the point cloud feature information of different scales, and finally convert the decoded information into a dimension through a shared multi-layer perceptron to obtain the point cloud semantic segmentation result, thereby completing the semantic segmentation of the point cloud data.

[0113] Embodiment 2:

[0114] When selecting the loss function of the model, cross entropy loss is a commonly used loss in processing point cloud segmentation. However, in the point cloud semantic segmentation model that adopts the random downsampling method in the present invention, when processing a serious imbalance in the number of labels of each category, the accuracy of individual labels with a small number may be affected due to the characteristics of random sampling. For this reason, the Focal Loss loss function is introduced. The Focal Loss loss function can help the model better focus on samples that are difficult to classify by adjusting the weights of easy-to-classify samples, thereby improving the overall classification accuracy, especially in processing datasets with unbalanced categories.

[0115] In this embodiment, the loss function of the point cloud semantic segmentation model is expressed as:

[0116] ;

[0117] In the formula, represents the Focal Loss loss function, represents the probability of the corresponding true category in the predicted probability vector, Indicates category The weight factor of Represents the modulation factor. When the model's predicted probability for a sample is close to 1, the loss value will be lower, and when the predicted probability for a sample is close to 0.5, the loss value is higher. In this way, the model is encouraged to pay more attention to samples that are difficult to classify correctly, thereby reducing the attention to samples that are easy to classify.

[0118] In this embodiment, the point cloud semantic segmentation model and the RandLA-Net model are also verified and tested on the bridge dataset.

[0119] The bridge dataset is mainly composed of bridges, including labels of bridge components (bridge deck, bridge piers, pier caps, and pad stones). The bridge dataset is obtained by radar acquisition of real bridge buildings. The characteristics of the dataset are large data volume and serious uneven distribution of the number of categories.

[0120] In this embodiment, six-fold cross validation is performed on the bridge dataset. The six-fold cross validation is performed by selecting datasets in different regions as test sets and performing training validation on them respectively. As shown in Table 1, it is the average value of the average intersection-over-union ratio of the point cloud semantic segmentation model and the RandLA-Net model under different test sets.

[0121] Table 1 Comparison of the average intersection-over-union ratios of the point cloud semantic segmentation model and the RandLA-Net model

[0122]

[0123] Table 2 gives the classification accuracy of each bridge component label and the average accuracy of all bridge component labels in different areas of the point cloud semantic segmentation model and the RandLA-Net model. Table 2 is a comparison table of the accuracy of the point cloud semantic segmentation model and the RandLA-Net model.

[0124] Table 2 Accuracy comparison of point cloud semantic segmentation model and RandLA-Net model

[0125]

[0126] It can be seen that the point cloud semantic segmentation model provided by the present invention shows obvious advantages in all categories and average accuracy compared with the RandLA-Net model which is also based on random sampling. By adding local coordinate self-attention and position encoding, it can better pay attention to the neighborhood information of the point and the relative information between neighboring points, better balance the accuracy and performance, improve the average intersection-over-union ratio of the model segmentation when the number of labels is unbalanced, and improve the accuracy of point cloud semantic segmentation.

[0127] Embodiment 3:

[0128] This embodiment provides a point cloud data processing device based on random sampling, the device comprising:

[0129] An acquisition unit, used for acquiring original point cloud information;

[0130] An encoding unit, used for performing feature encoding processing according to the original point cloud information, obtaining deep-level features based on a downsampling operation of random sampling to obtain encoding information;

[0131] A decoding unit, used to perform feature decoding processing on the encoded information, and obtain decoded information based on multi-dimensional interpolation, splicing and transposed convolution operations;

[0132] A processing unit is used to construct a point cloud semantic segmentation model based on the decoded information, and use the point cloud semantic segmentation model to perform data processing on the original point cloud information to obtain a point cloud semantic segmentation result.

[0133] The encoding unit comprises:

[0134] A construction subunit, configured to construct an encoder according to the original point cloud information, wherein the encoder comprises a plurality of sequentially connected encoding layers, and the encoding layers comprise sequentially connected encoding modules and sampling modules;

[0135] The encoding and sampling subunit is used to encode and downsample the input feature information through each encoding layer to obtain the encoded feature information output by each encoding layer;

[0136] The encoding output subunit is used to use the encoding feature information output by each encoding layer as the encoding information output by the encoder.

[0137] It should be noted that, regarding the device in the above embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment of the method, and will not be elaborated here.

[0138] Embodiment 4:

[0139] Corresponding to the above method embodiment, a point cloud data processing device based on random sampling is also provided in this embodiment. The point cloud data processing device based on random sampling described below and the point cloud data processing method based on random sampling described above can refer to each other.

[0140] Figure 8 is a block diagram of a point cloud data processing device 800 based on random sampling according to an exemplary embodiment. Figure 8 As shown, the point cloud data processing device 800 based on random sampling may include: a processor 801 and a memory 802. The point cloud data processing device 800 based on random sampling may also include one or more of a multimedia component 803, an I / O interface 804, and a communication component 805.

[0141] The processor 801 is used to control the overall operation of the random sampling-based point cloud data processing device 800 to complete all or part of the steps in the random sampling-based point cloud data processing method. The memory 802 is used to store various types of data to support the operation of the random sampling-based point cloud data processing device 800, and these data may include, for example, instructions for any application or method operating on the random sampling-based point cloud data processing device 800, and application-related data, such as contact data, sent and received messages, pictures, audio, video, etc. The memory 802 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, disk or optical disk. The multimedia component 803 may include a screen and an audio component. The screen may be, for example, a touch screen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signal may be further stored in the memory 802 or sent via the communication component 805. The audio component also includes at least one speaker for outputting audio signals. The I / O interface 804 provides an interface between the processor 801 and other interface modules, and the above-mentioned other interface modules can be keyboards, mice, buttons, etc. These buttons can be virtual buttons or physical buttons. The communication component 805 is used for wired or wireless communication between the point cloud data processing device 800 based on random sampling and other devices. Wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G or 4G, or a combination of one or more of them, so the corresponding communication component 805 can include: Wi-Fi module, Bluetooth module, NFC module.

[0142] In an exemplary embodiment, the point cloud data processing device 800 based on random sampling can be implemented by one or more application specific integrated circuits (Application Specific Integrated Circuit, referred to as ASIC), digital signal processors (Digital Signal Processing Device, referred to as DSP), digital signal processing devices (Digital Signal Processing Device, referred to as DSPD), programmable logic devices (Programmable Logic Device, referred to as PLD), field programmable gate arrays (Field Programmable Gate Array, referred to as FPGA), controllers, microcontrollers, microprocessors or other electronic components to execute the above-mentioned point cloud data processing method based on random sampling.

[0143] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, and when the program instructions are executed by a processor, the steps of the above-mentioned point cloud data processing method based on random sampling are implemented. For example, the computer-readable storage medium can be the above-mentioned memory 802 including program instructions, and the above-mentioned program instructions can be executed by the processor 801 of the point cloud data processing device 800 based on random sampling to complete the above-mentioned point cloud data processing method based on random sampling.

[0144] Embodiment 5:

[0145] Corresponding to the above method embodiment, a readable storage medium is also provided in this embodiment. The readable storage medium described below and the point cloud data processing method based on random sampling described above can refer to each other.

[0146] A readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the point cloud data processing method based on random sampling in the above method embodiment are implemented.

[0147] The readable storage medium may specifically be a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or other readable storage medium that can store program codes.

[0148] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

[0149] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A point cloud data processing method based on random sampling, characterized in that: include: Get original point cloud information; Perform feature encoding processing according to the original point cloud information, and obtain deep features based on random sampling downsampling operations to obtain encoding information; Performing feature decoding processing on the encoded information, and obtaining decoded information based on multi-dimensional interpolation, splicing and transposed convolution operations; A point cloud semantic segmentation model is constructed based on the decoded information, and the original point cloud information is processed using the point cloud semantic segmentation model to obtain a point cloud semantic segmentation result; The feature encoding process is performed according to the original point cloud information, and the deep-level features are obtained by a downsampling operation based on random sampling to obtain the encoding information, including: Constructing an encoder according to the original point cloud information, wherein the encoder comprises a plurality of sequentially connected encoding layers, and the encoding layers comprise sequentially connected encoding modules and sampling modules; The input feature information is encoded and downsampled through each encoding layer to obtain the encoded feature information output by each encoding layer; The coded feature information output by each coding layer is used together as the coded information output by the encoder; The step of encoding and downsampling the input feature information through each coding layer to obtain the coded feature information of each coding layer includes: The feature information inputted by each coding layer is transformed by two shared multi-layer perceptrons to obtain first transformed information and second transformed information; Position-encode the original point cloud information and the first transformation information through local coordinate attention to obtain attention weights, thereby obtaining first enhanced information; Performing feature transformation on the first enhanced information through a shared multi-layer perceptron to obtain third transformed information; The first enhanced information is sequentially subjected to a position binding module and attention pooling to enhance the position information, thereby obtaining second enhanced information; The third transformation information and the second enhancement information are spliced ​​to obtain first spliced ​​information, and the first spliced ​​information is sequentially subjected to activation function calculation and normalization processing to obtain first feature information; The first feature information is sequentially enhanced by a position binding module and an attention pooling to obtain third enhanced information; Performing feature transformation on the first feature information through a shared multi-layer perceptron to obtain fourth transformed information; After feature concatenation of the fourth transformed information and the third enhanced information, feature transformation is performed in sequence through activation function calculation, normalization processing, and a shared multi-layer perceptron to obtain fifth transformed information; After feature concatenation of the fifth transformation information and the second transformation information, point cloud coding information is obtained through activation function calculation and normalization processing, wherein the point cloud coding information is used as output feature information of the coding module corresponding to the coding layer; The point cloud coding information is randomly downsampled through the sampling module of the coding layer to obtain coding feature information of the coding layer; The specific steps of performing position encoding through local coordinate attention to obtain attention weights are: Acquire first input feature information and second input feature information, wherein the first input feature information and the second input feature information are both feature information of local coordinate attention input, and the second input feature information is original point cloud information; Obtaining first neighboring point information of each point in the first input feature information; Calculate the relative coordinate information and Euclidean distance between each point and its neighboring point based on the first neighboring point information; Calculate the position encoding of each point in multiple dimensions according to the trigonometric function and the first neighbor point information; Obtaining coordinate information and second neighboring point information of each point in the second input feature information; Based on the first neighbor point information, relative coordinate information, Euclidean distance and position code of each point, construct local coordinate codes of multiple local blocks, each of which is composed of a point and corresponding neighbor points; Performing point-by-point feature transformation on each point based on the local coordinate code, the second neighboring point information and the coordinate information to obtain first transformed feature information, where the first transformed feature information includes an attention weight of a local block corresponding to each point; After concatenating the first transformed feature information and the first input feature information, activation function calculation and normalization processing are performed to obtain feature information of the local coordinate attention output; The specific steps of enhancing the position information through the position binding module and attention pooling are: Acquire third neighboring point information of each point in the third input feature information, where the third input feature information is feature information input by the position binding module; The local coordinate encoding is converted into feature dimensions by a shared multi-layer perceptron to obtain feature conversion information; Splicing the third neighbor point information and the feature conversion information to obtain spliced ​​neighbor point information, where the spliced ​​neighbor point information is feature information output by the position binding module; The splicing neighbor point information is calculated sequentially through linear transformation and normalized exponential function to obtain second transformation feature information; Performing a dot product operation and a sum operation on the second transformed feature information and the spliced ​​neighbor point information in sequence to obtain feature fusion information; The feature fusion information is expanded in feature dimension by a shared multi-layer perceptron to obtain enhanced feature information, where the enhanced feature information is feature information output by attention pooling; The feature decoding process is performed on the encoded information to obtain the decoded information based on multi-dimensional interpolation, splicing and transposed convolution operations, including: Constructing a decoder based on the encoder, the decoder comprising a plurality of decoding layers connected in sequence, each of the decoding layers corresponding one to ... Up-sampling the fourth input feature information of each decoding layer by nearest neighbor interpolation to obtain first decoding feature information of the decoding layer, wherein the fourth input feature information is feature information output by a previous decoding layer corresponding to the decoding layer; After concatenating the first decoding feature information of each decoding layer with the fifth input feature information of the decoding layer, a transposed convolution operation is performed to obtain the second decoding feature information of the decoding layer, where the second decoding feature information is the feature information output by the decoding layer, and the fifth input feature information is the encoding feature information output by the encoding layer corresponding to the decoding layer; The second decoding feature information of the last decoding layer is used as the decoding information output by the decoder.

2. A point cloud data processing device based on random sampling, characterized in that: include: An acquisition unit, used for acquiring original point cloud information; An encoding unit, used for performing feature encoding processing according to the original point cloud information, obtaining deep-level features based on a downsampling operation of random sampling to obtain encoding information; A decoding unit, used to perform feature decoding processing on the encoded information, and obtain decoded information based on multi-dimensional interpolation, splicing and transposed convolution operations; A processing unit, configured to construct a point cloud semantic segmentation model based on the decoded information, and use the point cloud semantic segmentation model to perform data processing on the original point cloud information to obtain a point cloud semantic segmentation result; The encoding unit comprises: A construction subunit, configured to construct an encoder according to the original point cloud information, wherein the encoder comprises a plurality of sequentially connected encoding layers, and the encoding layers comprise sequentially connected encoding modules and sampling modules; The encoding and sampling subunit is used to encode and downsample the input feature information through each encoding layer to obtain the encoded feature information output by each encoding layer; The coding output subunit is used to use the coding feature information output by each coding layer as the coding information output by the encoder; The step of encoding and downsampling the input feature information through each coding layer to obtain the coded feature information of each coding layer includes: The feature information inputted by each coding layer is transformed by two shared multi-layer perceptrons to obtain first transformed information and second transformed information; Position-encode the original point cloud information and the first transformation information through local coordinate attention to obtain attention weights, thereby obtaining first enhanced information; Performing feature transformation on the first enhanced information through a shared multi-layer perceptron to obtain third transformed information; The first enhanced information is sequentially subjected to a position binding module and attention pooling to enhance the position information, thereby obtaining second enhanced information; The third transformation information and the second enhancement information are spliced ​​to obtain first spliced ​​information, and the first spliced ​​information is sequentially subjected to activation function calculation and normalization processing to obtain first feature information; The first feature information is sequentially enhanced by a position binding module and an attention pooling to obtain third enhanced information; Performing feature transformation on the first feature information through a shared multi-layer perceptron to obtain fourth transformed information; After feature concatenation of the fourth transformed information and the third enhanced information, feature transformation is performed in sequence through activation function calculation, normalization processing, and a shared multi-layer perceptron to obtain fifth transformed information; After feature concatenation of the fifth transformation information and the second transformation information, point cloud coding information is obtained through activation function calculation and normalization processing, wherein the point cloud coding information is used as output feature information of the coding module corresponding to the coding layer; The point cloud coding information is randomly downsampled through the sampling module of the coding layer to obtain coding feature information of the coding layer; The specific steps of performing position encoding through local coordinate attention to obtain attention weights are: Acquire first input feature information and second input feature information, wherein the first input feature information and the second input feature information are both feature information of local coordinate attention input, and the second input feature information is original point cloud information; Obtaining first neighboring point information of each point in the first input feature information; Calculate the relative coordinate information and Euclidean distance between each point and its neighboring point based on the first neighboring point information; Calculate the position encoding of each point in multiple dimensions according to the trigonometric function and the first neighbor point information; Obtaining coordinate information and second neighboring point information of each point in the second input feature information; Based on the first neighbor point information, relative coordinate information, Euclidean distance and position code of each point, construct local coordinate codes of multiple local blocks, each of which is composed of a point and corresponding neighbor points; Performing point-by-point feature transformation on each point based on the local coordinate code, the second neighboring point information and the coordinate information to obtain first transformed feature information, where the first transformed feature information includes an attention weight of a local block corresponding to each point; After concatenating the first transformed feature information and the first input feature information, activation function calculation and normalization processing are performed to obtain feature information of the local coordinate attention output; The specific steps of enhancing the position information through the position binding module and attention pooling are: Acquire third neighboring point information of each point in the third input feature information, where the third input feature information is feature information input by the position binding module; The local coordinate encoding is converted into feature dimensions by a shared multi-layer perceptron to obtain feature conversion information; Splicing the third neighbor point information and the feature conversion information to obtain spliced ​​neighbor point information, where the spliced ​​neighbor point information is feature information output by the position binding module; The splicing neighbor point information is calculated sequentially through linear transformation and normalized exponential function to obtain second transformation feature information; Performing a dot product operation and a sum operation on the second transformed feature information and the spliced ​​neighbor point information in sequence to obtain feature fusion information; The feature fusion information is expanded in feature dimension by a shared multi-layer perceptron to obtain enhanced feature information, where the enhanced feature information is feature information output by attention pooling; The feature decoding process is performed on the encoded information to obtain the decoded information based on multi-dimensional interpolation, splicing and transposed convolution operations, including: Constructing a decoder based on the encoder, the decoder comprising a plurality of decoding layers connected in sequence, each of the decoding layers corresponding one to ... Up-sampling the fourth input feature information of each decoding layer by nearest neighbor interpolation to obtain first decoding feature information of the decoding layer, wherein the fourth input feature information is feature information output by a previous decoding layer corresponding to the decoding layer; After concatenating the first decoding feature information of each decoding layer with the fifth input feature information of the decoding layer, a transposed convolution operation is performed to obtain the second decoding feature information of the decoding layer, where the second decoding feature information is the feature information output by the decoding layer, and the fifth input feature information is the encoding feature information output by the encoding layer corresponding to the decoding layer; The second decoding feature information of the last decoding layer is used as the decoding information output by the decoder.

3. A point cloud data processing device based on random sampling, characterized in that: include: Memory for storing computer programs; A processor is used to implement the steps of the point cloud data processing method based on random sampling as claimed in claim 1 when executing the computer program.

4. A readable storage medium, characterized in that: The readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the point cloud data processing method based on random sampling as claimed in claim 1 are implemented.

Images