A method, apparatus, storage medium, and electronic device for processing point cloud data

Abstract

An embodiment of the present invention discloses a method, device, storage medium, and electronic device for processing point cloud data. The method includes: obtaining point cloud data to be processed; for the data points to be transformed, transforming the local point cloud data where the data points are located into a local orthogonal coordinate system, and determining local rotation-invariant data, and transforming the point cloud data of the data points into a global orthogonal coordinate system, and determining global rotation-invariant data; performing fusion processing based on the local rotation-invariant data and the global rotation-invariant data to obtain target rotation-invariant data corresponding to the point cloud data to be processed. By performing local rotation-invariant processing on the local point cloud data where each processed data point is located, it is ensured that the processed data is not affected by local rotation. By performing global rotation-invariant processing on all processed data points, it is ensured that the processed data is not affected by global rotation.

Description

Technical Field

[0001] The embodiments of the present invention relate to the technical field of data processing, and in particular, to a method, apparatus, storage medium, and electronic device for processing point cloud data. Background Art

[0002] Three-dimensional point cloud data is very sensitive to geometric transformations. In particular, the impact of geometric transformations such as rotation on point cloud data cannot be eliminated through standardization operations.

[0003] Currently, common processing of point cloud data includes processing point cloud data through a machine learning model. The rotation transformation of point cloud data affects the processing accuracy of the machine learning model. Summary of the Invention

[0004] The embodiments of the present invention provide a method, apparatus, storage medium, and electronic device for processing point cloud data to achieve high-precision rotation-invariant processing of point cloud data.

[0005] In a first aspect, the embodiments of the present invention provide a method for processing point cloud data, including:

[0006] Obtaining point cloud data to be processed;

[0007] For the data points to be transformed, converting the local point cloud data where the data points are located into a local orthogonal coordinate system, and determining local rotation-invariant data, and converting the point cloud data of the data points into a global orthogonal coordinate system, and determining global rotation-invariant data;

[0008] Performing fusion processing based on the local rotation-invariant data and the global rotation-invariant data to obtain target rotation-invariant data corresponding to the point cloud data to be processed.

[0009] In a second aspect, the embodiments of the present invention further provide a device for processing point cloud data, including:

[0010] A point cloud data acquisition module for obtaining point cloud data to be processed;

[0011] A local conversion module for, for the data points to be transformed, converting the local point cloud data where the data points are located into a local orthogonal coordinate system, and determining local rotation-invariant data;

[0012] A global conversion module for, for the data points to be transformed, converting the point cloud data of the data points into a global orthogonal coordinate system, and determining global rotation-invariant data;

[0013] A data fusion module for performing fusion processing based on the local rotation-invariant data and the global rotation-invariant data to obtain target rotation-invariant data corresponding to the point cloud data to be processed.

[0014] In a third aspect, an embodiment of the present invention further provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the processing method of point cloud data provided in any embodiment of the present invention is implemented.

[0015] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the processing method of point cloud data provided in any embodiment of the present invention is implemented.

[0016] In the technical solution of this embodiment, by performing local rotation invariant processing on the local point cloud data where each data point to be processed is located, it is ensured that the processed data is not affected by local rotation. By performing global rotation invariant processing on all data points to be processed, it is ensured that the processed data is not affected by global rotation. Through data fusion of global rotation invariant data and local rotation invariant data, target rotation invariant data that can resist local rotation and is non-redundant and distributed in an orthogonal feature space is obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a schematic flowchart of a method for processing point cloud data provided by an embodiment of the present invention;

[0018] Figure 2 It is a schematic diagram of the point cloud data processing flow provided by an embodiment of the present invention;

[0019] Figure 3 It is a schematic flowchart of a method for processing point cloud data provided by an embodiment of the present invention;

[0020] Figure 4 It is a schematic flowchart of a method for processing point cloud data provided by an embodiment of the present invention;

[0021] Figure 5 It is a schematic structural diagram of a device for processing point cloud data provided by an embodiment of the present invention;

[0022] Figure 6 It is a schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0023] The present invention will be further described in detail below with reference to the drawings and embodiments. It can be understood that the specific embodiments described herein are only for explaining the present invention, rather than limiting the present invention. Additionally, it should be noted that for the sake of convenience of description, only parts related to the present invention are shown in the drawings rather than the entire structure.

[0024] Figure 1The figure is a schematic flowchart of a method for processing point cloud data provided by an embodiment of the present invention. This embodiment is applicable to the case of converting point cloud data into rotation-invariant features. The method can be executed by a point cloud data processing device provided by an embodiment of the present invention. The point cloud data processing device can be implemented by software and / or hardware and can be configured on an electronic computing device. The method specifically includes the following steps:

[0025] S110. Obtain the point cloud data to be processed.

[0026] S120. For the data points to be converted, convert the local point cloud data where the data points are located into a local orthogonal coordinate system, determine local rotation-invariant data, and convert the point cloud data of the data points into a global orthogonal coordinate system, and determine global rotation-invariant data.

[0027] S130. Perform fusion processing based on the local rotation-invariant data and the global rotation-invariant data to obtain the target rotation-invariant data corresponding to the point cloud data to be processed.

[0028] Point cloud data is a three-dimensional data set in the form of points, which can be obtained by scanning a target object with a three-dimensional scanning device. Exemplarily, the target object can be an object such as a table or a chair, a building such as a room, etc. Point cloud scanning can be performed on any target object as needed to obtain the point cloud data of the target object. The point cloud data includes a large number of data points, and each data point corresponds to coordinate information and feature information respectively. Among them, the coordinate information can be three-dimensional coordinate values used to represent the position of the data point, and the feature information includes but is not limited to color features and geometric features such as the normal vector of the object surface.

[0029] The processing of the point cloud data obtained by scanning the target object can include, but is not limited to, classification processing, segmentation processing, detection processing, etc. Among them, the above processing methods can be implemented by a machine learning model such as a neural network model.

[0030] It should be noted that for the point cloud data of the target object, when the target object is in different positions relative to the three-dimensional scanning device, the collected point cloud data is different. Exemplarily, when the target object is in the first position relative to the three-dimensional scanning device, the first point cloud data is collected, and when the target object is in the second position relative to the three-dimensional scanning device, the second point cloud data is collected. The data corresponding to the same position point of the target object in the first point cloud data and the second point cloud data is different. Specifically, the coordinate information is different and / or the feature information is different.

[0031] To avoid the influence of the rotation of point cloud data on subsequent processing, this embodiment provides a method for processing point cloud data. After obtaining the point cloud data, rotation-invariant data of the point cloud data is determined. This rotation-invariant data can be used as input information for subsequent processing (such as classification processing, segmentation processing, detection processing), eliminating the influence of point cloud data rotation on subsequent processing and improving the processing accuracy of subsequent processing.

[0032] The point cloud data obtained in this embodiment can be collected by a three-dimensional scanning device for a target object, or can be obtained by external import. The acquisition method of the point cloud data is not limited.

[0033] In some embodiments, all data points in the to-be-processed point cloud data obtained can be determined as data points to be transformed. The amount of data processed is large, and the accuracy of the obtained target rotation-invariant data is high. In some embodiments, local data points in the to-be-processed point cloud data can be determined as data points to be transformed. The amount of data processed is small, and the accuracy of the obtained target rotation-invariant data is relatively low. The data points to be transformed can be determined according to the processing efficiency requirement and the accuracy requirement.

[0034] Optionally, after obtaining the to-be-processed point cloud data, the method further includes downsampling the point cloud data to obtain data points to be transformed, that is, sampling points. By reducing the amount of data processed, the processing speed is improved. The downsampling method for the to-be-processed point cloud data can include, but is not limited to, the farthest point sampling algorithm (FarthestPointSampling, FPS), the uniform sampling algorithm, etc. Among them, the advantage of the farthest point sampling algorithm is that it can cover all points in space as much as possible, and the advantage of the uniform sampling algorithm is that the sampled data points are evenly distributed. The downsampling algorithm can be determined according to the sampling requirement, and this is not limited.

[0035] In some embodiments, the to-be-processed point cloud data can be sampled according to a preset sampling rule. Among them, the preset sampling rule can include a downsampling rate. For example, the number of data points obtained by downsampling is 1 / 4, 1 / 16, etc. of the total amount of original data points. In some embodiments, the to-be-processed point cloud data can be sampled according to the expected data volume. Among them, the expected data volume is the data volume of the data points obtained after downsampling processing. Exemplarily, the expected data volume can be determined according to the processing requirements of subsequent processing. Exemplarily, taking subsequent processing as classification processing as an example, the input data volume of the classification processing model is 256×256. Correspondingly, the expected data volume can be 256×256. Downsampling the to-be-processed point cloud data based on the expected data volume can reduce the amount of data processed while meeting the requirements of subsequent processing and reduce the process of secondary processing of the obtained target rotation-invariant data.

[0036] The acquired point cloud data to be processed is the original point cloud data obtained by a three-dimensional scanning device. This original point cloud data is located within a Cartesian coordinate system and is affected by the rotation of the object. In this embodiment, by constructing a rotation-invariant orthogonal coordinate system and converting the point cloud data into the rotation-invariant orthogonal coordinate system, target rotation-invariant data is obtained. Exemplarily, an orthogonal coordinate system can be constructed based on the coordinate information of the data points undergoing rotation and the principal component direction, the transformation matrix between the original coordinate system of the original data and the orthogonal coordinate system is determined, and each data point is transformed through the transformation matrix to obtain the target rotation-invariant data. Optionally, after acquiring the point cloud data to be processed and before performing the transformation operation on the point cloud data, preprocessing of the point cloud data to be processed is further included. This preprocessing is used to eliminate the influence of geometric operations such as translation and scaling on the point cloud data. Exemplarily, it can be achieved by invoking a preprocessing algorithm.

[0037] Based on the above embodiment, in order to avoid the influence of local rotation in the target object, the local point cloud data is subjected to rotation processing, and by fusing the local rotation processing result with the global rotation processing result, target rotation-invariant data that can resist local rotation and is non-redundant and distributed in the orthogonal feature space is obtained.

[0038] In this embodiment, the transformation of the local point cloud data and the global point cloud data can be executed synchronously or can be executed based on any order, and this is not limited. Among them, there are multiple sets of local point cloud data, and each data point undergoing rotation corresponds to a set of local point cloud data respectively. The local point cloud data can be the point cloud data of the data points within the neighborhood range centered on the data point undergoing rotation (such as the sampling point). Among them, the neighborhood range can be a range centered on the sampling point with a radius of r, and r can be set according to requirements. For example, it can be 0.5 m or 1 m, etc. All the data points within the neighborhood range can be determined as local data points to obtain the local point cloud data. It can also be to determine a preset number of data points within the neighborhood range as local data points. Exemplarily, the preset number can be 32. For example, randomly sample a preset number of data points within the neighborhood range, or, in the case where the number of data points within the neighborhood range is less than the preset number, repeat the sampling to obtain the local data points.

[0039] For each set of local point cloud data, transformation is performed to obtain local rotation-invariant data. Optionally, converting the local point cloud data where the data point is located into the local orthogonal coordinate system includes: constructing the local orthogonal coordinate system corresponding to the data point based on the local point cloud data where the data point is located; performing data transformation on the local point cloud data where the data point is located based on the first transformation matrix corresponding to the local orthogonal coordinate system.

[0040] The data points in this embodiment are the data points to be rotated, i.e., the sampling points. For the local point cloud data where each sampling point is located, a local orthogonal coordinate system is respectively constructed, and the local orthogonal coordinate system is determined according to the principal component direction and the coordinate information of the local data points. Optionally, the construction method of the local orthogonal coordinate system includes: for the data points to be transformed, constructing the first direction of the local orthogonal coordinate system based on the coordinate information of the data points and the origin coordinate information, constructing the second direction of the local orthogonal coordinate system based on the first direction of the local orthogonal coordinate system and the principal component direction of the point cloud data to be processed, and constructing the third direction of the local orthogonal coordinate system based on the first direction and the second direction of the local orthogonal coordinate system, so as to form the local orthogonal coordinate system corresponding to the data points. Among them, the principal component direction can be obtained by performing principal component analysis on the point cloud data to be processed based on the principal component analysis algorithm (Principal Component Analysis, PCA), and the principal component direction can be the direction of the first principal component obtained by analyzing the point cloud data to be processed.

[0041] Determine the first direction of the local orthogonal coordinate system based on the data origin in the point cloud data to be processed and the current sampling point. Among them, the data origin can be the centroid position of each data point in the point cloud data to be processed, and can be obtained by performing a mean calculation based on the coordinates of all data points in the point cloud data to be processed. Obtain the second direction of the local orthogonal coordinate system based on the cross product of the unit vector of the first direction and the unit vector of the principal component direction, and obtain the third direction of the local orthogonal coordinate system based on the cross product of the unit vector of the first direction and the unit vector of the second direction. Exemplarily, the first direction can be the x-axis direction, the second direction can be the y-axis direction, and the third direction can be the z-axis direction to obtain the local orthogonal coordinate system, and the local orthogonal coordinate system is not affected by the rotation of the local data points.

[0042] Construct the first transformation matrix corresponding to the above local orthogonal coordinate system, and transform each data point in the local point cloud data to the local orthogonal coordinate system based on the first transformation matrix. Each local orthogonal coordinate system corresponds to a first transformation matrix, and the first transformation matrix is determined by the unit vectors in the respective coordinate axis directions of the corresponding local orthogonal coordinate system. It should be noted that the point cloud data of each data point includes coordinate data and feature data. Among them, the feature data includes geometric feature data and non-geometric feature data. The geometric feature data is feature data related to coordinate information, such as, but not limited to, the normal vector of the object surface, etc. The non-geometric feature information is feature information unrelated to coordinate information, such as color, etc. Correspondingly, transforming each data point in the local rotation-invariant data to the local orthogonal coordinate system includes transforming the coordinate data in the local point cloud data to the local orthogonal coordinate system based on the first transformation matrix, and transforming the geometric feature data in the local point cloud data to the local orthogonal coordinate system based on the first transformation matrix. The point cloud data transformed to the local orthogonal coordinate system is the local rotation-invariant data.

[0043] Perform a global transformation on the data points to be transformed, that is, the sampling points, to obtain globally rotation-invariant data. Optionally, transforming the point cloud data of the data points into a global orthogonal coordinate system to determine globally rotation-invariant data includes: constructing a global orthogonal coordinate system based on the point cloud data of the data points; performing data transformation on the point cloud data based on the second transformation matrix corresponding to the global orthogonal coordinate system, where the data transformation includes the transformation of coordinate information and the transformation of feature information.

[0044] Specifically, construct a global orthogonal coordinate system based on all the sampling points. The global orthogonal coordinate system is determined according to the principal component direction and the coordinate information of all the sampling points. Optionally, the construction method of the global orthogonal coordinate system includes: constructing the first direction of the global orthogonal coordinate system based on the first direction of the local orthogonal coordinate system of each data point to be transformed; based on the first direction of the global orthogonal coordinate system, constructing the second direction of the global orthogonal coordinate system based on the first direction of the global orthogonal coordinate system and the principal component direction of the point cloud data to be processed; and constructing the third direction of the global orthogonal coordinate system based on the second direction and the first direction of the global orthogonal coordinate system to form the global orthogonal coordinate system.

[0045] Exemplarily, the first direction of the global orthogonal coordinate system is determined based on the mean value of the first direction in the local orthogonal coordinate systems of all sampling points. For example, the first direction can be the x-axis direction. The X-axes in the local orthogonal coordinate systems of all sampling points are obtained, and the mean value of the X-axes of the local orthogonal coordinate systems of each sampling point is used to determine the axis coordinate of the first direction of the global orthogonal coordinate system. The second direction of the global orthogonal coordinate system is obtained based on the cross product of the unit vector of the first direction and the unit vector of the principal component direction, and the third direction of the global orthogonal coordinate system is obtained based on the cross product of the unit vector of the first direction and the unit vector of the second direction. Exemplarily, the second direction can be the y-axis direction, and the third direction can be the z-axis direction, thus obtaining the global orthogonal coordinate system.

[0046] The second transformation matrix corresponding to the global orthogonal coordinate system is determined, and based on the second transformation matrix, the point cloud data of all sampling points is transformed into the global orthogonal coordinate system. Specifically, the coordinate data in the point cloud data of all sampling points is transformed into the global orthogonal coordinate system based on the second transformation matrix, and the feature data in the point cloud data of all sampling points is transformed into the local orthogonal coordinate system based on the second transformation matrix. The point cloud data transformed into the global orthogonal coordinate system is the globally rotation-invariant data.

[0047] Based on the above embodiments, the local rotation-invariant data and the global rotation-invariant data are fused to obtain the target rotation-invariant data corresponding to the point cloud data to be processed. This target rotation-invariant data is not affected by global rotation or local rotation, eliminating the influence of the rotation of the target object or the three-dimensional scanning device on the point cloud data during the acquisition process, facilitating the subsequent processing of the point cloud data.

[0048] Optionally, fusing the local rotation-invariant data and the global rotation-invariant data to obtain the target rotation-invariant data corresponding to the point cloud data to be processed includes: determining the feature weights of each dimension based on the global rotation-invariant data; processing the local rotation-invariant data based on the feature weights to obtain the target rotation-invariant data corresponding to the point cloud data to be processed.

[0049] In this embodiment, the global rotation-invariant data can be processed based on the softmax function to obtain the feature weights of each dimension, the local rotation-invariant data is weighted based on the feature weights of each dimension, and the weighted local rotation-invariant data is superimposed on the original local rotation-invariant data to obtain the target rotation-invariant data.

[0050] The feature information included in the target rotation-invariant data obtained by the above processing method is the original feature information that is not affected by rotation. In the subsequent processing, it is necessary to extract the features of the target rotation-invariant data and then determine the processing result based on the extracted feature data. To improve the processing efficiency of subsequent processing and simplify the subsequent processing process, on the basis of the above embodiments, after each data point in the local point cloud data is transformed into the local orthogonal coordinate system based on the first transformation matrix, it further includes extracting the feature information of the local rotation-invariant data to form local point cloud feature data. Specifically: splicing the coordinate information and feature information of the data points in the local orthogonal coordinate system to obtain the initial local feature; inputting the initial local feature into the local feature extraction model to obtain the local rotation-invariant data output by the local feature extraction model. Among them, the local feature extraction model can be a neural network model, which is used to extract features from the initial local feature to obtain local point cloud feature data. Exemplarily, the local feature extraction model can be obtained by connecting a multi-layer perceptron (MLP) and a max pooling layer.

[0051] Correspondingly, after the feature data in the point cloud data of all sampling points is transformed into the local orthogonal coordinate system based on the second transformation matrix, it further includes: extracting the feature information of the global rotation-invariant data to obtain global point cloud feature data. Specifically, inputting the initial global feature obtained by splicing the coordinate information and feature information of the data points in the global orthogonal coordinate system into the global transformation model to obtain the global point cloud feature data output by the global transformation model. The global feature extraction model can be a neural network model, which is used to extract features from the initial global feature. Exemplarily, the global feature extraction model can include a multi-layer perceptron.

[0052] Correspondingly, performing a fusion process based on the local rotation-invariant data and the global rotation-invariant data to obtain the target rotation-invariant data corresponding to the point cloud data to be processed, including: performing a fusion process based on the local point cloud feature data and the global point cloud feature data to obtain the target rotation-invariant data corresponding to the point cloud data to be processed.

[0053] Exemplarily, see Figure 2 , Figure 2It is a schematic diagram of the point cloud data processing flow provided by an embodiment of the present invention. In this embodiment, the point cloud data to be processed is subjected to sampling processing, local rotation invariant feature construction, global rotation invariant feature construction, and global-local feature fusion. Correspondingly, the processing model for processing the point cloud data to be processed includes the following structures, namely, a sampling module, a local rotation invariant feature construction module, a global rotation invariant feature construction module, and a global-local feature fusion module. Among them, the sampling module is used to downsample the point cloud data to be processed to obtain the data points to be transformed, that is, the sampling points. The local rotation invariant feature construction module is used to perform local transformation on the local point cloud data corresponding to each sampling point to obtain the local rotation invariant data that is locally rotation invariant. The global rotation invariant feature construction module is used to perform global transformation on all sampling points to obtain the global rotation invariant data that is globally rotation invariant. The global-local feature fusion module is used to fuse the global rotation invariant data and the local rotation invariant data to obtain the target rotation invariant data with high precision.

[0054] Based on the above embodiment, sampling can be performed on the target rotation invariant data according to the processing requirements of subsequent processing (such as data volume requirements) to obtain the rotation invariant data that meets the processing requirements of subsequent processing.

[0055] The technical solution of this embodiment ensures that the processed data is not affected by local rotation by performing local rotation invariant processing on the local point cloud data where each data point to be processed is located, and ensures that the processed data is not affected by global rotation by performing global rotation invariant processing on all data points to be processed. By fusing the global rotation invariant data and the local rotation invariant data, the target rotation invariant data that can resist local rotation and is non-redundant and distributed in the orthogonal feature space is obtained.

[0056] Based on the above technical solution, multi-layer transformation processing is performed on the point cloud data to be processed. Among them, each layer of transformation processing is used to downsample the point cloud data obtained from the previous processing, and based on the target rotation invariant data obtained from the previous layer, determine the local rotation invariant data, global rotation invariant data of the downsampled data points, and fuse the local rotation invariant data and the global rotation invariant data of the current layer to obtain the target rotation invariant data of the current layer.

[0057] Exemplarily, refer to Figure 3 , Figure 3 It is a schematic diagram of the process of a point cloud data processing provided by an embodiment of the present invention. Figure 3In this case, multi-layer conversion processing is performed on the point cloud data. The conversion processing of each layer uses the output data of the previous layer as the processing data. Specifically, the processing process of the conversion processing of each layer is the processing process of the point cloud data provided in the above embodiment. Specifically, the first-layer conversion processing uses the original point cloud data as the processing data, performs downsampling processing on the original point cloud data to obtain the data points of the first-layer conversion processing, that is, the sampling points P1. Determine the local point cloud data of each sampling point P1 in the original point cloud data, perform conversion processing with local rotational invariance to obtain locally rotation-invariant data; perform conversion processing with global rotational invariance on the sampling points P1 to obtain globally rotation-invariant data, and fuse the locally rotation-invariant data and the globally rotation-invariant data to obtain the target rotation-invariant data corresponding to the first-layer conversion processing.

[0058] The second-layer conversion processing uses the target rotation-invariant data output by the first-layer conversion processing as the input data for processing, and so on until the target rotation-invariant data of the last layer is output. The number of layers of the conversion processing can be preset and is not limited in this regard. By setting multi-layer conversion processing, the accuracy of the rotation-invariant data is improved.

[0059] Based on the above embodiment, the embodiment of the present invention also provides a preferred example of a method for processing point cloud data. Specifically, refer to Figure 3 , perform multi-layer conversion processing on the point cloud data. In each layer of conversion processing: eliminate the influence of translation and scaling on the point cloud through a normalization operation, and obtain the direction of the first principal component through the principal component analysis algorithm (PCA), denoted as the vector a (here a is a unit-length vector). For the sampling point p i , use the vector from the origin to p i (i.e., the vector where o is the coordinate origin) as the x-axis. Obtain the y-axis through the cross product of the x-axis unit vector x i and a, and obtain the z-axis through the cross product of x i and the y-axis unit vector y i . That is: y i = x i × a, z i = x i × y i .

[0060] For the data point where, is the neighborhood of p i , convert it to the coordinate system constructed above: p' ij = T i p ij = T i (p j - p i )

[0061] where T i is the projection matrix, i.e., the first transformation matrix:

[0062]

[0063] where the subscript i indicates that the rotation-invariant coordinate system and the projection matrix vary with different sampling points. The point cloud data contains not only the three-dimensional coordinates x, y, and z, but also some geometric-related features, such as the normal vector of the object surface and other feature information. Denote such geometric features as F (n) , and such feature information also needs to be projected into the constructed local coordinate system to ensure rotation invariance:

[0064]

[0065] The construction of the global feature is equivalent to considering all sampling points as in the neighborhood of the origin. Take the average value of the x-axis of all sampling points to obtain the x-axis of the global rotation-invariant coordinate system, i.e.:

[0066]

[0067] where n l is the number of sampling points in the l-th layer of the transformation process. The determination methods of the y-axis and z-axis of the global rotation-invariant coordinate system: y (glb) = x (glb) × a, z (glb) = x (glb) × y (glb)

[0068] On the basis of completing the construction of the local rotation-invariant coordinate system (i.e., the local orthogonal coordinate system) and the global rotation-invariant coordinate system (i.e., the global orthogonal coordinate system), perform transformation processing on the sampling points of the current layer. Perform query and grouping processing (Rotation-Invariant Query and Group) on the point cloud data output from the previous layer. Specifically, at the -th layer, sample sampling points (query points) through furthest point sampling For each sampling point, aggregate neighboring points within the radius , denoted as and its corresponding features , that is, the local point cloud data. Convert into the above local rotation-invariant coordinate system, that is, perform coordinate transformation on coordinate data:

[0069] Convert and are spliced together, where is the target rotation-invariant data output by the l-1 layer, and the local features are scheduled through the local feature extraction module (i.e., an MLP and a MaxPool) that is, the local rotation-invariant data features:

[0070]

[0071] All the sampling points are subjected to global rotation-invariant transformation processing (Global Rotation-Invariant Projection) and converted into global rotation-invariant coordinates. The global features of the layer can be expressed as:

[0072]

[0073] where T (gbl) = [x (glb) y (glb) z (glb) T is the global rotation-invariant projection matrix, that is, the second transformation matrix, is taken from the layer features the layer sampling points (query points) corresponding features.

[0074] Exemplarily, see Figure 4 , Figure 4 is a schematic diagram of a local transformation and a global transformation provided by an embodiment of the present invention.

[0075] The local rotation-invariant data and the global rotation-invariant data are fused to obtain the target rotation-invariant data:

[0076] where and respectively represent element-wise addition and multiplication. During the fusion process, is equivalent to calculating a weight coefficient along the "point dimension" for each dimension of the feature.

[0077] Based on the above embodiments, an embodiment of the present invention further provides a point cloud data processing device. See Figure 5 , Figure 5 is a schematic structural diagram of a point cloud data processing device further provided by an embodiment of the invention. The device includes:

[0078] A point cloud data acquisition module 210, configured to acquire point cloud data to be processed;

[0079] ​The local transformation module 220 is configured to transform the local point cloud data where the data point is located into a local orthogonal coordinate system for the data points to be transformed, and determine local rotation-invariant data;

[0080] The global transformation module 230 is configured to transform the point cloud data of the data points into a global orthogonal coordinate system for the data points to be transformed, and determine global rotation-invariant data;

[0081] The data fusion module 240 is configured to perform fusion processing based on the local rotation-invariant data and the global rotation-invariant data to obtain the target rotation-invariant data corresponding to the point cloud data to be processed.

[0082] Based on the above embodiments, optionally, the data points to be transformed are local data points in the point cloud data to be processed;

[0083] The apparatus further includes a sampling module configured to perform downsampling on the point cloud data after obtaining the point cloud data to be processed to obtain the data points to be transformed.

[0084] Based on the above embodiments, optionally, the local transformation module 220 is configured to:

[0085] Construct a local orthogonal coordinate system corresponding to the data point based on the local rotation-invariant data where the data point is located;

[0086] Perform data transformation on the local rotation-invariant data where the data point is located based on the first transformation matrix corresponding to the local orthogonal coordinate system.

[0087] Based on the above embodiments, optionally, the global transformation module 230 is configured to:

[0088] Construct a global orthogonal coordinate system based on the point cloud data of the data point; perform data transformation on the point cloud data based on the second transformation matrix corresponding to the global orthogonal coordinate system, where the data transformation includes transformation of coordinate information and transformation of feature information.

[0089] Optionally, the construction method of the local orthogonal coordinate system includes:

[0090] For the data points to be transformed, construct the first direction of the local orthogonal coordinate system based on the coordinate information of the data point and the origin coordinate information, construct the second direction of the local orthogonal coordinate system based on the first direction of the local orthogonal coordinate system and the principal component direction of the point cloud data to be processed, and construct the third direction of the local orthogonal coordinate system based on the first direction and the second direction of the local orthogonal coordinate system to form the local orthogonal coordinate system corresponding to the data point;

[0091] The construction method of the global orthogonal coordinate system includes:

[0092] Construct the first direction of the global orthogonal coordinate system based on the first direction of the local orthogonal coordinate system of each data point to be transformed. Based on the first direction of the global orthogonal coordinate system, construct the second direction of the global orthogonal coordinate system based on the first direction of the global orthogonal coordinate system and the principal component direction of the point cloud data to be processed. And, construct the third direction of the global orthogonal coordinate system based on the second direction and the first direction of the global orthogonal coordinate system to form the global orthogonal coordinate system.

[0093] On the basis of the above embodiments, optionally, the data fusion module 240 is configured to:

[0094] Determine the feature weights of each dimension based on the globally rotation-invariant data;

[0095] Process the locally rotation-invariant data based on the feature weights to obtain the target rotation-invariant data corresponding to the point cloud data to be processed.

[0096] On the basis of the above embodiments, optionally,

[0097] Perform multi-layer transformation processing on the point cloud data to be processed. Among them, each layer of transformation processing is used to downsample the point cloud data obtained from the previous processing, and determine the locally rotation-invariant data, globally rotation-invariant data of the downsampled data points based on the target rotation-invariant data obtained in the previous layer, and perform data fusion on the locally rotation-invariant data and the globally rotation-invariant data of the current layer to obtain the target rotation-invariant data of the current layer.

[0098] The point cloud data processing device provided by the embodiments of the present invention can execute the point cloud data processing method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the point cloud data processing method.

[0099] Figure 6 It is a schematic structural diagram of an electronic device provided in the fourth embodiment of the present invention. Figure 6 It shows a block diagram of an electronic device 12 suitable for implementing the embodiments of the present invention. Figure 6 The shown electronic device 12 is only an example and should not impose any limitation on the functions and usage scope of the embodiments of the present invention. The device 12 is typically an electronic device that undertakes the function of image classification.

[0100] As Figure 6 shown, the electronic device 12 is presented in the form of a general-purpose computing device. The components of the electronic device 12 may include, but are not limited to: one or more processors 16, a storage device 28, and a bus 18 connecting different system components (including the storage device 28 and the processor 16).

[0101] Bus 18 represents one or more of several types of bus architectures, including a memory bus or memory controller, a peripheral bus, an Accelerated Graphics Port, a processor, or a local bus using any of the various bus architectures. By way of example, these architectures include, but are not limited to, Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.

[0102] Electronic device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by electronic device 12, including both volatile and nonvolatile media, removable and non-removable media.

[0103] Storage device 28 may include computer system readable media in the form of volatile memory, such as Random Access Memory (RAM) 30 and / or cache memory 32. Electronic device 12 may further include other removable / non-removable, volatile / nonvolatile computer system storage media. By way of example only, storage system 34 can be used for reading and writing on non-removable, nonvolatile magnetic media ( Figure 6 not shown, typically referred to as a "hard disk drive"). Although Figure 6 not shown in the figure, a disk drive for reading and writing on a removable nonvolatile disk (such as a "floppy disk"), and an optical disk drive for reading and writing on a removable nonvolatile optical disk (such as a Compact Disc-Read Only Memory (CD-ROM), Digital Video Disc-Read Only Memory (DVD-ROM), or other optical media) may be provided. In these cases, each drive may be connected to bus 18 via one or more data media interfaces. Storage device 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the embodiments of the present invention.

[0104] A program 36 having a set (at least one) of program modules 26 can be stored, for example, in a storage device 28. Such program modules 26 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of the gateway environment. The program modules 26 generally execute the functions and / or methods in the embodiments described in the present invention.

[0105] The electronic device 12 can also communicate with one or more external devices 14 (such as a keyboard, a pointing device, a camera, a display 24, etc.), and can also communicate with one or more devices that enable a user to interact with the electronic device 12, and / or communicate with any device that enables the electronic device 12 to communicate with one or more other computing devices (such as a network card, a modem, etc.). Such communication can be carried out through an input / output (I / O) interface 22. Moreover, the electronic device 12 can also communicate with one or more gateways (such as a Local Area Network (LAN), a Wide Area Network (WAN), and / or a public gateway, such as the Internet) through a gateway adapter 20. As shown in the figure, the gateway adapter 20 communicates with other modules of the electronic device 12 through a bus 18. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in combination with the electronic device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, Redundant Arrays of Independent Disks (RAID) systems, tape drives, and data backup storage systems, etc.

[0106] The processor 16 executes various functional applications and data processing by running the program stored in the storage device 28, such as implementing the method for processing point cloud data provided in the above embodiments of the present invention.

[0107] Embodiment 5 of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, it implements the method for processing point cloud data provided in the embodiments of the present invention.

[0108] Certainly, the computer program stored on the computer-readable storage medium provided in the embodiments of the present invention is not limited to the method operations as described above, and can also execute the method for processing point cloud data provided in any embodiment of the present invention.

[0109] The computer storage medium of the embodiments of the present invention may adopt any combination of one or more computer-readable media. The computer-readable media may be computer-readable signal media or computer-readable storage media. The computer-readable storage media may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples (non-exhaustive list) of the computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, the computer-readable storage media may be any tangible medium that contains or stores a program, and this program can be used by or in combination with an instruction execution system, apparatus, or device.

[0110] The computer-readable signal media may include data signals propagated in a baseband or as part of a carrier wave, in which computer-readable source code is carried. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal media may also be any computer-readable media other than the computer-readable storage media, and this computer-readable media can send, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device.

[0111] The source code contained on the computer-readable media can be transmitted by any appropriate medium, including but not limited to wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0112] The computer source code for performing the operations of the present invention can be written in one or more programming languages or combinations thereof. The programming languages include object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The source code can be executed entirely on the user's computer, partially on the user's computer, executed as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of gateway, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, by using an Internet service provider to connect through the Internet).

[0113] Note that the above is only the preferred embodiment of the present invention and the technical principles applied. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, re-adjustments, and substitutions can be made by those skilled in the art without departing from the protection scope of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments only. Without departing from the concept of the present invention, more other equivalent embodiments can be included, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A method for processing point cloud data, characterized in that, Including: Obtain the point cloud data to be processed; perform multi-layer transformation processing on the point cloud data to be processed; For each layer of transformation processing, each layer of transformation processing is used to downsample the point cloud data obtained from the previous processing, and determine the local rotation invariant data and global rotation invariant data of the downsampled data points based on the target rotation invariant data obtained from the previous layer; Among them, for the data points corresponding to each layer of transformation, convert the local point cloud data where the data points are located into a local orthogonal coordinate system, and determine the local rotation invariant data, and convert the point cloud data of the data points into a global orthogonal coordinate system, and determine the global rotation invariant data; Perform fusion processing based on the local rotation invariant data and the global rotation invariant data of the current layer to obtain the target rotation invariant data of the current layer until the target rotation invariant data corresponding to the point cloud data to be processed is obtained; Among them, the local rotation invariant data is extracted from the initial local features spliced from the coordinate information and feature information of the data points in the local orthogonal coordinate system; among them, the coordinate data and feature information of the data points in the local orthogonal coordinate system are respectively obtained by converting the coordinate information and feature information in the local point cloud data through a first transformation matrix; The global rotation invariant data is extracted from the initial global features spliced from the coordinate information and feature information of the data points in the global orthogonal coordinate system, and the coordinate information and feature information of the data points in the global orthogonal coordinate system are respectively obtained by converting the coordinate information and feature information of the data points to be transformed through a second transformation matrix.

2. The method according to claim 1, characterized in that, The data points to be transformed are the local data points in the point cloud data to be processed; After obtaining the point cloud data to be processed, the method further includes downsampling the point cloud data to obtain the data points to be transformed.

3. The method according to claim 1, characterized in that, The conversion of the local point cloud data where the data points are located into a local orthogonal coordinate system includes: Based on the local point cloud data where the data points are located, construct the local orthogonal coordinate system corresponding to the data points; Perform data conversion on the local point cloud data where the data points are located based on the first transformation matrix corresponding to the local orthogonal coordinate system.

4. The method according to claim 1, characterized in that, The conversion of the point cloud data of the data points into a global orthogonal coordinate system and the determination of the global rotation invariant data include: Construct a global orthogonal coordinate system based on the point cloud data of the data points; perform data conversion on the point cloud data based on the second transformation matrix corresponding to the global orthogonal coordinate system, where the data conversion includes the conversion of coordinate information and the conversion of feature information.

5. The method according to claim 3 or 4, characterized in that, The construction method of the local orthogonal coordinate system includes: For the data points to be transformed, construct the first direction of the local orthogonal coordinate system based on the coordinate information of the data points and the origin coordinate information, construct the second direction of the local orthogonal coordinate system based on the first direction of the local orthogonal coordinate system and the principal component direction of the point cloud data to be processed, and construct the third direction of the local orthogonal coordinate system based on the first direction and the second direction of the local orthogonal coordinate system to form the local orthogonal coordinate system corresponding to the data points; The construction method of the global orthogonal coordinate system includes: Construct the first direction of the global orthogonal coordinate system based on the first direction of the local orthogonal coordinate system of each data point to be transformed. Based on the first direction of the global orthogonal coordinate system, construct the second direction of the global orthogonal coordinate system based on the first direction of the global orthogonal coordinate system and the principal component direction of the point cloud data to be processed. And, construct the third direction of the global orthogonal coordinate system based on the second direction and the first direction of the global orthogonal coordinate system to form the global orthogonal coordinate system.

6. The method according to claim 1, characterized in that, The fusion process based on the local rotation-invariant data and the global rotation-invariant data to obtain the target rotation-invariant data corresponding to the point cloud data to be processed includes: Determine the feature weights for each dimension based on the global rotation-invariant data; Process the local rotation-invariant data based on the feature weights to obtain the target rotation-invariant data corresponding to the point cloud data to be processed.

7. A device for processing point cloud data, characterized in that, It includes: A point cloud data acquisition module for acquiring the point cloud data to be processed; Perform multi-layer transformation processing on the point cloud data to be processed; For each layer of transformation processing, each layer of transformation processing is used to downsample the point cloud data obtained from the previous processing, and determine the local rotation-invariant data and the global rotation-invariant data of the data points to be downsampled based on the target rotation-invariant data obtained from the previous layer; A local transformation module for, corresponding to each data point to be transformed in each layer, transform the local point cloud data where the data point is located into the local orthogonal coordinate system and determine the local rotation-invariant data; A global transformation module for, corresponding to each data point to be transformed in each layer, transform the point cloud data of the data point into the global orthogonal coordinate system and determine the global rotation-invariant data; A data fusion module for performing a fusion process based on the local rotation-invariant data and the global rotation-invariant data of the current layer to obtain the target rotation-invariant data of the current layer until the target rotation-invariant data corresponding to the point cloud data to be processed is obtained; Among them, the local rotation-invariant data is extracted from the initial local features obtained by splicing the coordinate information and the feature information of the data points in the local orthogonal coordinate system; among them, the coordinate data and the feature information of the data points in the local orthogonal coordinate system are respectively obtained by transforming the coordinate information and the feature information in the local point cloud data through a first transformation matrix; The global rotation-invariant data is extracted from the initial global features obtained by splicing the coordinate information and the feature information of the data points in the global orthogonal coordinate system, and the coordinate information and the feature information of the data points in the global orthogonal coordinate system are respectively obtained by transforming the coordinate information and the feature information of the data points to be transformed through a second transformation matrix.

8. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method for processing point cloud data as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method for processing point cloud data as described in any one of claims 1-6.

Images