Processing method and device of three-dimensional point cloud data and unmanned vehicle

Abstract

The present disclosure relates to a method and device for processing three-dimensional point cloud data and an unmanned vehicle. The method comprises: obtaining three-dimensional point cloud data; determining a first relative position encoding and a second relative position encoding of the three-dimensional point cloud data according to a first encoding mode and a second encoding mode, wherein the first encoding mode is a relative position encoding mode when a target rotates relative to a collection device during collection of the three-dimensional point cloud data, and the second encoding mode is a relative position encoding mode when the target does not change position relative to the collection device during collection of the three-dimensional point cloud data; determining a weight of the first encoding mode and a weight of the second encoding mode corresponding to each point in the three-dimensional point cloud data based on an attention mechanism; and extracting a feature of the three-dimensional point cloud data according to the weight of the first encoding mode and the weight of the second encoding mode corresponding to each point in the three-dimensional point cloud data, and the first relative position encoding and the second relative position encoding of the three-dimensional point cloud data.

Description

Technical Field

[0001] This disclosure relates to the fields of artificial intelligence and computer technology, and in particular to the field of autonomous driving, specifically to a method, apparatus and unmanned vehicle for processing three-dimensional point cloud data. Background Technology

[0002] Currently, autonomous vehicles are used to automatically transport people or goods from one location to another. These vehicles collect environmental information through onboard sensors and complete the automated transport. Autonomous delivery vehicles controlled by autonomous driving technology greatly improve the convenience of production and daily life, and save labor costs.

[0003] In the field of autonomous driving, feature extraction from 3D point cloud data is a fundamental task. Deep neural networks have recently achieved widespread success in 3D point cloud analysis, where the input to the neural network is a series of 3D coordinate points obtained from a 3D camera or a 3D model. After extracting the features of the 3D coordinate points through the neural network, they can be used for downstream tasks such as classification or segmentation. Summary of the Invention

[0004] The inventors discovered that current feature extraction methods for 3D point cloud data have low rotation robustness. This means that when local or global rotation occurs in the 3D point cloud data, significant errors occur in the analysis of the point cloud data content, leading to inaccurate feature extraction and consequently, inaccurate downstream tasks. Considering that the shooting angle of objects often changes in practical applications, this instability related to rotation greatly affects the accuracy of feature extraction from 3D point cloud data.

[0005] One of the technical problems this disclosure aims to solve is: how to improve the accuracy of feature extraction from 3D point cloud data.

[0006] According to some embodiments of this disclosure, a method for processing three-dimensional point cloud data is provided, comprising: acquiring three-dimensional point cloud data; determining a first relative position code and a second relative position code of the three-dimensional point cloud data according to a first encoding method and a second encoding method, wherein the first encoding method is a relative position code when the target rotates relative to the acquisition device during the acquisition of the three-dimensional point cloud data, and the second encoding method is a relative position code when the target's position relative to the acquisition device remains unchanged during the acquisition of the three-dimensional point cloud data; determining the weight of the first encoding method and the weight of the second encoding method corresponding to each point in the three-dimensional point cloud data based on an attention mechanism; and extracting features from the three-dimensional point cloud data according to the weight of the first encoding method and the weight of the second encoding method corresponding to each point in the three-dimensional point cloud data, as well as the first relative position code and the second relative position code of the three-dimensional point cloud data.

[0007] In some embodiments, determining the first relative position code and the second relative position code of the three-dimensional point cloud data according to the first encoding method and the second encoding method includes: inputting the three-dimensional point cloud data into a feature extraction model, wherein the feature extraction model includes one or more cascaded position encoding modules; and in each position encoding module, determining the first relative position code and the second relative position code of the three-dimensional point cloud data in that position encoding module according to the input features of that position encoding module, the first encoding method, and the second encoding method.

[0008] In some embodiments, determining the weights of the first encoding method and the second encoding method corresponding to each point in the 3D point cloud data based on the attention mechanism includes: in each position encoding module, determining the weights of the first encoding method and the second encoding method corresponding to each point in the 3D point cloud data in that position encoding module based on the attention mechanism; extracting features from the 3D point cloud data includes: in each position encoding module, determining the output features of that position encoding module based on the weights of the first encoding method and the second encoding method corresponding to each point in the 3D point cloud data in that position encoding module, and the first relative position encoding and the second relative position encoding of the 3D point cloud data; and determining the features of the 3D point cloud data based on the output features of the last position encoding module.

[0009] In some embodiments, the first encoding method includes at least one of a Z-axis rotation-invariant encoding method and an arbitrary axis rotation-invariant encoding method. The Z-axis rotation-invariant encoding method is a relative position encoding method when the target rotates relative to the acquisition device around the Z-axis during the acquisition of 3D point cloud data. The arbitrary axis rotation-invariant encoding method is a relative position encoding method when the target rotates relative to the acquisition device around any axis during the acquisition of 3D point cloud data. The first relative position encoding includes at least one of a Z-axis rotation position encoding method and an arbitrary axis rotation position encoding method. The weight of the first encoding method includes at least one of the weight of the Z-axis rotation-invariant encoding method and the weight of the arbitrary axis rotation-invariant encoding method.

[0010] In some embodiments, in each position encoding module, determining the first relative position encoding and the second relative position encoding of the 3D point cloud data in the position encoding module according to the input features of the position encoding module, the first encoding method, and the second encoding method includes: in each position encoding module, mapping the input features of the position encoding module to features with a preset number of channels, wherein the preset number of channels corresponding to each position encoding module is the same or different; dividing the features with the preset number of channels into multiple features by dividing the number of channels by the number of types of encoding methods; encoding the 3D point cloud data according to at least one of the Z-axis rotation-invariant encoding method and the arbitrary axis rotation-invariant encoding method, as well as the second encoding method, to obtain multiple encoding results; and determining the first relative position encoding and the second relative position encoding of the 3D point cloud data in the position encoding module based on multiple fusion features obtained by fusing the multiple features with the corresponding encoding results.

[0011] In some embodiments, the Z-axis rotation-invariant encoding method includes the following method: For each point and each of its neighboring points in the three-dimensional point cloud data, the point is encoded based on at least one of the following: the difference between the Z-coordinates of the point and the neighboring point, the distance between the projections of the point and the neighboring point on the XY plane, the distance from the projection of the point on the XY plane to the origin, the distance from the projection of the neighboring point on the XY plane to the origin, and the angle formed by the projections of the point and the neighboring point on the XY plane and the origin, to obtain the Z-axis rotation-invariant relative position encoding of the point and the neighboring point.

[0012] In some embodiments, the arbitrary axis rotation-invariant encoding method includes the following method: for each point in the three-dimensional point cloud data, determine the centroid of all adjacent points of the point, the intersection of the straight line passing through the point and the origin and the unit circle centered at the point; for each adjacent point of the point, encode the point according to at least one of the following: the distance from the point to the origin, the distance from the centroid to the intersection, the distance from the point to the intersection, the distance from the centroid to the adjacent point, the distance from the intersection to the adjacent point, the distance from the point to the adjacent point, and the angle between the first plane and the second plane, to obtain the arbitrary axis rotation-invariant relative position encoding between the point and the adjacent point, wherein the first plane passes through the point, the centroid, and the intersection, and the second plane passes through the point, the centroid, and the adjacent points of the point.

[0013] In some embodiments, the second encoding method includes the following approach: for each point in the three-dimensional point cloud data and each of its neighboring points, the point is encoded according to the coordinate difference between the point and the neighboring point to obtain a second relative position encoding between the point and the neighboring point.

[0014] In some embodiments, each fusion feature among multiple fusion features is determined by the following method: for each feature among multiple features, the feature difference between each point in the feature and its neighboring points is determined as the feature difference corresponding to the feature; the feature, the feature difference corresponding to the feature, and the encoding result corresponding to the feature are concatenated and then input into the corresponding multilayer perceptron to obtain the fusion feature that fuses the feature and the corresponding encoding result.

[0015] In some embodiments, when the feature extraction model includes multiple location encoding modules, the feature extraction model includes one or more stages after the second location encoding module. Each stage includes: a location encoding module with downsampling, which divides the features of a preset number of channels into multiple features by dividing the number of channels by the number of encoding methods. This includes: for each location encoding module with downsampling, downsampling the features of a preset number of channels at a preset downsampling frequency, wherein the preset downsampling frequency corresponding to each location encoding module with downsampling decreases sequentially from input to output; dividing the features of the preset number of channels into multiple features by dividing the number of channels by the number of encoding methods; and encoding the 3D point cloud data according to at least one of the Z-axis rotation-invariant encoding method and the arbitrary axis rotation-invariant encoding method, as well as a second encoding method, to obtain multiple encoding results. This includes: for each location encoding module with downsampling, downsampling the 3D point cloud data at a preset downsampling frequency; and encoding the downsampled 3D point cloud data according to at least one of the Z-axis rotation-invariant encoding method and the arbitrary axis rotation-invariant encoding method, as well as a second encoding method, to obtain multiple encoding results.

[0016] In some embodiments, each location encoding module includes an attention mechanism submodule, which includes a fully connected layer and a normalization layer. Based on the attention mechanism, determining the weights of the first encoding method and the second encoding method corresponding to each point in the 3D point cloud data in the location encoding module includes: inputting the input features of the location encoding module into the fully connected layer of the attention mechanism submodule of the location encoding module to obtain the initial weights of the first encoding method and the second encoding method corresponding to each point in the 3D point cloud data; and inputting the initial weights of the first encoding method and the second encoding method corresponding to each point in the 3D point cloud data into the normalization layer of the attention mechanism submodule of the location encoding module to obtain the weights of the first encoding method and the second encoding method corresponding to each point in the 3D point cloud data in the location encoding module.

[0017] In some embodiments, determining the output features of the position encoding module includes: performing a weighted summation of the first relative position encoding and the second relative position encoding of the three-dimensional point cloud data in the position encoding module based on the weights of the first encoding method and the second encoding method corresponding to each point in the three-dimensional point cloud data in the position encoding module; and determining the output features of the position encoding module based on the result of the weighted summation and the input features of the position encoding module.

[0018] In some embodiments, when the feature extraction model includes multiple location encoding modules, the feature extraction model includes one or more stages after the second location encoding module. Each stage includes: a location encoding module with downsampling, and determining the output features of the location encoding module based on the weighted summation result and the input features of the location encoding module, including: for each location encoding module with downsampling, mapping the input features of the location encoding module to features with a preset number of channels; downsampling the features with a preset number of channels according to a preset downsampling frequency, wherein the preset downsampling frequency corresponding to each location encoding module after the second location encoding module decreases sequentially; and determining the output features of the location encoding module based on the weighted summation result and the features with a preset number of channels after downsampling.

[0019] In some embodiments, the method further includes: determining the category of the target corresponding to the three-dimensional point cloud data based on the characteristics of the three-dimensional point cloud data.

[0020] In some embodiments, the method further includes: segmenting the three-dimensional point cloud data based on the characteristics of the three-dimensional point cloud data.

[0021] According to some embodiments of this disclosure, a method for processing three-dimensional point cloud data is provided, comprising: inputting sample three-dimensional point cloud data into a feature extraction model, wherein the sample three-dimensional point cloud data is configured with corresponding annotation information, and the feature extraction model includes one or more position encoding modules; in each position encoding module, determining a first relative position encoding and a second relative position encoding of the sample three-dimensional point cloud data in the position encoding module according to the input features of the position encoding module, a first encoding method, and a second encoding method, wherein the first encoding method is a relative position encoding method when the target rotates relative to the acquisition device during the acquisition of the sample three-dimensional point cloud data, and the second encoding method is a relative position encoding method when the target's position remains unchanged relative to the acquisition device during the acquisition of the sample three-dimensional point cloud data. The location encoding method involves: determining the weights of the first and second encoding methods for each point in the sample 3D point cloud data based on an attention mechanism; determining the output features of the location encoding module based on the weights of the first and second encoding methods for each point in the sample 3D point cloud data, as well as the first and second relative location codes of the sample 3D point cloud data; determining the features of the sample 3D point cloud data based on the output features of the last location encoding module; determining the classification result of each point in the sample 3D point cloud data based on the features of the sample 3D point cloud data; and training the feature extraction model based on the classification result of each point in the sample 3D point cloud data and its corresponding annotation information.

[0022] According to some embodiments of this disclosure, a feature extraction model is provided, comprising: an input module, one or more cascaded position encoding modules, and an output module, wherein the input module is used to receive input three-dimensional point cloud data; each position encoding module is used to determine a first relative position encoding and a second relative position encoding of the three-dimensional point cloud data in the position encoding module according to the input features of the position encoding module, a first encoding method, and a second encoding method, wherein the first encoding method is a relative position encoding method when the target rotates relative to the acquisition device during the acquisition of the three-dimensional point cloud data, and the second encoding method is a relative position encoding method when the target's position relative to the acquisition device remains unchanged during the acquisition of the three-dimensional point cloud data; based on an attention mechanism, the weights of the first encoding method and the second encoding method corresponding to each point in the three-dimensional point cloud data in the position encoding module are determined; based on the weights of the first encoding method and the second encoding method corresponding to each point in the three-dimensional point cloud data in the position encoding module, and the first relative position encoding and the second relative position encoding of the three-dimensional point cloud data, the output features of the position encoding module are determined; the output module is used to output the features of the three-dimensional point cloud data according to the output features of the last position encoding module.

[0023] According to further embodiments of this disclosure, a three-dimensional point cloud data processing apparatus is provided, comprising: an acquisition module for acquiring three-dimensional point cloud data; an encoding module for determining a first relative position encoding and a second relative position encoding of the three-dimensional point cloud data according to a first encoding method and a second encoding method, wherein the first encoding method is a relative position encoding method when the target rotates relative to the acquisition device during the acquisition of the three-dimensional point cloud data, and the second encoding method is a relative position encoding method when the target's position relative to the acquisition device remains unchanged during the acquisition of the three-dimensional point cloud data; an attention module for determining the weight of the first encoding method and the weight of the second encoding method corresponding to each point in the three-dimensional point cloud data based on an attention mechanism; and an extraction module for extracting features from the three-dimensional point cloud data according to the weight of the first encoding method and the weight of the second encoding method corresponding to each point in the three-dimensional point cloud data, and the first relative position encoding and the second relative position encoding of the three-dimensional point cloud data.

[0024] According to some embodiments of this disclosure, a three-dimensional point cloud data processing apparatus is provided, comprising: an input module for inputting sample three-dimensional point cloud data into a feature extraction model, wherein the sample three-dimensional point cloud data is configured with corresponding annotation information, and the feature extraction model includes one or more position encoding modules; and an encoding module for determining, in each position encoding module, a first relative position encoding and a second relative position encoding of the sample three-dimensional point cloud data in that position encoding module, based on the input features of that position encoding module, a first encoding method, and a second encoding method, wherein the first encoding method is a relative position encoding method when the target rotates relative to the acquisition device during the acquisition of the sample three-dimensional point cloud data, and the second encoding method is a relative position encoding method when the target's position relative to the acquisition device remains unchanged during the acquisition of the sample three-dimensional point cloud data. The system comprises the following modules: an attention module, which determines the weights of the first and second encoding methods for each point in the sample 3D point cloud data based on an attention mechanism; a determination module, which determines the output features of the position encoding module based on the weights of the first and second encoding methods for each point in the sample 3D point cloud data, as well as the first and second relative position codes of the sample 3D point cloud data; and a feature module, which determines the features of the sample 3D point cloud data based on the output features of the last position encoding module; a classification module, which determines the classification result of each point in the sample 3D point cloud data based on the features of the sample 3D point cloud data; and a training module, which trains the feature extraction model based on the classification result and corresponding annotation information of each point in the sample 3D point cloud data.

[0025] According to some embodiments of the present disclosure, a three-dimensional point cloud data processing apparatus is provided, comprising: a processor; and a memory coupled to the processor for storing instructions, which, when executed by the processor, cause the processor to perform a three-dimensional point cloud data processing method as described in any of the foregoing embodiments.

[0026] According to some other embodiments of the present disclosure, a non-transitory computer-readable storage medium is provided, on which a computer program is stored, wherein when the program is executed by a processor, it implements the method for processing three-dimensional point cloud data of any of the foregoing embodiments.

[0027] According to further embodiments of this disclosure, an unmanned vehicle includes: a device for processing three-dimensional point cloud data of any of the foregoing embodiments.

[0028] This disclosure employs different first and second encoding methods for relative position encoding of 3D point cloud data. The first encoding method encodes the relative position of the target when it rotates relative to the acquisition device, while the second encoding method encodes the relative position of the target when its position relative to the acquisition device remains unchanged. Then, based on an attention mechanism, the weights of the first and second encoding methods are determined for each point. Finally, based on the weights of the first and second encoding methods for each point, and the first and second relative position encodings of the 3D point cloud data, features of the 3D point cloud data are extracted. This method, based on an attention mechanism, determines the degree of dependence of different regions on rotation robustness for 3D point cloud data. For regions with high rotation robustness requirements, a rotation-invariant relative position encoding method is used; for regions that do not require rotation robustness, a rotation-sensitive but higher-performance relative position encoding method is used. This dynamic encoding method can effectively improve the accuracy of feature extraction from 3D point cloud data.

[0029] Other features and advantages of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 A flowchart illustrating a method for processing three-dimensional point cloud data according to some embodiments of this disclosure is shown.

[0032] Figure 2AA schematic diagram of the structure of a feature extraction model according to some embodiments of this disclosure is shown.

[0033] Figure 2B A schematic diagram of the structure of a standard location encoding module is shown, representing some embodiments of this disclosure.

[0034] Figure 2C A schematic diagram of the structure of a position encoding module with downsampling is shown for some embodiments of this disclosure.

[0035] Figure 2D A schematic diagram of the structure of a dynamic position-coded multilayer perceptron according to some embodiments of the present disclosure is shown.

[0036] Figure 3 A flowchart illustrating a method for processing three-dimensional point cloud data according to other embodiments of this disclosure is shown.

[0037] Figure 4 A schematic diagram of the structure of a three-dimensional point cloud data processing apparatus according to some embodiments of the present disclosure is shown.

[0038] Figure 5 A schematic diagram of the structure of a three-dimensional point cloud data processing apparatus according to other embodiments of the present disclosure is shown.

[0039] Figure 6 A schematic diagram of the structure of a three-dimensional point cloud data processing apparatus according to further embodiments of the present disclosure is shown.

[0040] Figure 7 A schematic diagram of the structure of a three-dimensional point cloud data processing apparatus according to further embodiments of the present disclosure is shown.

[0041] Figure 8 A schematic diagram of the structure of an unmanned vehicle according to some embodiments of the present disclosure is shown.

[0042] Figure 9 A schematic diagram of the structure of an unmanned vehicle according to some embodiments of the present disclosure is shown.

[0043] Figure 10 A side view of an unmanned vehicle according to some embodiments of the present disclosure is shown. Detailed Implementation

[0044] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0045] In actual acquisition of 3D point cloud data, the acquisition angle of the acquisition device relative to the target often changes. For example, the target may be photographed from different angles such as the front and side, and the resulting point cloud data is used as the overall 3D point cloud data of the target. In this case, the rotation between different points is different. If the same relative position encoding method is used for each point, it will lead to inaccurate feature extraction from the 3D point cloud data. To address this problem, this disclosure provides a method for processing 3D point cloud data, which is described below in conjunction with... Figures 1-3 Describe it.

[0046] Figure 1 Flowcharts are shown for some embodiments of the method for processing 3D point cloud data disclosed herein. For example... Figure 1 As shown, the method of this embodiment includes steps S102 to S108.

[0047] In step S102, three-dimensional point cloud data is acquired.

[0048] For example, 3D point cloud data is a series of 3D coordinate points collected by acquisition devices such as 3D cameras and LiDAR, or obtained from 3D models, and is not limited to the examples given.

[0049] In step S104, the first relative position code and the second relative position code of the three-dimensional point cloud data are determined according to the first encoding method and the second encoding method.

[0050] In some embodiments, the first encoding method is a relative position encoding method when the target rotates relative to the acquisition device during the acquisition of 3D point cloud data, and the second encoding method is a relative position encoding method when the target's position relative to the acquisition device remains unchanged during the acquisition of 3D point cloud data.

[0051] To further improve the accuracy of feature extraction from 3D point cloud data, the first encoding method is further subdivided. In some embodiments, the first encoding method includes at least one of a Z-axis rotation-invariant encoding method and an arbitrary-axis rotation-invariant encoding method. The Z-axis rotation-invariant encoding method is a relative position encoding method when the target rotates relative to the acquisition device around the Z-axis during the acquisition of 3D point cloud data. The arbitrary-axis rotation-invariant encoding method is a relative position encoding method when the target rotates relative to the acquisition device around any axis during the acquisition of 3D point cloud data. Further, the first relative position encoding includes at least one of a Z-axis rotation position encoding method and an arbitrary-axis rotation position encoding method.

[0052] Z-axis rotation, where the vertical coordinate of the 3D point cloud data remains unchanged while rotating, is the most common case of global rotation in practical applications. Different shooting angles will cause the obtained point cloud data of the target to rotate around the Z-axis. In this case, using Z-axis rotation invariance encoding can improve encoding accuracy. Arbitrary axis rotation usually occurs in practical applications when small targets undergo local rotation, meaning the angle at which the small target is placed can rotate arbitrarily. In this case, arbitrary axis rotation invariance encoding can improve encoding accuracy.

[0053] The size of the target can be determined by whether the volume of the 3D point cloud data is smaller than a preset value, thus allowing selection of whether to use an arbitrary-axis rotation-invariant encoding method. Alternatively, the arbitrary-axis rotation-invariant encoding method can be used instead of the Z-axis rotation-invariant encoding method, meaning only the arbitrary-axis rotation-invariant encoding method is applied as the primary encoding method. Simultaneously applying both the Z-axis rotation-invariant encoding method and the arbitrary-axis rotation-invariant encoding method can further improve the accuracy of feature extraction from 3D point cloud data.

[0054] The second encoding method is applied when the target does not rotate at all relative to the acquisition device. In this case, the neural network is not required to be rotationally robust, and it is usually used as a reference input to determine the impact of rotation on performance.

[0055] The following sections describe the specific methods of the Z-axis rotation-invariant encoding method, the arbitrary axis rotation-invariant encoding method, and the second encoding method.

[0056] In some embodiments, the Z-axis rotation-invariant encoding method includes the following approach: for each point and each of its neighboring points in the three-dimensional point cloud data, the point is encoded based on at least one of the following: the difference in Z-coordinates between the point and the neighboring point, the distance between the projections of the point and the neighboring point on the XY plane, the distance from the projection of the point on the XY plane to the origin, the distance from the projection of the neighboring point on the XY plane to the origin, and the angle formed by the projections of the point and the neighboring point on the XY plane and the origin, to obtain the Z-axis rotation-invariant relative position encoding of the point and its neighboring point.

[0057] Z-axis rotation-invariant encoding can be called Z-RI (Z-axis Rotation-Invariant) encoding. It improves the robustness of 3D point cloud data features by addressing the most common Z-axis rotation scenarios in practical applications. For example, for the i-th point (hereinafter referred to as the center point) and its j-th neighboring point (hereinafter referred to as the neighboring point) in a 3D point cloud, Z-RI encoding includes at least one of the following: First part: the coordinate difference Δz between the Z-coordinates of the center point and the neighboring point. i,j This coordinate difference does not change as the target rotates relative to the acquisition device around the Z-axis; Part Two: The coordinates p of the center point i and adjacent point pj Projecting onto the XY two-dimensional plane yields p ′ i p j ′ , will p ′ i and p j ′ The distance r′ between i,j p i i and p j ′ Distance r′ from the origin i 、r′ j p ′ i and p j ′ The angle θ′ to the origin i,j As a Z-RI code, the formula is as follows:

[0058] P Z-RI (p i ,p j )=[Δz i,j ,r′ i,j ,r′ i ,r′ j ,θ′ i,j (1)

[0059] None of the above five characteristics change as the target rotates relative to the acquisition device around the Z-axis; that is, they are characteristics that remain unchanged when the Z-axis rotates.

[0060] In some embodiments, the arbitrary axis rotation invariant encoding method includes the following method: for each point in the three-dimensional point cloud data, determine the centroid of all adjacent points of the point, the intersection of the straight line passing through the point and the origin and the unit circle centered at the point; for each adjacent point of the point, encode the point according to at least one of the following: the distance from the point to the origin, the distance from the centroid to the intersection, the distance from the point to the intersection, the distance from the centroid to the adjacent point, the distance from the intersection to the adjacent point, the distance from the point to the adjacent point, and the angle between the first plane and the second plane, to obtain the arbitrary axis rotation relative position encoding of the point and the adjacent point, wherein the first plane passes through the point, the centroid, and the intersection, and the second plane passes through the point, the centroid, and the adjacent points of the point.

[0061] Arbitrary rotation-invariant (A-RI) encoding, building upon Z-RI encoding, further enhances the requirement for rotation invariance, meaning it remains unchanged regardless of any rotation of the target relative to the acquisition device. In practical applications, arbitrary axis rotation often occurs with small objects, and different object orientations can cause the acquired point cloud to rotate about any axis. To construct a more descriptive A-RI encoding, this disclosure proposes rotation-invariant features including auxiliary points. The auxiliary points include the centroid m of all adjacent points of the center point. i The intersection point s of the straight line passing through the center point and the origin and the unit circle centered at the center point. i With the help of these two auxiliary points, the relative positions of the center point and adjacent points can be better described.

[0062] A-RI encoding includes at least one of the following: distance r from the center point to the origin. i The distance r from the center of gravity to the intersection point ms,i The distance r from the center point to the intersection point ps,i The distance r from the centroid to the adjacent point mp,i,j The distance r from the centroid to the adjacent point sp,i,j The distance r from the center point to the adjacent point pp,i,j and s i -p i -m i plane to s i -p i -p j angle θ of the plane mp,i,j The formula is as follows:

[0063] P A-RI (p i ,p j )=[r i ,r ms,i ,r ps,i ,r pm,i ,r mp,i,j ,r sp,i,j ,r pp,i,j ,θ mp,i,j (2)

[0064] The relative position encoding features contained in A-RI encoding do not change under any rotation, that is, the target has invariance to any rotation relative to the acquisition device.

[0065] In some embodiments, the second encoding method includes the following approach: for each point in the three-dimensional point cloud data and each of its neighboring points, the point is encoded according to the coordinate difference between the point and the neighboring point to obtain a second relative position encoding between the point and the neighboring point.

[0066] The second encoding method can be called CD (Coordinate Difference) encoding, which directly uses the three-dimensional coordinate difference between the center point and adjacent points as the encoding feature. CD encoding can be expressed by the following formula.

[0067] P cD (p i ,p j ) = p j -p i (3)

[0068] CD encoding is a widely used relative position encoding method, and it has achieved good performance on point cloud data where the target does not rotate relative to the acquisition device. However, this encoding method is very sensitive to rotation; point cloud data of the same target taken from different angles will have different features under this position encoding, resulting in incorrect classification results. To solve this problem, this disclosure uses the aforementioned Z-RI encoding and A-RI encoding methods to improve the robustness of the feature extraction model to rotation.

[0069] This disclosure also provides an improved neural network model, namely a feature extraction model. The method of this disclosure is described below in conjunction with the structure of the feature extraction model.

[0070] In some embodiments, the feature extraction model includes one or more cascaded position encoding modules; three-dimensional point cloud data is input into the feature extraction model, and in each position encoding module, a first relative position encoding and a second relative position encoding of the three-dimensional point cloud data are determined according to the input features of the position encoding module, a first encoding method and a second encoding method.

[0071] In some embodiments, the feature extraction model includes an input layer, one or more cascaded positional encoding modules, and an output layer. The input layer and a positional encoding module can be combined into an input mapping layer, and further subdivisions of one or more positional encoding modules are included, divided into one or more stages. For example, the first stage includes a positional encoding module, and the second and subsequent stages include a positional encoding module with downsampling and a positional encoding module. Experiments demonstrate that the five-stage approach yields better feature extraction results for 3D point cloud data.

[0072] like Figure 2AAs shown, the structure of the feature extraction model references the design concept of convolutional neural networks. As the number of layers gradually increases, the number of feature channels increases while the number of points in the point cloud decreases. For example, when the feature extraction model is input with the coordinates of a 3D point cloud containing N points, these coordinates are first projected into the feature space by a fully connected layer and a position encoding module (which can be called a SEP (Selective Position Encoding) Block). Each point is projected into a feature space of dimension C1, and the feature dimension output by the input mapping layer is C1×N.

[0073] After input mapping, a series of positional encoding blocks (SPE blocks) are applied to the input features. The entire 3D point cloud neural network can consist of five stages. At the beginning of each of the second to fifth stages, there is a positional encoding block with downsampling, used to increase the feature dimension and reduce the number of points in the point cloud. For example, in the second to fifth stages, the number of points is reduced to N / 2, N / 8, N / 32, and N / 128 respectively. After the five stages of positional encoding blocks, the output point count is downsampled to, for example, N / 128, and the feature dimension (number of channels) of each point is C5.

[0074] The structure and function of the Position Encoding Module (SPE Block) are described in detail below.

[0075] In some embodiments, in each position encoding module, the input features of the position encoding module are mapped to features with a preset number of channels, wherein the preset number of channels corresponding to each position encoding module is the same or different; the features with the preset number of channels are divided into multiple features by dividing the number of channels by the number of encoding methods; the 3D point cloud data is encoded according to at least one of the Z-axis rotation-invariant encoding method and the arbitrary axis rotation-invariant encoding method, as well as the second encoding method, to obtain multiple encoding results; based on multiple fusion features obtained by fusing the multiple features with the corresponding encoding results, the first relative position encoding and the second relative position encoding of the 3D point cloud data in the position encoding module are determined.

[0076] Furthermore, in some embodiments, when the feature extraction model includes multiple location encoding modules, the feature extraction model includes one or more stages after the second location encoding module. Each stage includes: a location encoding module with downsampling; for each location encoding module with downsampling, downsampling the features with a preset number of channels according to a preset downsampling frequency, wherein the preset downsampling frequency corresponding to each location encoding module with downsampling decreases sequentially from input to output; dividing the features with the preset number of channels into multiple features by dividing the number of channels by the number of encoding methods; downsampling the 3D point cloud data according to the preset downsampling frequency; and encoding the downsampled 3D point cloud data according to at least one of the Z-axis rotation-invariant encoding method and the arbitrary axis rotation-invariant encoding method, as well as the second encoding method, to obtain multiple encoding results.

[0077] Furthermore, based on multiple fusion features that are fused with the corresponding encoding results, the first relative position encoding and the second relative position encoding of the 3D point cloud data in the position encoding module are determined.

[0078] Furthermore, in some embodiments, for each feature among multiple features, the feature difference between each point in the feature and its neighboring points is determined as the feature difference corresponding to the feature; the feature, the feature difference corresponding to the feature, and the encoding result corresponding to the feature are concatenated and then input into the corresponding multilayer perceptron to obtain the fused feature that fuses the feature and the corresponding encoding result.

[0079] like Figure 2B and 2C As shown, the Position Encoding Module (SPE Block) includes a standard position encoding module and a position encoding module with downsampling. The core of the position encoding module is a dynamic position encoding multilayer perceptron. The position encoding module is a specially designed residual module containing a dynamic position encoding multilayer perceptron and two fully connected layers preceding and following it. These three layers jointly learn the feature residuals and add them to the input features to obtain the output features.

[0080] For example, such as Figure 2B As shown, a standard location encoding module may include a sub-input layer for mapping input features to features with a preset number of channels. For example, the sub-input layer may include a fully connected layer, a normalization layer, and an activation layer. Furthermore, the location encoding module includes a dynamic location encoding multilayer perceptron, which is the core component of the module. The location encoding module may also include a sub-output layer, which may include a fully connected layer, a normalization layer, and an activation layer. The activation layer's function is to fuse the input features of the location encoding module with the features obtained after passing through the fully connected layer and the normalization layer.

[0081] like Figure 2B As shown, the input feature of the i-th standard position coding module is, for example, C. i ×N i The sub-input layer can map the input features to C. i / 2×N i The feature after dynamic position coding multilayer perceptron is C. i / 2×N i The feature after the fully connected layer and the normalization layer is C. i ×N i .

[0082] For example, such as Figure 2C As shown, the positional encoding module with downsampling has a similar structure to the standard positional encoding module. The downsampling function is implemented in the dynamic positional encoding multilayer perceptron and the input feature mapping layer. The dynamic positional encoding multilayer perceptron mainly implements downsampling in the positional encoding process, and the input feature mapping layer can include fully connected layers and normalization layers to implement downsampling and mapping of input features.

[0083] like Figure 2C As shown, the input feature of the i-th position encoding module with downsampling is, for example, C. i ×N i The sub-input layer can map the input features to C. O / 2×N i The feature after dynamic position coding multilayer perceptron is C. O / 2×N O The feature after the fully connected layer and the normalization layer is C. O ×N O The features after the input feature mapping layer are C. O ×N O For example, in the second stage, C O For C2, N O It is N / 2.

[0084] The structure and function of the dynamic position coding multilayer perceptron are described in detail below.

[0085] Dynamic Position Coding Multilayer Perceptron (SPE-MLP): In point cloud analysis networks (feature extraction models), the core problem is how to encode the relative position of any point with its neighbors and apply the encoded features to the structural judgment of the point cloud. The SPE-MLP structure consists of two parts: three different relative position codes and an encoding selector. The three relative position codes are rotation-sensitive CD encoding, Z-axis rotation-invariant Z-RI encoding, and A-RI encoding, which is invariant to any rotation. These three position encoding methods yield features with different degrees of sensitivity to rotation. The encoding selector determines the most needed features for each point based on the input features, thus selecting the features that best describe that point. Here, the encoding selector's role is equivalent to pre-judging the situation where the data undergoes local rotation. For regions where the semantics change during rotation, rotation-sensitive encoding features are selected; for regions where rotation does not affect the semantics, rotation-invariant features are used, thus providing a more targeted description of the local structure of the point cloud.

[0086] like Figure 2D As shown, in a dynamic position-coded multilayer perceptron, the features with a preset number of channels are first divided into multiple features by dividing the number of channels by the number of coding schemes. For example, when using CD coding, Z-RI coding, and A-RI coding simultaneously, the features are divided according to the number of channels C. x The feature set with a preset number of channels is divided into three parts using a / 3 method. The 3D point cloud data features are then input into the CD encoding module, Z-RI encoding module, and A-RI encoding module, respectively. These modules not only implement CD encoding, Z-axis rotation-invariant encoding, and arbitrary axis rotation-invariant encoding, but also fuse multiple feature sets with their corresponding encoding results. For example, after CD encoding, the feature is... K represents the number of adjacent points, N O In the location encoding module with downsampling, the number of points is the number of points after downsampling; in the standard location encoding module, it is the number of points corresponding to the input features. Downsampling both the input features and the 3D point cloud data reduces subsequent data processing and improves efficiency. This represents a feature corresponding to the CD encoding, the feature difference corresponding to that feature, and the number of channels after concatenating the encoding results corresponding to that feature according to the channels.

[0087] After Z-RI encoding, the features are After A-RI encoding, the feature is The features output from the CD encoding module, Z-RI encoding module, and A-RI encoding module are then input into a multilayer perceptron (MLP) for feature fusion. The feature output from each MLP is C.O / 3×N O ×K.

[0088] For example, for a dynamic position-encoded multilayer perceptron, its input feature is f i By uniformly dividing the number of channels, three equally divided sub-features f are obtained. i (1) ,f i (2) ,f i (3) Each feature and a location encoding method are fused using a multilayer perceptron. The fusion method of the three sub-features and location encoding is as follows:

[0089]

[0090]

[0091]

[0092] Among them, F1, F2, and F3 are three multilayer perceptron architectures. and These are three integrated features.

[0093] Thus, the first relative position code and the second relative position code of the 3D point cloud data in each position coding module are obtained.

[0094] In step S106, based on the attention mechanism, the weights of the first encoding method and the second encoding method corresponding to each point in the 3D point cloud data are determined.

[0095] In some embodiments, in each location encoding module, based on an attention mechanism, the weights of a first encoding method and a second encoding method corresponding to each point in the 3D point cloud data in that location encoding module are determined; the weights of the first encoding method include at least one of the weights of the Z-axis rotation-invariant encoding method and the weights of the arbitrary axis rotation-invariant encoding method.

[0096] In some embodiments, each location encoding module includes an attention mechanism submodule, which includes a fully connected layer and a normalization layer. The input features of the location encoding module are input into the fully connected layer of the attention mechanism submodule of the location encoding module to obtain the initial weights of the first encoding method and the initial weights of the second encoding method for each point in the 3D point cloud data. The initial weights of the first encoding method and the initial weights of the second encoding method for each point in the 3D point cloud data are input into the normalization layer of the attention mechanism submodule of the location encoding module to obtain the weights of the first encoding method and the second encoding method for each point in the 3D point cloud data in that location encoding module.

[0097] The attention mechanism submodule, which can be called an encoding selector, learns the importance of various encoding methods for each point from the input features of each positional encoding module. By assigning higher weights to more important encodings, it dynamically adjusts the rotational robustness of the relative positional encoding, thus achieving the goal of dynamic positional encoding. For example, for CD encoding, Z-RI encoding, and A-RI encoding, the weights of various encoding methods can be determined using the following formula.

[0098]

[0099] Here, FC is a fully connected layer used to predict weights, and Sigmoid is a Sigmoid function used to normalize the weights. These are the weights for the three encoding methods respectively.

[0100] In step S108, features of the three-dimensional point cloud data are extracted based on the weights of the first and second encoding methods corresponding to each point in the three-dimensional point cloud data, as well as the first and second relative position codes of the three-dimensional point cloud data.

[0101] In some embodiments, in each position encoding module, the output features of the position encoding module are determined based on the weights of the first encoding method and the second encoding method corresponding to each point in the 3D point cloud data in the position encoding module, as well as the first relative position encoding and the second relative position encoding of the 3D point cloud data; and the features of the 3D point cloud data are determined based on the output features of the last position encoding module.

[0102] Furthermore, in some embodiments, in each location encoding module, a weighted sum is performed on the first relative location encoding and the second relative location encoding of the 3D point cloud data in the location encoding module according to the weight of the first encoding method and the weight of the second encoding method corresponding to each point in the 3D point cloud data in the location encoding module; and the output features of the location encoding module are determined according to the result of the weighted sum and the input features of the location encoding module.

[0103] like Figure 2D As shown, the weights of the first and second encoding methods for each point in the 3D point cloud data are determined by an encoding selector, and then the first and second relative position codes are weighted and summed. For example, for CD encoding, Z-RI encoding, and A-RI encoding, the following formula can be used for weighted summation.

[0104]

[0105] The weighted summation result can be further reduced in dimensionality by a max pooling layer to obtain the output features of the dynamic position-coded multilayer perceptron.

[0106] like Figure 2B As shown, in the standard position coding module, the output features of the dynamic position coding multilayer perceptron pass through the fully connected layer and normalization layer in the sub-output layer, and then are fused with the input features of the position coding module through the activation layer to obtain the output features of the position coding module.

[0107] like Figure 2C As shown, in the position coding module with downsampling, the output features of the dynamic position coding multilayer perceptron pass through the fully connected layer and the normalization layer in the sub-output layer, the input features of the position coding module pass through the input feature mapping layer, and then the two are fused through the activation layer to obtain the output features of the position coding module.

[0108] The output features of the last location encoding module are passed through the output layer of the feature extraction model and used as features of the output 3D point cloud data.

[0109] The method in the above embodiments employs different first and second encoding methods for relative position encoding of 3D point cloud data. The first encoding method encodes the relative position of the target when it rotates relative to the acquisition device, while the second encoding method encodes the relative position of the target when its position relative to the acquisition device remains unchanged. Then, based on an attention mechanism, the weights of the first and second encoding methods are determined for each point. Finally, based on the weights of the first and second encoding methods for each point, and the first and second relative position encodings of the 3D point cloud data, features of the 3D point cloud data are extracted. The method in the above embodiments, based on an attention mechanism, determines the degree of dependence of different regions on rotation robustness for 3D point cloud data. For regions with high rotation robustness requirements, a rotation-invariant relative position encoding method is used; for regions that do not require rotation robustness, a rotation-sensitive but higher-performance relative position encoding method is used. This dynamic encoding method can effectively improve the accuracy of feature extraction from 3D point cloud data.

[0110] In some embodiments, the category of the target corresponding to the 3D point cloud data is determined based on the characteristics of the 3D point cloud data.

[0111] In other embodiments, the 3D point cloud data is segmented based on its characteristics.

[0112] The features extracted from 3D point cloud data can be used in tasks such as identifying the category of an object or segmenting the point cloud region of an object. For example, after average pooling, the features of the 3D point cloud data are used with a fully connected layer to predict the category of each point, thereby determining the category of the object or segmenting the point cloud region of the object.

[0113] The following is combined Figure 3 The training method of the feature extraction model in this disclosure is described.

[0114] Figure 3 Flowcharts are shown for some other embodiments of the method for processing 3D point cloud data disclosed herein. For example... Figure 3 As shown, the method of this embodiment includes steps S302 to S314.

[0115] In step S302, the sample 3D point cloud data is input into the feature extraction model, wherein the sample 3D point cloud data is configured with corresponding annotation information.

[0116] The feature extraction model includes one or more location encoding modules; the structure of the feature extraction model can be referred to the aforementioned embodiments, and will not be repeated here. The annotation information is, for example, the annotation category of each point in the sample 3D point cloud data.

[0117] In step S304, in each position encoding module, the first relative position encoding and the second relative position encoding of the sample 3D point cloud data are determined according to the input features of the position encoding module, the first encoding method and the second encoding method.

[0118] In some embodiments, the first encoding method is a relative position encoding method when the target rotates relative to the acquisition device during the acquisition of sample 3D point cloud data, and the second encoding method is a relative position encoding method when the target's position relative to the acquisition device remains unchanged during the acquisition of sample 3D point cloud data.

[0119] In step S306, in each location encoding module, based on the attention mechanism, the weights of the first encoding method and the second encoding method corresponding to each point in the sample 3D point cloud data in that location encoding module are determined.

[0120] In step S308, in each position encoding module, the output features of the position encoding module are determined based on the weights of the first encoding method and the second encoding method corresponding to each point in the sample 3D point cloud data in the position encoding module, as well as the first relative position encoding and the second relative position encoding of the sample 3D point cloud data.

[0121] In step S310, the features of the sample 3D point cloud data are determined based on the output features of the last position encoding module.

[0122] In step S312, the classification result of each point in the sample 3D point cloud data is determined based on the characteristics of the sample 3D point cloud data.

[0123] In step S314, the feature extraction model is trained based on the classification results and corresponding annotation information of each point in the sample 3D point cloud data.

[0124] The process of extracting features from 3D point cloud data can be referred to in the foregoing embodiments, and will not be repeated here.

[0125] This disclosure also provides a feature extraction model, including: an input module, one or more cascaded position encoding modules, and an output module, wherein the input module is used to receive input three-dimensional point cloud data; each position encoding module is used to determine a first relative position encoding and a second relative position encoding of the three-dimensional point cloud data in the position encoding module according to the input features of the position encoding module, a first encoding method, and a second encoding method, wherein the first encoding method is a relative position encoding method when the target rotates relative to the acquisition device during the acquisition of the three-dimensional point cloud data, and the second encoding method is a relative position encoding method when the target's position relative to the acquisition device remains unchanged during the acquisition of the three-dimensional point cloud data; based on an attention mechanism, the weights of the first encoding method and the second encoding method corresponding to each point in the three-dimensional point cloud data in the position encoding module are determined; based on the weights of the first encoding method and the second encoding method corresponding to each point in the three-dimensional point cloud data in the position encoding module, and the first relative position encoding and the second relative position encoding of the three-dimensional point cloud data, the output features of the position encoding module are determined; the output module is used to output the features of the three-dimensional point cloud data according to the output features of the last position encoding module.

[0126] The structure and function of the feature extraction model can be referred to the aforementioned embodiments, and will not be repeated here.

[0127] This disclosure proposes a neural network structure based on dynamic positional encoding (SPE) to improve the rotation robustness of 3D point cloud analysis. The feature extraction model of this disclosure belongs to the category of neural networks that perform content analysis on input raw 3D point cloud data. Compared with existing point cloud neural networks, this disclosure proposes the concept of dynamic positional encoding (SPE), which uses the optimal encoding method to extract features based on the relative positions between different points, thereby improving the performance of point cloud analysis. The basic idea is to use the differences in features in the point cloud to determine the degree of dependence of different regions on rotation robustness. For regions with high rotation robustness requirements, a rotation-invariant relative positional encoding method is used; for regions that do not require rotation robustness, a rotation-sensitive but higher-performance relative positional encoding method is used. This dynamic encoding method allows the neural network to use features with different rotation robustness for different regions of the point cloud, thereby improving the performance of point cloud analysis.

[0128] In practical applications, since the rotation of the point cloud is unknown, it is difficult to pre-select the optimal position encoding method. If an incorrect position encoding method is selected, performance will be significantly affected. To solve this problem, the dynamic position encoding method designed in this disclosure uses an attention mechanism to dynamically select three different position encoding methods when the point cloud is input into the neural network. This allows different points in the point cloud to choose different position encoding features, thereby more accurately describing the structural information of the point cloud.

[0129] This disclosure proposes three different relative position encoding equations, each targeting varying degrees of rotation robustness requirements, including rotation-sensitive CD encoding, Z-RI encoding, and A-RI encoding. Furthermore, it proposes using an attention mechanism to weight the features obtained from different encodings, using the weighted features as the module's output features, thereby dynamically adjusting the rotation robustness of features from different regions. Combining these methods improves the overall performance of point cloud analysis and enhances the accuracy of feature extraction from 3D point cloud data.

[0130] This disclosure provides a device for processing three-dimensional point cloud data, which will be described below in conjunction with... Figure 4 Describe it.

[0131] Figure 4 These are structural diagrams of some embodiments of the three-dimensional point cloud data processing apparatus of this disclosure. Figure 4 As shown, the apparatus 40 in this embodiment includes: an acquisition module 410, an encoding module 420, an attention module 430, and an extraction module 440.

[0132] The acquisition module 410 is used to acquire 3D point cloud data. The acquisition module 410 is used to execute the method of step S102 and related embodiments, which will not be described in detail here.

[0133] The encoding module 420 is used to determine the first relative position encoding and the second relative position encoding of the three-dimensional point cloud data according to the first encoding method and the second encoding method. The first encoding method is the relative position encoding method when the target rotates relative to the acquisition device during the acquisition of the three-dimensional point cloud data, and the second encoding method is the relative position encoding method when the target's position relative to the acquisition device remains unchanged during the acquisition of the three-dimensional point cloud data.

[0134] The encoding module 420 is used to execute the method of step S104 and its related embodiments, which will not be described in detail here.

[0135] Attention module 430 is used to determine the weight of the first encoding method and the weight of the second encoding method for each point in the 3D point cloud data based on the attention mechanism.

[0136] The attention module 430 is used to perform the method of step S106 and related embodiments, which will not be described in detail here.

[0137] The extraction module 440 is used to extract features of the three-dimensional point cloud data based on the weights of the first encoding method and the second encoding method corresponding to each point in the three-dimensional point cloud data, as well as the first relative position encoding and the second relative position encoding of the three-dimensional point cloud data.

[0138] The extraction module 440 is used to perform the method of step S108 and its related embodiments, which will not be described in detail here.

[0139] In some embodiments, the device 40 further includes a target recognition module 450, configured to determine the category of the target corresponding to the three-dimensional point cloud data based on the characteristics of the three-dimensional point cloud data.

[0140] In some embodiments, the apparatus 40 further includes a segmentation module 460, configured to segment the three-dimensional point cloud data according to the features of the three-dimensional point cloud data.

[0141] Figure 5 These are structural diagrams of other embodiments of the three-dimensional point cloud data processing apparatus of this disclosure. For example... Figure 5 As shown, the device 50 in this embodiment includes: an input module 510, an encoding module 520, an attention module 530, a determination module 540, a classification module 550, and a training module 560.

[0142] The input module 510 is used to input sample 3D point cloud data into the feature extraction model. The sample 3D point cloud data is configured with corresponding annotation information, and the feature extraction model includes one or more position encoding modules.

[0143] The encoding module 520 is used in each position encoding module to determine the first relative position encoding and the second relative position encoding of the sample three-dimensional point cloud data in the position encoding module according to the input features of the position encoding module, the first encoding method and the second encoding method. The first encoding method is the relative position encoding method when the target rotates relative to the acquisition device when the sample three-dimensional point cloud data is acquired, and the second encoding method is the relative position encoding method when the target's position relative to the acquisition device remains unchanged when the sample three-dimensional point cloud data is acquired.

[0144] Attention module 530 is used to determine the weight of the first encoding method and the weight of the second encoding method for each point in the sample 3D point cloud data in the location encoding module based on the attention mechanism.

[0145] The determining module 540 is used to determine the output features of the position encoding module based on the weights of the first encoding method and the second encoding method corresponding to each point in the sample 3D point cloud data in the position encoding module, as well as the first relative position encoding and the second relative position encoding of the sample 3D point cloud data; and to determine the features of the sample 3D point cloud data based on the output features of the last position encoding module.

[0146] The classification module 550 is used to determine the classification result of each point in the sample 3D point cloud data based on the characteristics of the sample 3D point cloud data.

[0147] The training module 560 is used to train the feature extraction model based on the classification results and corresponding annotation information of each point in the sample 3D point cloud data.

[0148] The input module 510 is used to execute the method of step S302 and related embodiments, the encoding module 520 is used to execute the method of step S304 and related embodiments, the attention module 530 is used to execute the method of step S306 and related embodiments, the determination module 540 is used to execute the method of steps S308 to S310 and related embodiments, the classification module 550 is used to execute the method of step S312 and related embodiments, and the training module 560 is used to execute the method of step S314 and related embodiments, which will not be described in detail here.

[0149] The process of extracting features from 3D point cloud data can be referred to in the foregoing embodiments, and will not be repeated here.

[0150] The three-dimensional point cloud data processing device in the embodiments of this disclosure can be implemented by various computing devices or computer systems, as described below. Figure 6 as well as Figure 7 Describe it.

[0151] Figure 6 These are structural diagrams of some embodiments of the three-dimensional point cloud data processing apparatus of this disclosure. Figure 6 As shown, the apparatus 60 of this embodiment includes a memory 610 and a processor 620 coupled to the memory 610. The processor 620 is configured to execute a method for processing three-dimensional point cloud data in any of the embodiments of this disclosure based on instructions stored in the memory 610.

[0152] The memory 610 may include, for example, system memory, fixed non-volatile storage media, etc. The system memory may store, for example, an operating system, application programs, a boot loader, a database, and other programs.

[0153] Figure 7 These are structural diagrams of other embodiments of the three-dimensional point cloud data processing apparatus of this disclosure. For example... Figure 7 As shown, the device 70 of this embodiment includes a memory 710 and a processor 720, which are similar to the memory 610 and processor 620, respectively. It may also include an input / output interface 730, a network interface 740, a storage interface 750, etc. These interfaces 730, 740, 750, and the memory 710 and processor 720 can be connected, for example, via a bus 760. The input / output interface 730 provides a connection interface for input / output devices such as a display, mouse, keyboard, and touchscreen. The network interface 740 provides a connection interface for various networked devices, such as connecting to a database server or cloud storage server. The storage interface 750 provides a connection interface for external storage devices such as SD cards and USB flash drives.

[0154] This disclosure also provides an unmanned vehicle, which is described below in conjunction with... Figure 8-10 Describe it.

[0155] Figure 8 This is a structural diagram of some embodiments of the unmanned vehicle disclosed herein. Figure 8 As shown, the unmanned vehicle 8 in this embodiment includes a 3D point cloud data processing device 40 / 50 / 60 / 70. The unmanned vehicle may also include a data acquisition device 82 for acquiring 3D point cloud data. The data acquisition device 82 may be, for example, a lidar, a 3D camera, etc.

[0156] Figure 9 These are structural diagrams of other embodiments of the unmanned vehicle disclosed herein. For example... Figure 9 As shown, the unmanned vehicle 9 in this embodiment includes four main parts: an autonomous driving module 91, a chassis module 92, a remote monitoring and propulsion module 93, and a cargo box module 94.

[0157] The autonomous driving module 91 includes an autonomous driving sensor and core processing unit (Orin or Xavier module) assembly 911, a traffic light recognition camera 912, front, rear, left, and right surround view cameras 9131, 9132, 9133, and 9134, a multi-line LiDAR 914, a positioning module 915 (such as BeiDou, GPS, etc.), and an inertial navigation unit 916. The cameras and the autonomous driving module can communicate; to improve transmission speed and reduce wiring, a GMSL link communication can be used. The autonomous driving sensor and core processing unit (Orin or Xavier module) assembly 911 includes a three-dimensional point cloud data processing device as described in any of the embodiments of this disclosure and is configured to execute the three-dimensional point cloud data processing methods of any of the embodiments of this disclosure. The front, rear, left, and right surround view cameras 9131, 9132, 9133, and 9134 or the multi-line LiDAR 914 can be used to acquire three-dimensional point cloud data.

[0158] In some embodiments, the autonomous driving module 91 further includes a switch 917 and front, rear, left, and right blind spot radars 9181, 9182, 9183, and 9184.

[0159] The chassis module 92 mainly includes a battery 921, a power management device 922, a chassis controller 923, a motor driver 924, and a drive motor 925. The battery 921 provides power to the entire autonomous vehicle system. The power management device 922 converts the battery output into different voltage levels usable by various functional modules and controls power-on and power-off. The chassis controller 923 receives motion commands from the autonomous driving module and controls the autonomous vehicle's steering, forward movement, reverse movement, braking, etc. The chassis module 92 also includes a main battery 926.

[0160] The remote monitoring streaming module 93 consists of a front monitoring camera 931, a rear monitoring camera 932, a left monitoring camera 933, a right monitoring camera 934, and a streaming module 934. This module transmits the video data collected by the monitoring cameras to the backend server for viewing by backend operators.

[0161] The cargo container module 94 is the cargo-carrying device for the unmanned vehicle, including a delivery container 941. The cargo container module 94 also features a display and interaction module 942. This module allows interaction between the unmanned vehicle and the user, enabling operations such as picking up items, storing goods, and purchasing goods. The type of cargo container can be changed according to actual needs. For example, in logistics scenarios, the cargo container can include multiple sub-containers of different sizes, which can be used to load goods for delivery. In retail scenarios, the cargo container can be made transparent so that users can clearly see the products for sale.

[0162] The cargo box module 94 also includes an antenna 943. The chassis module 92 also includes a wireless communication module 927. The wireless communication module 927 communicates with the back-end server through the antenna 943, enabling remote control of the unmanned vehicle by back-end operators.

[0163] Figure 10 This is a side view showing an unmanned vehicle according to some embodiments of the present disclosure.

[0164] like Figure 10 As shown, the unmanned vehicle includes a display and interaction module 101, a chassis 102, a left-side blind spot radar 103, a right-side blind spot radar 104, a rear-side blind spot radar 105, a lidar 106, a right-side camera 107, and a cargo box 108. The functions of the interaction module 101, chassis 102, left-side blind spot radar 103, right-side blind spot radar 104, rear-side blind spot radar 105, lidar 106, right-side camera 107, and cargo box 108 can be found in [reference needed]. Figure 9 The description in the text will not be repeated here.

[0165] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, systems, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable non-transitory storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0166] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0167] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0168] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0169] The above description is only a preferred embodiment of this disclosure and is not intended to limit this disclosure. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the protection scope of this disclosure.

Claims

1. A method for processing three-dimensional point cloud data, comprising: Acquire 3D point cloud data; According to the first encoding method and the second encoding method, the first relative position encoding and the second relative position encoding of the three-dimensional point cloud data are determined, wherein the first encoding method is the relative position encoding method when the target rotates relative to the acquisition device when acquiring the three-dimensional point cloud data, and the second encoding method is the relative position encoding method when the target's position relative to the acquisition device remains unchanged when acquiring the three-dimensional point cloud data. In each location encoding module, based on an attention mechanism, the weights of the first encoding method and the second encoding method corresponding to each point in the 3D point cloud data in that location encoding module are determined; Based on the weights of the first and second encoding methods corresponding to each point in the 3D point cloud data, as well as the first and second relative position codes of the 3D point cloud data, the features of the 3D point cloud data are extracted.

2. The processing method according to claim 1, wherein, The step of determining the first relative position code and the second relative position code of the 3D point cloud data according to the first encoding method and the second encoding method includes: The three-dimensional point cloud data is input into the feature extraction model, wherein the feature extraction model includes one or more cascaded position encoding modules; In each location encoding module, the first relative location encoding and the second relative location encoding of the three-dimensional point cloud data in that location encoding module are determined according to the input features of that location encoding module, the first encoding method and the second encoding method.

3. The processing method according to claim 2, wherein, The features extracted from the 3D point cloud data include: In each location encoding module, the output features of the location encoding module are determined based on the weights of the first encoding method and the second encoding method corresponding to each point in the 3D point cloud data in that location encoding module, as well as the first relative location encoding and the second relative location encoding of the 3D point cloud data. The features of the 3D point cloud data are determined based on the output features of the last location encoding module.

4. The processing method according to claim 2, wherein, The first encoding method includes at least one of a Z-axis rotation invariant encoding method and an arbitrary axis rotation invariant encoding method. The Z-axis rotation invariant encoding method is a relative position encoding method when the target rotates around the Z-axis relative to the acquisition device during the acquisition of the three-dimensional point cloud data. The arbitrary axis rotation invariant encoding method is a relative position encoding method when the target rotates around an arbitrary axis relative to the acquisition device during the acquisition of the three-dimensional point cloud data. The first relative position encoding includes at least one of: Z-axis rotational position encoding and arbitrary axis rotational position encoding; The weight of the first encoding method includes at least one of the weight of the Z-axis rotation-invariant encoding method and the weight of the arbitrary axis rotation-invariant encoding method.

5. The processing method according to claim 4, wherein, In each location encoding module, determining the first relative location encoding and the second relative location encoding of the 3D point cloud data in that location encoding module, based on the input features of that location encoding module, the first encoding method, and the second encoding method, includes: In each location encoding module, the input features of the location encoding module are mapped to features with a preset number of channels, wherein the preset number of channels corresponding to each location encoding module may be the same or different; The features of the preset number of channels are divided into multiple features by dividing the number of channels by the number of encoding methods; The three-dimensional point cloud data is encoded according to at least one of the Z-axis rotation-invariant encoding method and the arbitrary axis rotation-invariant encoding method, as well as the second encoding method, to obtain multiple encoding results; Based on multiple fusion features obtained by fusing the multiple features with the corresponding encoding results, the first relative position code and the second relative position code of the three-dimensional point cloud data in the position encoding module are determined.

6. The processing method according to claim 4, wherein, The Z-axis rotation-invariant encoding method includes the following methods: For each point and each of its adjacent points in the three-dimensional point cloud data, the point is encoded based on at least one of the following: the difference between the Z coordinates of the point and the adjacent point, the distance between the projections of the point and the adjacent point on the XY plane, the distance from the projection of the point on the XY plane to the origin, the distance from the projection of the adjacent point on the XY plane to the origin, and the angle formed by the projections of the point and the adjacent point on the XY plane and the origin, so as to obtain the rotation-invariant relative position code of the point and the adjacent point on the Z-axis.

7. The processing method according to claim 4, wherein, The arbitrary axis rotation invariant encoding method includes the following methods: For each point in the three-dimensional point cloud data, determine the centroid of all adjacent points of that point, the intersection of the straight line passing through that point and the origin with the unit circle centered at that point; For each adjacent point of the given point, the point is encoded based on at least one of the following: the distance from the point to the origin, the distance from the centroid to the intersection, the distance from the point to the intersection, the distance from the centroid to the adjacent point, the distance from the intersection to the adjacent point, the distance from the point to the adjacent point, and the angle between the first plane and the second plane. This results in a relative position code between the point and the adjacent point that remains unchanged regardless of any axis rotation. The first plane passes through the point, the centroid, and the intersection, and the second plane passes through the point, the centroid, and the adjacent points of the point.

8. The processing method according to claim 4, wherein, The second encoding method includes the following methods: For each point and each of its neighboring points in the three-dimensional point cloud data, the point is encoded according to the coordinate difference between the point and the neighboring point to obtain the second relative position code of the point and the neighboring point.

9. The processing method according to claim 5, wherein, Each of the multiple fusion features is determined using the following method: For each of the multiple features, the feature difference between each point in that feature and its neighboring points is determined as the feature difference corresponding to that feature. The feature, the feature difference corresponding to the feature, and the encoding result corresponding to the feature are concatenated and then input into the corresponding multilayer perceptron to obtain the fused feature that combines the feature and the corresponding encoding result.

10. The processing method according to claim 5, wherein, In the case where the feature extraction model includes multiple location encoding modules, the feature extraction model includes one or more stages after the second location encoding module, each stage including: a location encoding module with downsampling, wherein dividing the preset number of channel features into multiple features by dividing the number of channels by the number of encoding methods includes: For each position encoding module with downsampling, the features of the preset number of channels are downsampled according to a preset downsampling frequency, wherein the preset downsampling frequency corresponding to each position encoding module with downsampling decreases sequentially from input to output. The features of the preset number of downsampled channels are divided into multiple features by dividing the number of channels by the number of encoding methods. The step of encoding the three-dimensional point cloud data according to at least one of the Z-axis rotation-invariant encoding method and the arbitrary axis rotation-invariant encoding method, as well as the second encoding method, to obtain multiple encoding results includes: For each position encoding module with downsampling, the three-dimensional point cloud data is downsampled according to a preset downsampling frequency; The downsampled 3D point cloud data is encoded using at least one of the following methods: Z-axis rotation invariant encoding, arbitrary axis rotation invariant encoding, and the second encoding method, to obtain multiple encoding results.

11. The processing method according to claim 3, wherein, Each location encoding module includes an attention mechanism submodule, which comprises a fully connected layer and a normalization layer. The determination of the weights of the first encoding method and the second encoding method for each point in the 3D point cloud data based on the attention mechanism within the location encoding module includes: The input features of the position encoding module are input into the fully connected layer of the attention mechanism submodule of the position encoding module to obtain the initial weights of the first encoding method and the initial weights of the second encoding method for each point in the 3D point cloud data. The initial weights of the first encoding method and the second encoding method corresponding to each point in the 3D point cloud data are input into the normalization layer of the attention mechanism submodule of the position encoding module to obtain the weights of the first encoding method and the second encoding method corresponding to each point in the 3D point cloud data in the position encoding module.

12. The processing method according to claim 3, wherein, The determination of the output features of the position encoding module includes: Based on the weights of the first encoding method and the second encoding method corresponding to each point in the three-dimensional point cloud data in the location encoding module, the first relative position encoding and the second relative position encoding of the three-dimensional point cloud data in the location encoding module are weighted and summed. Based on the weighted summation result and the input features of the position encoding module, the output features of the position encoding module are determined.

13. The processing method according to claim 12, wherein, In the case where the feature extraction model includes multiple location encoding modules, the feature extraction model includes one or more stages after the second location encoding module, each stage including: a location encoding module with downsampling, wherein determining the output features of the location encoding module based on the weighted summation result and the input features of the location encoding module includes: For each position encoding module with downsampling, the input features of the position encoding module are mapped to features with a preset number of channels; The features of the preset number of channels are downsampled according to a preset downsampling frequency, wherein the preset downsampling frequency corresponding to each position encoding module after the second position encoding module decreases sequentially. Based on the weighted summation result and the characteristics of the preset number of channels after downsampling, the output characteristics of the encoding module at this position are determined.

14. The processing method according to claim 1, further comprising: Based on the characteristics of the 3D point cloud data, the category of the target corresponding to the 3D point cloud data is determined.

15. The processing method according to claim 1, further comprising: The three-dimensional point cloud data is segmented based on its characteristics.

16. A method for processing three-dimensional point cloud data, comprising: The sample 3D point cloud data is input into the feature extraction model, wherein the sample 3D point cloud data is configured with corresponding annotation information, and the feature extraction model includes one or more position encoding modules; In each position encoding module, based on the input features of the position encoding module, the first encoding method, and the second encoding method, a first relative position encoding and a second relative position encoding of the sample 3D point cloud data in the position encoding module are determined. The first encoding method is the relative position encoding method when the target rotates relative to the acquisition device when the sample 3D point cloud data is acquired, and the second encoding method is the relative position encoding method when the target's position relative to the acquisition device remains unchanged when the sample 3D point cloud data is acquired. In each location encoding module, based on an attention mechanism, the weights of the first encoding method and the second encoding method corresponding to each point in the sample 3D point cloud data in that location encoding module are determined; Based on the weights of the first and second encoding methods corresponding to each point in the sample 3D point cloud data in the position encoding module, and the first and second relative position codes of the sample 3D point cloud data, the output features of the position encoding module are determined. The features of the sample 3D point cloud data are determined based on the output features of the last position encoding module. Based on the characteristics of the sample 3D point cloud data, determine the classification result of each point in the sample 3D point cloud data; The feature extraction model is trained based on the classification results and corresponding annotation information of each point in the sample 3D point cloud data.

17. A feature extraction model, comprising: The input module includes one or more cascaded position encoding modules, and the output module includes... The input module is used to receive input three-dimensional point cloud data; Each position encoding module is used to determine the first relative position encoding and the second relative position encoding of the three-dimensional point cloud data in the position encoding module according to the input features of the position encoding module, the first encoding method and the second encoding method. The first encoding method is the relative position encoding method when the target rotates relative to the acquisition device when acquiring the three-dimensional point cloud data, and the second encoding method is the relative position encoding method when the target's position relative to the acquisition device remains unchanged when acquiring the three-dimensional point cloud data. In each location encoding module, based on an attention mechanism, the weights of the first encoding method and the second encoding method corresponding to each point in the 3D point cloud data in that location encoding module are determined; based on the weights of the first encoding method and the second encoding method corresponding to each point in the 3D point cloud data in that location encoding module, as well as the first relative location encoding and the second relative location encoding of the 3D point cloud data, the output features of that location encoding module are determined. The output module is used to output the features of the three-dimensional point cloud data based on the output features of the last position encoding module.

18. A processing device for three-dimensional point cloud data, comprising: The acquisition module is used to acquire 3D point cloud data; The encoding module is used to determine a first relative position encoding and a second relative position encoding of the three-dimensional point cloud data according to a first encoding method and a second encoding method, wherein the first encoding method is a relative position encoding method when the target rotates relative to the acquisition device during the acquisition of the three-dimensional point cloud data, and the second encoding method is a relative position encoding method when the target's position relative to the acquisition device remains unchanged during the acquisition of the three-dimensional point cloud data. An attention module is used, in each location encoding module, to determine, based on an attention mechanism, the weights of the first encoding method and the second encoding method corresponding to each point in the 3D point cloud data in that location encoding module; The extraction module is used to extract features of the three-dimensional point cloud data based on the weights of the first encoding method and the second encoding method corresponding to each point in the three-dimensional point cloud data, as well as the first relative position encoding and the second relative position encoding of the three-dimensional point cloud data.

19. A processing apparatus for three-dimensional point cloud data, comprising: An input module is used to input sample 3D point cloud data into a feature extraction model, wherein the sample 3D point cloud data is configured with corresponding annotation information, and the feature extraction model includes one or more position encoding modules; An encoding module is used to determine, in each position encoding module, a first relative position encoding and a second relative position encoding of the sample 3D point cloud data in that position encoding module, based on the input features of that position encoding module, a first encoding method, and a second encoding method. The first encoding method is a relative position encoding method when the target rotates relative to the acquisition device during the acquisition of the sample 3D point cloud data, and the second encoding method is a relative position encoding method when the target's position relative to the acquisition device remains unchanged during the acquisition of the sample 3D point cloud data. An attention module is used, in each location encoding module, to determine, based on an attention mechanism, the weights of the first encoding method and the second encoding method corresponding to each point in the sample 3D point cloud data in that location encoding module; The determination module is used to determine the output features of the position encoding module based on the weights of the first encoding method and the second encoding method corresponding to each point in the sample 3D point cloud data in the position encoding module, as well as the first relative position encoding and the second relative position encoding of the sample 3D point cloud data; and to determine the features of the sample 3D point cloud data based on the output features of the last position encoding module. The classification module is used to determine the classification result of each point in the sample 3D point cloud data based on the characteristics of the sample 3D point cloud data. The training module is used to train the feature extraction model based on the classification results and corresponding annotation information of each point in the sample 3D point cloud data.

20. A processing device for three-dimensional point cloud data, comprising: processor; as well as A memory coupled to the processor is used to store instructions that, when executed by the processor, cause the processor to perform the method for processing three-dimensional point cloud data as described in any one of claims 1-16.

21. A non-transitory computer-readable storage medium having a computer program stored thereon, wherein, When executed by a processor, the program performs the steps of the method according to any one of claims 1-16.

22. An unmanned vehicle includes: The apparatus for processing three-dimensional point cloud data according to any one of claims 18-20.

Images