Context-aware three-dimensional point cloud model processing method and device

Abstract

The present invention discloses a context-aware 3D point cloud model processing method and device, including: extracting multiple target points included in the 3D point cloud model; establishing a global weighted local coordinate system corresponding to each target point and generating multiple corresponding global anchor points; determining multiple neighborhood points corresponding to each target point from all points included in the 3D point cloud model, and projecting the neighborhood points and global anchor points corresponding to each target point into the corresponding global weighted local coordinate system to obtain projected global anchor points and projected neighborhood points; extracting the convolution features of each neighborhood point corresponding to each target point according to the projected global anchor points, projected neighborhood points and a pre-determined convolution feature extraction formula corresponding to each target point, and determining the rotation-invariant convolution operator corresponding to each target point according to the convolution features of all neighborhood points corresponding to each target point. It can be seen that the present invention can improve the calculation accuracy and reduce the calculation complexity while solving the anti-rotation problem.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional point cloud data processing, and in particular to a three-dimensional point cloud model processing method and device based on context perception. Background Art

[0002] In recent years, the rapid rise of deep learning technology has brought a qualitative leap in the performance improvement of computer vision tasks, especially the introduction of convolutional neural networks (CNN), which has achieved outstanding results in image processing. The accuracy of recognition, classification, retrieval, detection, segmentation and other tasks even exceeds that of the human eye. CNN can be trained and extract more effective features than traditional methods through data-driven methods with its powerful feature extraction capabilities. With the development of technologies such as virtual reality, robotics, and autonomous driving, the application of deep learning in three-dimensional computer vision has attracted widespread attention from researchers. Due to the sparsity and disorder of three-dimensional point cloud data, traditional deep learning networks cannot be directly applied to three-dimensional point cloud data. After deep learning technology was first applied to three-dimensional point cloud data processing in 2018, a series of three-dimensional deep learning technologies (such as PointNet++, DGCNN, PointCNN, ShellNet, etc.) have been continuously proposed, and the performance has been continuously improved, and good results have been achieved on many three-dimensional data sets such as ModelNet, ShapeNet, and Semantic3D.

[0003] In practice, it is found that since rotated three-dimensional data is not used in the process of training neural networks, when the three-dimensional data objects in actual tests are rotated, the above methods cannot effectively solve the problem of anti-rotation. In order to solve this problem, data augmentation is often added during the training process, that is, during the training process, the three-dimensional point cloud data is rotated randomly or along the z-axis, so that the problem of decreased accuracy caused by rotation can be alleviated in actual testing. However, in actual applications, three-dimensional objects can rotate freely along any axis and have infinite rotation conditions. Therefore, no matter how much random rotation data is added during the training process, it still cannot cover all rotation conditions in reality, and expanding the data during the training process also increases the training time. In addition, the methods for rotation problems proposed by researchers in the past two years, such as methods based on spherical projection and methods based on rotation-invariant geometric features, have problems of high computational complexity and low computational accuracy.

[0004] It can be seen that when processing three-dimensional point cloud models, it is particularly important to improve calculation accuracy and reduce calculation complexity while solving the anti-rotation problem. Summary of the invention

[0005] The present invention provides a context-aware three-dimensional point cloud model processing method and device, which can solve the anti-rotation problem in the process of processing the three-dimensional point cloud model while improving the calculation accuracy and reducing the calculation complexity.

[0006] The first aspect of the present invention discloses a context-aware 3D point cloud model processing method, the method comprising:

[0007] After determining the three-dimensional point cloud model to be processed, extracting a plurality of target points included in the three-dimensional point cloud model, wherein the three-dimensional point cloud model includes a plurality of points distributed in a three-dimensional space;

[0008] Establishing a global weighted local coordinate system corresponding to each of the target points, and generating a plurality of global anchor points corresponding to each of the target points, wherein different target points correspond to the same number of global anchor points;

[0009] Determine a plurality of neighboring points corresponding to each of the target points from all points included in the three-dimensional point cloud model, and project all of the neighboring points and all of the global anchor points corresponding to each of the target points into a global weighted local coordinate system corresponding to each of the target points, to obtain a plurality of projected global anchor points and a plurality of projected neighboring points corresponding to each of the target points;

[0010] According to a plurality of projected global anchor points, a plurality of projected neighborhood points and a predetermined convolution feature extraction formula corresponding to each of the target points, the convolution feature of each neighborhood point corresponding to each of the target points is extracted, and according to the convolution feature of each neighborhood point corresponding to each of the target points, a rotationally invariant convolution operator corresponding to each of the target points is determined;

[0011] Among them, the rotation invariant convolution operators corresponding to all the target points are used to input into the determined convolutional neural network to perform matching operations on the three-dimensional point cloud model.

[0012] As an optional embodiment, in the first aspect of the present invention, the method further comprises:

[0013] Inputting the rotation invariant convolution operators corresponding to all the target points into the determined convolutional neural network to perform matching target operations on the three-dimensional point cloud model;

[0014] The target operation includes one or more combinations of an item recognition operation, an item classification operation, an item retrieval operation, an item detection operation, and an item segmentation operation.

[0015] As an optional implementation manner, in the first aspect of the present invention, the establishing of a global weighted local coordinate system corresponding to each target point includes:

[0016] For any of the target points, construct a covariance matrix corresponding to the target point, and determine a unit eigenvector of the covariance matrix corresponding to the target point;

[0017] For any target point, a global weighted local coordinate system corresponding to the target point is constructed with the target point as the center and the unit eigenvector of the covariance matrix corresponding to the target point as three coordinate axes.

[0018] As an optional implementation, in the first aspect of the present invention, for any of the target points, before constructing a global weighted local coordinate system corresponding to the target point with the target point as the center and the unit eigenvector of the covariance matrix corresponding to the target point as the three coordinate axes, the method further includes:

[0019] For any of the target points, a global direction centered on the target point is calculated, where the global direction centered on the target point represents the direction indicated by the weighted vector formed by all remaining target points and the target point;

[0020] For any of the target points, if there is a unit eigenvector among all the unit eigenvectors of the covariance matrix corresponding to the target point, the angle between the unit eigenvector and the global direction centered on the target point is an obtuse angle, an inversion operation is performed on the direction of the unit eigenvector to update the direction of the unit eigenvector.

[0021] As an optional implementation, in the first aspect of the present invention, for any of the target points, the covariance matrix corresponding to the target point is:

[0022]

[0023] Among them, Σ q is the covariance matrix corresponding to the target point, p represents the vector corresponding to the target point, N is the number of all remaining target points except the target point, w i is the weight corresponding to the i-th target point among all remaining target points, q i represents the vector corresponding to the i-th target point among all the remaining target points, and T represents the transpose of the matrix; the weight corresponding to the i-th target point among all the remaining target points is:

[0024]

[0025] Where m is the maximum distance between the target point and each remaining target point, i is an integer greater than or equal to 1 and less than or equal to N, ||q i -p|| represents the distance between the i-th target point and this target point among all remaining target points.

[0026] As an optional implementation, in the first aspect of the present invention, for any of the target points, the weighted vector formed by all remaining target points and the target point is expressed as:

[0027]

[0028] Among them, O q is the weighted vector formed by all remaining target points and the target point, p represents the vector corresponding to the target point, N is the number of all remaining target points except the target point, and w i is the weight corresponding to the i-th target point among all remaining target points, q i Represents the vector corresponding to the i-th target point among all remaining target points; the weight corresponding to the i-th target point among all remaining target points is:

[0029]

[0030] Wherein, m is the maximum distance between the target point and each remaining target point, and i is an integer greater than or equal to 1 and less than or equal to N.

[0031] As an optional implementation, in the first aspect of the present invention, extracting the convolution feature of each neighborhood point corresponding to each target point according to the multiple projected global anchor points, the multiple projected neighborhood points and the predetermined convolution feature extraction formula corresponding to each target point includes:

[0032] In a globally weighted local coordinate system corresponding to each of the target points, constructing a relationship feature between each projected neighborhood point corresponding to each of the target points and each projected global anchor point;

[0033] According to the relationship features between each projected neighborhood point corresponding to each of the target points and each projected global anchor point and a predetermined convolution feature extraction formula, the convolution features of each neighborhood point corresponding to each of the target points are extracted.

[0034] As an optional implementation, in the first aspect of the present invention, determining the rotation invariant convolution operator corresponding to each target point according to the convolution feature of each neighborhood point corresponding to each target point includes:

[0035] Performing a pooling operation on the convolution features of all the neighborhood points corresponding to each of the target points to extract significant features from the convolution features of all the neighborhood points corresponding to each of the target points;

[0036] The salient features among the extracted convolution features of all the neighborhood points corresponding to each of the target points are determined as the rotation invariant convolution operator corresponding to each of the target points.

[0037] As an optional implementation, in the first aspect of the present invention, for any of the target points, the relationship characteristics between each projected neighborhood point corresponding to the target point and each projected global anchor point are:

[0038] h(x' l ,a' n )=(x' l -a' n ,||x' l -a' n ||);

[0039] Among them, x' l is the coordinate of the lth projection neighborhood point corresponding to the target point, a' n is the coordinate of the nth projection global anchor point corresponding to the target point, h(x' l ,a' n ) is the feature relationship between the lth projected neighborhood point corresponding to the target point and the nth projected global anchor point corresponding to the target point; and the pre-determined convolution feature extraction formula is:

[0040]

[0041] Among them, K is a 1x8 convolution kernel, It is the convolution feature of the neighborhood point that matches the lth projected neighborhood point corresponding to the target point.

[0042] As an optional implementation, in the first aspect of the present invention, for any of the target points, the function for performing a pooling operation on the convolution features of all the neighborhood points corresponding to the target point is:

[0043]

[0044] Among them, Α represents the pooling operation and σ is the activation function.

[0045] As an optional implementation manner, in the first aspect of the present invention, generating a plurality of global anchor points corresponding to each of the target points includes:

[0046] For any of the target points, the three-dimensional point cloud model is divided into a plurality of quadrants according to a global weighted local coordinate system corresponding to the target point, so as to obtain a plurality of quadrants corresponding to the target point;

[0047] For any of the target points, the average coordinate values ​​of all points included in each quadrant corresponding to the target point in the global weighted local coordinate system corresponding to the target point are calculated to obtain the average coordinate value of each quadrant corresponding to the target point, wherein when the points included in the quadrant corresponding to the target point are empty, the average coordinate value of the quadrant corresponding to the target point is [0,0,0];

[0048] For any of the target points, the average coordinate value of each quadrant corresponding to the target point is determined as all global anchor points corresponding to the target point, and the number of global anchor points corresponding to the target point is equal to the number of quadrants corresponding to the target point.

[0049] As an optional implementation, in the first aspect of the present invention, extracting a plurality of target points included in the three-dimensional point cloud model includes:

[0050] After a certain point is extracted from the three-dimensional point cloud model, a point with the largest distance value from the certain point is extracted from all the remaining points that have not been extracted as the next point of the certain point, the next point of the extracted certain point is determined as a new point extracted from the three-dimensional point cloud model, and the operation of extracting the point with the largest distance value from the certain point from all the remaining points that have not been extracted as the next point of the certain point is triggered to execute until a preset number of target points are extracted from the three-dimensional point cloud model.

[0051] A second aspect of the present invention discloses a context-aware 3D point cloud model processing device, the device comprising:

[0052] A first extraction module is used to extract a plurality of target points included in the three-dimensional point cloud model after determining the three-dimensional point cloud model to be processed, wherein the three-dimensional point cloud model includes a plurality of points distributed in a three-dimensional space;

[0053] An establishment module, used for establishing a global weighted local coordinate system corresponding to each of the target points;

[0054] A generating module, used for generating a plurality of global anchor points corresponding to each of the target points, wherein different target points correspond to the same number of global anchor points;

[0055] A first determination module is used to determine a plurality of neighborhood points corresponding to each of the target points from all the points included in the three-dimensional point cloud model;

[0056] A projection module, used for projecting all the neighborhood points and all the global anchor points corresponding to each of the target points into a globally weighted local coordinate system corresponding to each of the target points, to obtain a plurality of projected global anchor points and a plurality of projected neighborhood points corresponding to each of the target points;

[0057] A second extraction module is used to extract the convolution feature of each neighborhood point corresponding to each target point according to a plurality of projected global anchor points, a plurality of projected neighborhood points and a predetermined convolution feature extraction formula corresponding to each target point;

[0058] A second determination module is used to determine the rotation invariant convolution operator corresponding to each target point according to the convolution feature of each neighborhood point corresponding to each target point;

[0059] Among them, the rotation invariant convolution operators corresponding to all the target points are used to input into the determined convolutional neural network to perform matching operations on the three-dimensional point cloud model.

[0060] As an optional implementation, in the second aspect of the present invention, the device further includes:

[0061] An output module, used for inputting the rotation invariant convolution operators corresponding to all the target points into the determined convolutional neural network to perform matching target operations on the three-dimensional point cloud model;

[0062] The target operation includes one or more combinations of an item recognition operation, an item classification operation, an item retrieval operation, an item detection operation, and an item segmentation operation.

[0063] As an optional implementation, in the second aspect of the present invention, the establishment module includes:

[0064] A construction submodule, used for constructing a covariance matrix corresponding to any of the target points;

[0065] A first determination submodule is used to determine, for any target point, a unit eigenvector of a covariance matrix corresponding to the target point;

[0066] The construction submodule is also used to construct, for any of the target points, a global weighted local coordinate system corresponding to the target point, with the target point as the center and the unit eigenvector of the covariance matrix corresponding to the target point as three coordinate axes.

[0067] As an optional implementation, in the second aspect of the present invention, the establishment module further includes:

[0068] The inversion submodule is used to calculate the global direction centered on any of the target points, the global direction centered on the target point represents the direction indicated by the weighted vector formed by all remaining target points and the target point, if there is a unit eigenvector whose angle with the global direction centered on the target point is an obtuse angle among all the unit eigenvectors of the covariance matrix corresponding to the target point, an inversion operation is performed on the direction of the unit eigenvector to update the direction of the unit eigenvector.

[0069] As an optional implementation, in the second aspect of the present invention, for any of the target points, the covariance matrix corresponding to the target point is:

[0070]

[0071] Among them, Σ q is the covariance matrix corresponding to the target point, p represents the vector corresponding to the target point, N is the number of all remaining target points except the target point, w i is the weight corresponding to the i-th target point among all remaining target points, q i represents the vector corresponding to the i-th target point among all the remaining target points, and T represents the transpose of the matrix; the weight corresponding to the i-th target point among all the remaining target points is:

[0072]

[0073] Where m is the maximum distance between the target point and each remaining target point, i is an integer greater than or equal to 1 and less than or equal to N, ||q i -p|| represents the distance between the i-th target point and this target point among all remaining target points.

[0074] As an optional implementation, in the second aspect of the present invention, for any of the target points, the weighted vector formed by all remaining target points and the target point is expressed as:

[0075]

[0076] Among them, O q is the weighted vector formed by all remaining target points and the target point, p represents the vector corresponding to the target point, N is the number of all remaining target points except the target point, and w i is the weight corresponding to the i-th target point among all remaining target points, q i Represents the vector corresponding to the i-th target point among all remaining target points; the weight corresponding to the i-th target point among all remaining target points is:

[0077]

[0078] Wherein, m is the maximum distance between the target point and each remaining target point, and i is an integer greater than or equal to 1 and less than or equal to N.

[0079] As an optional implementation, in the second aspect of the present invention, the second extraction module extracts the convolution feature of each neighborhood point corresponding to each target point according to the multiple projected global anchor points, the multiple projected neighborhood points and the predetermined convolution feature extraction formula corresponding to each target point in a specific manner as follows:

[0080] In a globally weighted local coordinate system corresponding to each of the target points, constructing a relationship feature between each projected neighborhood point corresponding to each of the target points and each projected global anchor point;

[0081] According to the relationship features between each projected neighborhood point corresponding to each of the target points and each projected global anchor point and a predetermined convolution feature extraction formula, the convolution features of each neighborhood point corresponding to each of the target points are extracted.

[0082] As an optional implementation, in the second aspect of the present invention, the second determination module determines the rotation invariant convolution operator corresponding to each target point according to the convolution feature of each neighborhood point corresponding to each target point in a specific manner as follows:

[0083] Performing a pooling operation on the convolution features of all the neighborhood points corresponding to each of the target points to extract significant features from the convolution features of all the neighborhood points corresponding to each of the target points;

[0084] The salient features among the extracted convolution features of all the neighborhood points corresponding to each of the target points are determined as the rotation invariant convolution operator corresponding to each of the target points.

[0085] As an optional implementation, in the second aspect of the present invention, for any of the target points, the relationship characteristics between each projected neighborhood point corresponding to the target point and each projected global anchor point are:

[0086] h(x' l ,a' n )=(x' l -a' n ,||x' l -a' n ||);

[0087] Among them, x' l is the coordinate of the lth projection neighborhood point corresponding to the target point, a' n is the coordinate of the nth projection global anchor point corresponding to the target point, h(x' l ,a' n ) is the feature relationship between the lth projected neighborhood point corresponding to the target point and the nth projected global anchor point corresponding to the target point; and the predetermined convolution feature extraction formula is:

[0088]

[0089] Among them, K is a 1x8 convolution kernel, It is the convolution feature of the neighborhood point that matches the lth projected neighborhood point corresponding to the target point.

[0090] As an optional implementation, in the second aspect of the present invention, for any of the target points, the function for performing a pooling operation on the convolution features of all the neighborhood points corresponding to the target point is:

[0091]

[0092] Among them, Α represents the pooling operation and σ is the activation function.

[0093] As an optional implementation, in the second aspect of the present invention, the generating module includes:

[0094] A division submodule, for dividing the three-dimensional point cloud model into a plurality of quadrants for any of the target points according to a global weighted local coordinate system corresponding to the target point, to obtain a plurality of quadrants corresponding to the target point;

[0095] A calculation submodule is used for obtaining, for any of the target points, the average value of the coordinate values ​​of all points included in each quadrant corresponding to the target point in the global weighted local coordinate system corresponding to the target point, to obtain the average coordinate value of each quadrant corresponding to the target point, wherein when the points included in the quadrant corresponding to the target point are empty, the average coordinate value of the quadrant corresponding to the target point is [0,0,0];

[0096] The second determination submodule is used to determine, for any of the target points, the average coordinate value of each quadrant corresponding to the target point as all global anchor points corresponding to the target point, and the number of global anchor points corresponding to the target point is equal to the number of quadrants corresponding to the target point.

[0097] As an optional implementation, in the second aspect of the present invention, the specific manner in which the first extraction module extracts the multiple target points included in the three-dimensional point cloud model is:

[0098] After a certain point is extracted from the three-dimensional point cloud model, a point with the largest distance value from the certain point is extracted from all the remaining points that have not been extracted as the next point of the certain point, the next point of the extracted certain point is determined as a new point extracted from the three-dimensional point cloud model, and the operation of extracting the point with the largest distance value from the certain point from all the remaining points that have not been extracted as the next point of the certain point is triggered to execute until a preset number of target points are extracted from the three-dimensional point cloud model.

[0099] The third aspect of the present invention discloses another context-aware 3D point cloud model processing device, the device comprising:

[0100] A memory storing executable program code;

[0101] a processor coupled to the memory;

[0102] The processor calls the executable program code stored in the memory to execute part or all of the steps in the context-aware three-dimensional point cloud model processing method disclosed in the first aspect of the present invention.

[0103] The fourth aspect of the present invention discloses a computer-storable medium, which stores computer instructions. When the computer instructions are called, they are used to execute part or all of the steps in the context-aware three-dimensional point cloud model processing method disclosed in the first aspect of the present invention.

[0104] Compared with the prior art, the present invention has the following beneficial effects:

[0105] The present invention can evenly extract some target points from all the points included in the three-dimensional point cloud model, so that the original three-dimensional point cloud model can be abstracted while reducing the amount of calculation; after extracting the target points, a global weighted local coordinate system corresponding to each target point is constructed, and the global weighted local coordinate system has high robustness and can resist noise; after determining the unit eigenvector of the global weighted local coordinate system, the direction of the corresponding unit eigenvector can be corrected according to the global direction centered on the target point to improve the accuracy of the constructed global weighted local coordinate system; after constructing the global weighted local coordinate system, the three-dimensional point cloud model can be divided into multiple quadrants (such as eight quadrants) with the target point as the center, so that different parts of the three-dimensional point cloud model are divided into different quadrants, and the average coordinate values ​​of all points in each quadrant are obtained to obtain useful global information, which is conducive to improving the accuracy and reliability of the subsequent extracted rotation invariant convolution operator; after the global weighted local coordinate system is constructed and the global anchor point is generated After that, the global anchor point and neighborhood points corresponding to the target point are projected into the global weighted local coordinate system. No matter how the three-dimensional point cloud model is rotated, the relevant points in the three-dimensional point cloud model are unified in the same coordinate system for operation to achieve rotation invariance, solving the problem of anti-rotation. After the convolution feature of each neighborhood point is extracted according to the predetermined convolution feature extraction formula, a pooling operation is performed on the convolution features of all neighborhood points to extract the most significant features as the rotation invariant convolution operator corresponding to the target point (that is, the global context-aware rotation invariant convolution operator), and the rotation invariant convolution operator corresponding to the target point can be further integrated into the convolutional neural network to achieve matching operations, such as classification operations, segmentation operations, etc., without the need to train the convolutional neural network through a large amount of data expansion. While reducing the computational complexity, the computational accuracy is greatly improved, which is beneficial to saving time and computing resources, and can also improve the practicality, stability and usability of the method described in the present invention in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0106] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0107] Figure 1 It is a flowchart of a context-aware 3D point cloud model processing method disclosed in an embodiment of the present invention;

[0108] Figure 2 It is a flowchart of another context-aware 3D point cloud model processing method disclosed in an embodiment of the present invention;

[0109] Figure 3 is a comparison chart of classification results of the classification experiment disclosed in the embodiment of the present invention;

[0110] Figure 4 is a comparison diagram of segmentation results of a segmentation experiment disclosed in an embodiment of the present invention;

[0111] Figure 5 It is a structural schematic diagram of a context-aware three-dimensional point cloud model processing device disclosed in an embodiment of the present invention;

[0112] Figure 6 It is a structural schematic diagram of another context-aware three-dimensional point cloud model processing device disclosed in an embodiment of the present invention;

[0113] Figure 7 It is a structural schematic diagram of another context-aware three-dimensional point cloud model processing device disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0114] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0115] The terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish different objects rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, device, product or end including a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units inherent to these processes, methods, products or ends.

[0116] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0117] The present invention discloses a context-aware 3D point cloud model processing method and device, which can combine local rotation-invariant geometric features with global context information, and designs a global context-aware rotation-invariant convolution operator (Global Context Aware Convolution, GCAConv) based on this, and further integrates the rotation-invariant convolution operator into a convolutional neural network, which can be used in many applications such as classification and segmentation of 3D point cloud models, and no longer needs to train the convolutional neural network through a large amount of data expansion. While achieving rotation invariance, it can reduce the computational complexity and greatly improve the computational accuracy, which is conducive to saving time and computing resources, and can also improve the practicality, stability and usability of the method described in the present invention in practical applications. The following are detailed descriptions.

[0118] Embodiment 1

[0119] See also Figure 1 , Figure 1 is a flow chart of a context-aware 3D point cloud model processing method disclosed in an embodiment of the present invention. Figure 1 The described method can be applied to corresponding processing devices, such as processing terminals, processing equipment or servers, etc., and the embodiments of the present invention are not limited thereto. Figure 1 As shown, the context-aware 3D point cloud model processing method may include the following operations:

[0120] 101. After determining a three-dimensional point cloud model to be processed, a processing device extracts a plurality of target points included in the three-dimensional point cloud model.

[0121] The three-dimensional point cloud model includes a plurality of points distributed in the three-dimensional space, and the processing device extracts a plurality of target points from all the points included in the three-dimensional point cloud model, thereby reducing the amount of calculation and abstracting the original three-dimensional point cloud model.

[0122] As an optional implementation manner, the processing device extracts multiple target points included in the three-dimensional point cloud model, which may include:

[0123] After extracting a certain point from the three-dimensional point cloud model, the processing device extracts the point with the largest distance value from the certain point from all the remaining points that have not been extracted as the next point of the certain point, determines the next point of the extracted certain point as a new point extracted from the three-dimensional point cloud model, and repeatedly triggers the execution of the operation of extracting the point with the largest distance value from the certain point from all the remaining points that have not been extracted as the next point of the certain point, until a preset number of target points are extracted from the three-dimensional point cloud model.

[0124] Optionally, the preset number may be a pre-set number, or a number determined according to the number of all points included in the three-dimensional point cloud model, which is not limited in the embodiment of the present invention.

[0125] Optionally, when any point included in the three-dimensional point cloud model is not selected, the processing device extracts a point from the three-dimensional point cloud model, which may include:

[0126] The processing device randomly selects a point from all points included in the three-dimensional point cloud model; or,

[0127] The processing device selects a point that meets preset conditions from all points included in the three-dimensional point cloud model.

[0128] Further optionally, the processing device selects a point that meets a preset condition from all points included in the three-dimensional point cloud model, which may include:

[0129] The processing device determines a plane model of the three-dimensional point cloud model at a predetermined recognition angle;

[0130] The processing device identifies the boundary of the plane model and determines the central area of ​​the plane model based on the boundary of the plane model;

[0131] The processing device selects the center point of the central area or randomly selects a point from the central area as a point extracted from the three-dimensional point cloud model.

[0132] It can be seen that in this optional implementation, when extracting multiple target points from all the points included in the three-dimensional point cloud model, the first point can be extracted first, and then the second point farthest from the first point can be extracted from the remaining points, and then the third point farthest from the second point can be extracted from all the remaining points that have not been extracted, and so on, until a preset number of target points are extracted. This method of extracting target points can achieve uniform extraction of target points from all the points included in the three-dimensional point cloud model, thereby improving the accuracy and reliability of the abstraction of the original three-dimensional point cloud model.

[0133] 102. The processing device establishes a global weighted local coordinate system corresponding to each target point, and generates multiple global anchor points corresponding to each target point.

[0134] In the embodiment of the present invention, each target point corresponds to a globally weighted local coordinate system (Local Reference Frame, LRF), and different target points correspond to the same number of global anchor points.

[0135] 103. The processing device determines a plurality of neighborhood points corresponding to each target point from all points included in the three-dimensional point cloud model, and projects all neighborhood points and all global anchor points corresponding to each target point into a global weighted local coordinate system corresponding to each target point, to obtain a plurality of projected global anchor points and a plurality of projected neighborhood points corresponding to each target point.

[0136] 104. The processing device extracts the convolution feature of each neighborhood point corresponding to each target point according to multiple projected global anchor points, multiple projected neighborhood points and a predetermined convolution feature extraction formula corresponding to each target point.

[0137] 105. The processing device determines a rotationally invariant convolution operator corresponding to each target point according to the convolution feature of each neighborhood point corresponding to each target point.

[0138] In an embodiment of the present invention, after generating a global anchor point and establishing a globally weighted LRF, a convolution operator can be designed to extract features. Taking a certain target point as an example, the neighborhood in the surrounding circle is used as the local information of the target point for convolution operation, which is determined by the inherent definition of the convolution operation, that is, the current target point and its corresponding neighborhood. In order to achieve rotation invariance, each neighborhood point in the neighborhood and each global anchor point of the target point are first projected to the same coordinate system through the globally weighted LRF corresponding to the target point. Since no matter how the data is rotated, it is eventually unified in the same coordinate system for calculation, through this operation, rotation invariance can be achieved. It should be noted that the size of the surrounding circle can be determined by the distance value of the neighborhood point from the target point and the number of required neighborhood points.

[0139] In the embodiment of the present invention, the rotation invariant convolution operators corresponding to all target points (also referred to as global context-aware rotation invariant convolution operators) are used to input the determined convolutional neural network to perform matching operations on the three-dimensional point cloud model. Specifically, the rotation invariant convolution operators corresponding to all target points are used to input the determined convolutional neural network, and the convolutional neural network performs multi-level feature extraction. For example: the extracted feature dimension is Ni X Ci, Ni is the number of feature points, set to 512, 128, 32, Ci is the dimension of the feature, set to 128, 256 and 512, and i is an integer greater than or equal to 0 and less than or equal to 2.

[0140] It can be seen that the method described in the embodiment of the present invention can project the global anchor points and neighborhood points corresponding to the target point into the global weighted local coordinate system after constructing the global weighted local coordinate system and generating the global anchor points. No matter how the three-dimensional point cloud model rotates, the relevant points in the three-dimensional point cloud model are unified in the same coordinate system for calculation to achieve rotation invariance, solving the problem of anti-rotation; and there is no need to train the convolutional neural network through a large amount of data expansion, while reducing the computational complexity, the computational accuracy is greatly improved, which is conducive to saving time and computing resources, and can also improve the practicality, stability and usability of the method described in the present invention in practical applications. In addition, it is also possible to evenly extract some target points from all the points included in the three-dimensional point cloud model, so that the original three-dimensional point cloud model can be abstracted while reducing the amount of calculation, and after extracting the target points, a global weighted local coordinate system corresponding to each target point is constructed. The global weighted local coordinate system has high robustness and can resist noise.

[0141] Embodiment 2

[0142] See also Figure 2 , Figure 2 is a flow chart of another context-aware 3D point cloud model processing method disclosed in an embodiment of the present invention. Figure 2 The described method can be applied to corresponding processing devices, such as processing terminals, processing equipment or servers, etc., and the embodiments of the present invention are not limited thereto. Figure 2 As shown, the context-aware 3D point cloud model processing method may include the following operations:

[0143] 201. After determining the three-dimensional point cloud model to be processed, the processing device extracts a plurality of target points included in the three-dimensional point cloud model.

[0144] In the embodiment of the present invention, for the detailed description of step 201, please refer to the detailed description of step 101 in the first embodiment, and the embodiment of the present invention will not be repeated.

[0145] 202. For any target point, the processing device constructs a covariance matrix corresponding to the target point, and determines a unit eigenvector of the covariance matrix corresponding to the target point.

[0146] In the embodiment of the present invention, after step 202 is executed, step 203 may be triggered, or step 204 may be triggered, and the former is preferred.

[0147] Optionally, for any target point, the covariance matrix corresponding to the target point can be expressed as:

[0148]

[0149] Among them, Σ q is the covariance matrix corresponding to the target point, p represents the vector corresponding to the target point, N is the number of all remaining target points except the target point, w i is the weight corresponding to the i-th target point among all remaining target points, q i represents the vector corresponding to the i-th target point among all the remaining target points, and T represents the transpose of the matrix; the weight corresponding to the i-th target point among all the remaining target points can be expressed as:

[0150]

[0151] Where m is the maximum distance between the target point and each remaining target point, i is an integer greater than or equal to 1 and less than or equal to N, ||q i -p|| represents the distance between the i-th target point and this target point among all remaining target points. i The calculation formula (1-2) indicates that the farther the remaining target points are from the target point, the smaller the corresponding weights are. In this way, the covariance matrix calculated by the above formula (1-1) is highly robust and resistant to noise.

[0152] 203. For any target point, the processing device calculates the global direction centered on the target point. If there is a unit eigenvector whose angle with the global direction centered on the target point is an obtuse angle among all the unit eigenvectors of the covariance matrix corresponding to the target point, an inversion operation is performed on the direction of the unit eigenvector to update the direction of the unit eigenvector.

[0153] In the embodiment of the present invention, due to the inherent error in eigenvalue calculation, the unit eigenvector of the covariance matrix calculated according to the above formula (1-1) has a sign (i.e., direction) reversal problem. For example, in different calculations, the positive x-axis will become the negative x-axis, and the corresponding y-axis and z-axis will also have the same sign reversal. Therefore, in order to solve this problem, the embodiment of the present invention can correct the sign-reversed unit eigenvector through a global direction centered on the current target point.

[0154] Among them, for any target point, the global direction centered on the target point represents the direction pointed by the weighted vector formed by all remaining target points and the target point. Further, for any target point, the weighted vector formed by all remaining target points and the target point can be expressed as:

[0155]

[0156] Among them, O q is the weighted vector formed by all remaining target points and the target point, p represents the vector corresponding to the target point, N is the number of all remaining target points except the target point, and w i is the weight corresponding to the i-th target point among all the remaining target points. Its calculation formula can be found in the above formula (1-2). i Represents the vector corresponding to the i-th target point among all remaining target points, where i is an integer greater than or equal to 1 and less than or equal to N.

[0157] 204. For any target point, the processing device constructs a global weighted local coordinate system corresponding to the target point with the target point as the center and the unit eigenvector of the covariance matrix corresponding to the target point as three coordinate axes.

[0158] In the embodiment of the present invention, after executing step 203, for any target point, all unit feature vectors whose angles with the global direction of the target point are obtuse are reversed, that is, for the unit feature vectors whose angles with the global direction of the target point are obtuse, in step 204, they are unit feature vectors whose directions have been corrected.

[0159] 205. The processing device generates multiple global anchor points corresponding to each target point.

[0160] As an optional implementation manner, the processing device generating multiple global anchor points corresponding to each target point may include:

[0161] For any target point, the processing device divides the three-dimensional point cloud model into a plurality of quadrants (preferably 8 quadrants) according to the global weighted local coordinate system corresponding to the target point, and obtains a plurality of quadrants corresponding to the target point;

[0162] For any target point, the processing device calculates the average value of the coordinate values ​​of all points included in each quadrant corresponding to the target point in the global weighted local coordinate system corresponding to the target point, and obtains the average coordinate value of each quadrant corresponding to the target point, wherein when the points included in the quadrant corresponding to the target point are empty, the average coordinate value of the quadrant corresponding to the target point is [0,0,0];

[0163] For any target point, the processing device determines the average coordinate value of each quadrant corresponding to the target point as all global anchor points corresponding to the target point, and the number of global anchor points corresponding to the target point is equal to the number of quadrants corresponding to the target point.

[0164] In this optional implementation, all global anchor points corresponding to all target points can be used as a rough representation of the three-dimensional point cloud model and carry useful global information, which can not only obtain useful global information but also reduce the amount of calculation. It should be noted that the method for obtaining global information is not limited to this optional implementation, and can also be implemented through other optional implementations, such as: dividing more cubes (i.e. quadrants) or using all points to obtain global information.

[0165] 206. The processing device determines a plurality of neighborhood points corresponding to each target point from all points included in the three-dimensional point cloud model, and projects all the neighborhood points and all the global anchor points corresponding to each target point into a global weighted local coordinate system corresponding to each target point, to obtain a plurality of projected global anchor points and a plurality of projected neighborhood points corresponding to each target point.

[0166] In the embodiment of the present invention, for any target point, the projected neighborhood point and the projected global anchor point can be respectively expressed as:

[0167] x' l =LRF(x l );a' n =LRF(a n ); (1-4)

[0168] Among them, x' l Indicates the lth neighboring point x corresponding to the target point l The projected neighborhood point, a' n is the nth global anchor point a corresponding to the target point n The projection global anchor point, l and n are both positive integers, and n is less than or equal to the total number of all quadrants corresponding to the target point.

[0169] 207. In the global weighted local coordinate system corresponding to each target point, the processing device constructs a relationship feature between each projected neighborhood point corresponding to each target point and each projected global anchor point.

[0170] Among them, for any target point, the relationship characteristics between each projected neighborhood point corresponding to the target point and each projected global anchor point are:

[0171] h(x' l ,a' n )=(x' l -a' n ,||x' l -a'n ||); (1-5)

[0172] Among them, x' l is the coordinate of the lth projection neighborhood point corresponding to the target point, a' n is the coordinate of the nth projection global anchor point corresponding to the target point, h(x' l ,a' n ) is the feature relationship between the lth projected neighborhood point corresponding to the target point and the nth projected global anchor point corresponding to the target point. And relative to each projected global anchor point, its relationship with the projected neighborhood point can be expressed as a 1x4 matrix, and 8 projected global anchor points are 8x4 matrices.

[0173] 208. The processing device extracts the convolution feature of each neighborhood point corresponding to each target point according to the relationship feature between each projected neighborhood point corresponding to each target point and each projected global anchor point and a predetermined convolution feature extraction formula.

[0174] Among them, for any target point, the predetermined convolution feature extraction formula is:

[0175]

[0176] Among them, K is a 1x8 convolution kernel, It is the convolution feature of the neighborhood point that matches the lth projected neighborhood point corresponding to the target point.

[0177] 209. The processing device performs a pooling operation on the convolution features of all the neighborhood points corresponding to each target point to extract significant features from the convolution features of all the neighborhood points corresponding to each target point, and determines the significant features from the extracted convolution features of all the neighborhood points corresponding to each target point as the rotationally invariant convolution operator corresponding to each target point.

[0178] In the embodiment of the present invention, for any target point, the function for performing a pooling operation on the convolution features of all the neighborhood points corresponding to the target point is:

[0179]

[0180] Among them, Α represents the pooling operation and σ is the activation function.

[0181] 210. The processing device inputs the rotation invariant convolution operators corresponding to all target points into the determined convolutional neural network to perform matching target operations on the three-dimensional point cloud model.

[0182] The processing device inputs the rotation invariant convolution operators corresponding to all target points into the determined convolutional neural network, and the convolutional neural network performs multi-level feature extraction. For example, the extracted feature dimension is Ni X Ci, Ni is the number of feature points, which is set to 512, 128, and 32, Ci is the dimension of the feature, which is set to 128, 256, and 512, and i is an integer greater than or equal to 0 and less than or equal to 2.

[0183] Optionally, the target operation includes a combination of one or more of an item recognition operation, an item classification operation, an item retrieval operation, an item detection operation, and an item segmentation operation.

[0184] The method described in the embodiment of the present invention is used to perform classification and segmentation experiments on point cloud data on the ModelNet40 and ShapeNet datasets. The results of the classification experiment on the ModelNet40 dataset can be referred to Figure 3 As shown in the table, the results of the segmentation experiment on the ShapeNet dataset can be referred to Figure 4 As shown in the table.

[0185] Specifically, the ModelNet40 dataset contains 9843 models for training convolutional neural networks and 2468 models for testing. The experiment designed three test modes: z / z, SO3 / SO3, and z / SO3. Among them, z indicates that the three-dimensional point cloud model is randomly rotated along the z-axis, and SO3 indicates that the three-dimensional point cloud model is rotated in any direction. z / z means that the z-axis rotation is added during training, and the z-axis random rotation is also performed on the three-dimensional point cloud model during testing; SO3 / SO3 means that the three-dimensional point cloud model is arbitrarily rotated during training, and the three-dimensional point cloud model is also rotated at any angle during testing; z / SO3 means that the z-axis rotation is added during training, and the three-dimensional point cloud model is rotated at any angle during testing. The difficulty gradually increases from z / z to z / SO3. From Figure 3 It can be seen from the experimental results shown that the method (Ours) described in the embodiment of the present invention achieves the best training parameters for the convolutional neural network, with the number of parameters being only 0.39M, which greatly reduces the complexity and achieves more improvements in computational accuracy. It is worth noting that the accuracy of the method (Ours) described in the embodiment of the present invention is very uniform in each test mode, while some existing methods perform well only in the z / z mode or SO3 / SO3 mode, but perform poorly in the z / SO3 mode, indicating that they are not truly rotationally invariant methods. On the other hand, the method (Ours) described in the embodiment of the present invention has achieved good performance in each mode, indicating that it is a truly rotationally invariant method.

[0186] Among them, the segmentation experiment is completed on the ShapeNet dataset, such as Figure 4 As shown, the method (Ours) described in the embodiment of the present invention exceeds the existing similar algorithms in terms of calculation accuracy.

[0187] For example, the method described in the embodiment of the present invention can be applied to scenes such as semantic segmentation of three-dimensional scenes, recognition and visual positioning of three-dimensional objects, etc., which are specifically described below with examples.

[0188] Scenario 1: Application to 3D visual positioning and robot disordered grasping. In the disordered grasping process of the robot arm, visual positioning is required to detect the position and posture of the target object. In the actual scene, the posture of the target object is varied, and the angle and placement posture of the target object cannot be predicted in advance. To solve this problem, the traditional method relies on the search of local features, such as first detecting a corner of a box or a corner of a sack, which is inefficient and has poor accuracy. Especially when the target object is deformed, it is often impossible to detect it accurately. The method proposed in the embodiment of the present invention can solve the rotation problem well, and the target object can be well detected regardless of whether the target object is rotated. Specifically, the three-dimensional scene can be semantically segmented to obtain the approximate position of the object, and the method described in the embodiment of the present invention can be used in the area where the object is located to perform instance segmentation of the object, so as to segment each object, and then classify the objects to obtain the similarity between each object and the target object, and select the object with the highest similarity as the target object for grasping. Since the method described in the embodiment of the present invention can solve the rotation problem well, no matter how the object in the scene rotates, it can be detected very accurately.

[0189] Scenario 2: Applied to obstacle recognition in autonomous driving systems. In complex environments, autonomous driving systems need to identify obstacles in order to take reasonable measures. In many cases, it is difficult to identify obstacles using only 2D colors and textures. 3D data can provide geometric features such as the shape and size of objects, which helps to correctly identify objects. Since obstacles may appear in random positions with random postures, the method described in the embodiment of the present invention is insensitive to the rotation or other posture changes of obstacles, and can improve the accuracy of obstacle recognition using 3D information in complex environmental conditions.

[0190] It can be seen that the method described in the embodiment of the present invention can evenly extract some target points from all the points included in the three-dimensional point cloud model, so that the original three-dimensional point cloud model can be abstracted while reducing the amount of calculation; after extracting the target point, a global weighted local coordinate system corresponding to each target point is constructed, and the global weighted local coordinate system has high robustness and can resist noise; after determining the unit eigenvector of the global weighted local coordinate system, the direction of the corresponding unit eigenvector can be corrected according to the global direction centered on the target point to improve the accuracy of the constructed global weighted local coordinate system; after constructing the global weighted local coordinate system, the three-dimensional point cloud model can be divided into multiple quadrants (such as eight quadrants) with the target point as the center, so that different parts of the three-dimensional point cloud model are divided into different quadrants, and the average coordinate values ​​of all points in each quadrant are obtained to obtain useful global information, which is beneficial to improve the accuracy and reliability of the subsequent extracted rotation invariant convolution operator; after the global weighted local coordinate system is constructed and After generating the global anchor point, the global anchor point and the neighborhood points corresponding to the target point are projected into the global weighted local coordinate system. No matter how the three-dimensional point cloud model is rotated, the relevant points in the three-dimensional point cloud model are unified in the same coordinate system for calculation to achieve rotation invariance, solving the problem of anti-rotation. After the convolution feature of each neighborhood point is extracted according to the predetermined convolution feature extraction formula, the convolution feature of all neighborhood points is pooled to extract the most significant features as the rotation invariant convolution operator corresponding to the target point (that is, the global context-aware rotation invariant convolution operator), and the rotation invariant convolution operator corresponding to the target point can be further integrated into the convolutional neural network to achieve matching operations, such as classification operations, segmentation operations, etc., without the need to train the convolutional neural network through a large amount of data expansion. While reducing the computational complexity, the computational accuracy is greatly improved, which is beneficial to saving time and computing resources, and can also improve the practicality, stability and usability of the method described in the present invention in practical applications.

[0191] Embodiment 3

[0192] See also Figure 5 , Figure 5 is a schematic diagram of the structure of a context-aware 3D point cloud model processing device disclosed in an embodiment of the present invention. Figure 5 The described apparatus may be applied to a processing terminal, a processing device, or a server, etc., and the embodiments of the present invention are not limited thereto. Figure 5 As shown, the context-aware three-dimensional point cloud model processing device may include:

[0193] The first extraction module 301 is used to extract a plurality of target points included in the three-dimensional point cloud model after determining the three-dimensional point cloud model to be processed. The three-dimensional point cloud model includes a plurality of points distributed in the three-dimensional space.

[0194] The establishment module 302 is used to establish a global weighted local coordinate system corresponding to each target point.

[0195] The generating module 303 is used to generate a plurality of global anchor points corresponding to each target point, wherein different target points correspond to the same number of global anchor points.

[0196] The first determination module 304 is used to determine a plurality of neighborhood points corresponding to each target point from all points included in the three-dimensional point cloud model.

[0197] The projection module 305 is used to project all the neighborhood points and all the global anchor points corresponding to each target point into the global weighted local coordinate system corresponding to each target point, so as to obtain multiple projected global anchor points and multiple projected neighborhood points corresponding to each target point.

[0198] The second extraction module 306 is used to extract the convolution feature of each neighborhood point corresponding to each target point according to multiple projected global anchor points, multiple projected neighborhood points and a pre-determined convolution feature extraction formula corresponding to each target point.

[0199] The second determination module 307 is used to determine the rotation invariant convolution operator corresponding to each target point according to the convolution feature of each neighborhood point corresponding to each target point.

[0200] Among them, the rotation invariant convolution operators corresponding to all target points are used to input the determined convolutional neural network to perform matching operations on the three-dimensional point cloud model.

[0201] It can be seen that implementation Figure 5 The described device can, after constructing a global weighted local coordinate system and generating a global anchor point, project the global anchor point and neighborhood points corresponding to the target point into the global weighted local coordinate system. No matter how the three-dimensional point cloud model is rotated, the relevant points in the three-dimensional point cloud model are unified in the same coordinate system for calculation to achieve rotation invariance, solving the problem of anti-rotation. There is no need to train the convolutional neural network through a large amount of data expansion. While reducing the computational complexity, the computational accuracy is greatly improved, which is beneficial to saving time and computing resources, and can also improve the practicality, stability and usability of the method described in the present invention in practical applications.

[0202] In an optional embodiment, if Figure 6 As shown, the device may also include:

[0203] The output module 308 is used to input the rotation invariant convolution operators corresponding to all target points into the determined convolutional neural network to perform matching target operations on the three-dimensional point cloud model.

[0204] The target operation includes one or more combinations of an item recognition operation, an item classification operation, an item retrieval operation, an item detection operation, and an item segmentation operation.

[0205] In another optional embodiment, Figure 6 As shown, the establishment module 302 may include:

[0206] A construction submodule 3021 is used to construct a covariance matrix corresponding to any target point;

[0207] The first determination submodule 3022 is used to determine, for any target point, the unit eigenvector of the covariance matrix corresponding to the target point;

[0208] The construction submodule 3021 is also used to construct, for any target point, a global weighted local coordinate system corresponding to the target point with the target point as the center and the unit eigenvector of the covariance matrix corresponding to the target point as the three coordinate axes.

[0209] Further optional, such as Figure 6 As shown, the establishment module 302 may also include:

[0210] The reverse submodule 3023 is used to calculate, for any target point, the global direction centered on the target point, where the global direction centered on the target point represents the direction indicated by the weighted vector formed by all remaining target points and the target point. If, among all the unit eigenvectors of the covariance matrix corresponding to the target point, there exists a unit eigenvector whose included angle with the global direction centered on the target point is an obtuse angle, an inversion operation is performed on the direction of the unit eigenvector to update the direction of the unit eigenvector.

[0211] In yet another optional embodiment, for any target point, the covariance matrix corresponding to the target point is:

[0212]

[0213] Among them, Σ q is the covariance matrix corresponding to the target point, p represents the vector corresponding to the target point, N is the number of all remaining target points except the target point, w i is the weight corresponding to the i-th target point among all remaining target points, q i represents the vector corresponding to the i-th target point among all the remaining target points, and T represents the transpose of the matrix; the weight corresponding to the i-th target point among all the remaining target points is:

[0214]

[0215] Where m is the maximum distance between the target point and each remaining target point, i is an integer greater than or equal to 1 and less than or equal to N, ||q i -p|| represents the distance between the i-th target point and this target point among all remaining target points.

[0216] In yet another optional embodiment, for any target point, the weighted vector formed by all remaining target points and the target point is expressed as:

[0217]

[0218] Among them, O q is the weighted vector formed by all remaining target points and the target point, p represents the vector corresponding to the target point, N is the number of all remaining target points except the target point, and w i is the weight corresponding to the i-th target point among all remaining target points, q i Represents the vector corresponding to the i-th target point among all remaining target points.

[0219] In another optional embodiment, the second extraction module 306 extracts the convolution feature of each neighborhood point corresponding to each target point according to the multiple projected global anchor points, the multiple projected neighborhood points and the pre-determined convolution feature extraction formula:

[0220] In the global weighted local coordinate system corresponding to each target point, the relationship features between each projected neighborhood point corresponding to each target point and each projected global anchor point are constructed;

[0221] According to the relationship features between each projected neighborhood point corresponding to each target point and each projected global anchor point and a predetermined convolution feature extraction formula, the convolution features of each neighborhood point corresponding to each target point are extracted.

[0222] In this optional embodiment, for any target point, the projected neighborhood point and the projected global anchor point can be expressed as:

[0223] x' l =LRF(x l );a' n =LRF(a n ); (1-4)

[0224] Among them, x' l Indicates the lth neighboring point x corresponding to the target point l The projected neighborhood point, a' n is the nth global anchor point a corresponding to the target point nThe projection global anchor point, l and n are both positive integers, and n is less than or equal to the total number of all quadrants corresponding to the target point.

[0225] In this optional embodiment, further optionally, for any target point, the relationship characteristics between each projected neighborhood point corresponding to the target point and each projected global anchor point are:

[0226] h(x' l ,a' n )=(x' l -a' n ,||x' l -a' n ||); (1-5)

[0227] Among them, x' l is the coordinate of the lth projection neighborhood point corresponding to the target point, a' n is the coordinate of the nth projection global anchor point corresponding to the target point, h(x' l ,a' n ) is the feature relationship between the lth projected neighborhood point corresponding to the target point and the nth projected global anchor point corresponding to the target point; and the pre-determined convolution feature extraction formula is:

[0228]

[0229] Among them, K is a 1x8 convolution kernel, It is the convolution feature of the neighborhood point that matches the lth projected neighborhood point corresponding to the target point.

[0230] In yet another optional embodiment, the second determination module 307 determines the rotation invariant convolution operator corresponding to each target point according to the convolution feature of each neighborhood point corresponding to each target point in a specific manner as follows:

[0231] Perform a pooling operation on the convolution features of all the neighborhood points corresponding to each target point to extract the significant features in the convolution features of all the neighborhood points corresponding to each target point;

[0232] The salient features among the extracted convolution features of all neighborhood points corresponding to each target point are determined as the rotation invariant convolution operator corresponding to each target point.

[0233] In this optional embodiment, further optionally, for any target point, the function for performing a pooling operation on the convolution features of all neighborhood points corresponding to the target point is:

[0234]

[0235] Among them, Α represents the pooling operation and σ is the activation function.

[0236] In yet another optional embodiment, Figure 6 As shown, the generation module 303 may include:

[0237] The division submodule 3031 is used to divide the three-dimensional point cloud model into multiple quadrants for any target point according to the global weighted local coordinate system corresponding to the target point, so as to obtain multiple quadrants corresponding to the target point.

[0238] The calculation submodule 3032 is used to obtain, for any target point, the average coordinate value of all points included in each quadrant corresponding to the target point in the global weighted local coordinate system corresponding to the target point, and obtain the average coordinate value of each quadrant corresponding to the target point, wherein when the points included in the quadrant corresponding to the target point are empty, the average coordinate value of the quadrant corresponding to the target point is [0,0,0];

[0239] The second determination submodule 3033 is used to determine, for any target point, the average coordinate value of each quadrant corresponding to the target point as all global anchor points corresponding to the target point, and the number of global anchor points corresponding to the target point is equal to the number of quadrants corresponding to the target point.

[0240] In yet another optional embodiment, the specific manner in which the first extraction module 301 extracts the multiple target points included in the three-dimensional point cloud model is:

[0241] After a certain point is extracted from the three-dimensional point cloud model, the point with the largest distance value from the certain point is extracted from all the remaining points that have not been extracted as the next point of the certain point, the next point of the extracted certain point is determined as a new point extracted from the three-dimensional point cloud model, and the operation of extracting the point with the largest distance value from the certain point from all the remaining points that have not been extracted as the next point of the certain point is triggered, until a preset number of target points are extracted from the three-dimensional point cloud model.

[0242] It can be seen that implementation Figure 6The described device can evenly extract some target points from all the points included in the three-dimensional point cloud model, so that the original three-dimensional point cloud model can be abstracted while reducing the amount of calculation; after extracting the target points, a global weighted local coordinate system corresponding to each target point is constructed, and the global weighted local coordinate system has high robustness and can resist noise; after determining the unit eigenvector of the global weighted local coordinate system, the direction of the corresponding unit eigenvector can be corrected according to the global direction centered on the target point to improve the accuracy of the constructed global weighted local coordinate system; after constructing the global weighted local coordinate system, the three-dimensional point cloud model can be divided into multiple quadrants (such as eight quadrants) with the target point as the center, so that different parts of the three-dimensional point cloud model are divided into different quadrants, and the average coordinate values ​​of all points in each quadrant are obtained to obtain useful global information, which is conducive to improving the accuracy and reliability of the subsequent extracted rotation invariant convolution operator; after the global weighted local coordinate system is constructed and the global anchor is generated After the point is extracted, the global anchor point and neighborhood points corresponding to the target point are projected into the global weighted local coordinate system. No matter how the 3D point cloud model is rotated, the relevant points in the 3D point cloud model are unified in the same coordinate system for operation to achieve rotation invariance, solving the problem of anti-rotation. After the convolution feature of each neighborhood point is extracted according to the predetermined convolution feature extraction formula, a pooling operation is performed on the convolution features of all neighborhood points to extract the most significant features as the rotation invariant convolution operator corresponding to the target point (that is, the global context-aware rotation invariant convolution operator), and the rotation invariant convolution operator corresponding to the target point can be further integrated into the convolutional neural network to achieve matching operations, such as classification operations, segmentation operations, etc., without the need to train the convolutional neural network through a large amount of data expansion. While reducing the computational complexity, the computational accuracy is greatly improved, which is beneficial to saving time and computing resources, and can also improve the practicality, stability and usability of the method described in the present invention in practical applications.

[0243] Embodiment 4

[0244] See also Figure 7 , Figure 7 is a schematic diagram of the structure of another context-aware 3D point cloud model processing device disclosed in an embodiment of the present invention. Figure 7 The described apparatus may be applied to a processing terminal, a processing device, or a server, etc., and the embodiments of the present invention are not limited thereto. Figure 7 As shown, the context-aware three-dimensional point cloud model processing device may include:

[0245] A memory 401 storing executable program codes;

[0246] a processor 402 coupled to the memory 401;

[0247] The processor 402 calls the executable program code stored in the memory 401 to execute the steps in the context-aware three-dimensional point cloud model processing method described in the first embodiment of the present invention or the second embodiment of the present invention.

[0248] Embodiment 5

[0249] An embodiment of the present invention discloses a computer-storable medium, which stores computer instructions. When the computer instructions are called, they are used to execute the steps in the context-aware three-dimensional point cloud model processing method described in Embodiment 1 or Embodiment 2 of the present invention.

[0250] Embodiment 6

[0251] An embodiment of the present invention discloses a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to enable a computer to execute the steps in the context-aware three-dimensional point cloud model processing method described in Example 1 or Example 2.

[0252] The device embodiments described above are only illustrative, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, i.e., they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative work.

[0253] Through the specific description of the above embodiments, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the above technical solution can be essentially or partly contributed to the prior art in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable rewritable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0254] Finally, it should be noted that the context-aware three-dimensional point cloud model processing method and device disclosed in the embodiment of the present invention only discloses the preferred embodiment of the present invention, which is only used to illustrate the technical solution of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments can still be modified, or some of the technical features can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A context-aware 3D point cloud model processing method, characterized in that: The method comprises: After determining the three-dimensional point cloud model to be processed, extracting a plurality of target points included in the three-dimensional point cloud model, wherein the three-dimensional point cloud model includes a plurality of points distributed in a three-dimensional space; Establishing a global weighted local coordinate system corresponding to each of the target points, and generating a plurality of global anchor points corresponding to each of the target points, wherein different target points correspond to the same number of global anchor points; Determine a plurality of neighboring points corresponding to each of the target points from all points included in the three-dimensional point cloud model, and project all of the neighboring points and all of the global anchor points corresponding to each of the target points into a global weighted local coordinate system corresponding to each of the target points, to obtain a plurality of projected global anchor points and a plurality of projected neighboring points corresponding to each of the target points; According to a plurality of projected global anchor points, a plurality of projected neighborhood points and a predetermined convolution feature extraction formula corresponding to each of the target points, the convolution feature of each neighborhood point corresponding to each of the target points is extracted, and according to the convolution feature of each neighborhood point corresponding to each of the target points, a rotationally invariant convolution operator corresponding to each of the target points is determined; wherein the rotationally invariant convolution operators corresponding to all the target points are used to input into the determined convolutional neural network to perform matching operations on the three-dimensional point cloud model; And, extracting the convolution feature of each neighborhood point corresponding to each target point according to the multiple projected global anchor points, the multiple projected neighborhood points and the predetermined convolution feature extraction formula corresponding to each target point, comprises: In a globally weighted local coordinate system corresponding to each of the target points, constructing a relationship feature between each projected neighborhood point corresponding to each of the target points and each projected global anchor point; Extracting the convolution feature of each neighborhood point corresponding to each target point according to the relationship feature between each projected neighborhood point corresponding to each target point and each projected global anchor point and a pre-determined convolution feature extraction formula; And, determining the rotation invariant convolution operator corresponding to each target point according to the convolution feature of each neighborhood point corresponding to each target point includes: Performing a pooling operation on the convolution features of all the neighborhood points corresponding to each of the target points to extract significant features from the convolution features of all the neighborhood points corresponding to each of the target points; Determine the significant features among the extracted convolution features of all neighborhood points corresponding to each of the target points as the rotation invariant convolution operator corresponding to each of the target points; For any of the target points, the relationship characteristics between each projected neighborhood point corresponding to the target point and each projected global anchor point are: h(x’ l ,a’ n )=(x’ l -a’ n ,||x’ l -a’ n ||); Among them, x' l is the coordinate of the lth projection neighborhood point corresponding to the target point, a' n is the coordinate of the nth projection global anchor point corresponding to the target point, h(x' l ,a' n ) is the feature relationship between the lth projected neighborhood point corresponding to the target point and the nth projected global anchor point corresponding to the target point; and the pre-determined convolution feature extraction formula is: Among them, K is a 1x8 convolution kernel, It is the convolution feature of the neighborhood point that matches the lth projected neighborhood point corresponding to the target point.

2. The context-aware 3D point cloud model processing method according to claim 1, characterized in that: The method further comprises: Inputting the rotation invariant convolution operators corresponding to all the target points into the determined convolutional neural network to perform matching target operations on the three-dimensional point cloud model; The target operation includes one or more combinations of an item recognition operation, an item classification operation, an item retrieval operation, an item detection operation, and an item segmentation operation.

3. The context-aware 3D point cloud model processing method according to claim 1, characterized in that: The establishing of a global weighted local coordinate system corresponding to each target point comprises: For any of the target points, construct a covariance matrix corresponding to the target point, and determine a unit eigenvector of the covariance matrix corresponding to the target point; For any target point, a global weighted local coordinate system corresponding to the target point is constructed with the target point as the center and the unit eigenvector of the covariance matrix corresponding to the target point as three coordinate axes.

4. The context-aware 3D point cloud model processing method according to claim 3, characterized in that: Before constructing a global weighted local coordinate system corresponding to any target point with the target point as the center and the unit eigenvector of the covariance matrix corresponding to the target point as three coordinate axes, the method further includes: For any of the target points, a global direction centered on the target point is calculated, where the global direction centered on the target point represents the direction indicated by the weighted vector formed by all remaining target points and the target point; For any of the target points, if there is a unit eigenvector among all the unit eigenvectors of the covariance matrix corresponding to the target point, the angle between the unit eigenvector and the global direction centered on the target point is an obtuse angle, an inversion operation is performed on the direction of the unit eigenvector to update the direction of the unit eigenvector.

5. The context-aware 3D point cloud model processing method according to claim 3 or 4, characterized in that: For any of the target points, the covariance matrix corresponding to the target point is: Among them, Σ q is the covariance matrix corresponding to the target point, p represents the vector corresponding to the target point, N is the number of all remaining target points except the target point, w i is the weight corresponding to the i-th target point among all remaining target points, q i represents the vector corresponding to the i-th target point among all the remaining target points, and T represents the transpose of the matrix; the weight corresponding to the i-th target point among all the remaining target points is: Where m is the maximum distance between the target point and each remaining target point, i is an integer greater than or equal to 1 and less than or equal to N, ||q i -p|| represents the distance between the i-th target point and this target point among all remaining target points.

6. The context-aware 3D point cloud model processing method according to claim 4, characterized in that: For any of the target points, the weighted vector formed by all remaining target points and the target point is expressed as: Among them, O q is the weighted vector formed by all remaining target points and the target point, p represents the vector corresponding to the target point, N is the number of all remaining target points except the target point, and w i is the weight corresponding to the i-th target point among all remaining target points, q i Represents the vector corresponding to the i-th target point among all remaining target points; the weight corresponding to the i-th target point among all remaining target points is: Wherein, m is the maximum distance between the target point and each remaining target point, and i is an integer greater than or equal to 1 and less than or equal to N.

7. The context-aware 3D point cloud model processing method according to claim 1, characterized in that: For any target point, the function of pooling the convolution features of all the neighboring points corresponding to the target point is: Among them, Α represents the pooling operation and σ is the activation function.

8. The context-aware 3D point cloud model processing method according to any one of claims 1, 2, 3, 4, 6 and 7, characterized in that: The generating of a plurality of global anchor points corresponding to each of the target points comprises: For any of the target points, the three-dimensional point cloud model is divided into a plurality of quadrants according to a global weighted local coordinate system corresponding to the target point, so as to obtain a plurality of quadrants corresponding to the target point; For any of the target points, the average coordinate values ​​of all points included in each quadrant corresponding to the target point in the global weighted local coordinate system corresponding to the target point are calculated to obtain the average coordinate value of each quadrant corresponding to the target point, wherein when the points included in the quadrant corresponding to the target point are empty, the average coordinate value of the quadrant corresponding to the target point is [0,0,0]; For any of the target points, the average coordinate value of each quadrant corresponding to the target point is determined as all global anchor points corresponding to the target point, and the number of global anchor points corresponding to the target point is equal to the number of quadrants corresponding to the target point.

9. The context-aware 3D point cloud model processing method according to any one of claims 1, 2, 3, 4, 6 and 7, characterized in that: The extracting of a plurality of target points included in the three-dimensional point cloud model comprises: After any point is extracted from the three-dimensional point cloud model, the point with the largest distance value from the any point is extracted from all the remaining points that have not been extracted as the next point of the any point, the next point of the extracted any point is determined as a new any point extracted from the three-dimensional point cloud model, and the operation of extracting the point with the largest distance value from the any point from all the remaining points that have not been extracted as the next point of the any point is triggered to execute until a preset number of target points are extracted from the three-dimensional point cloud model.

10. A context-aware 3D point cloud model processing device, characterized in that: The device comprises: A first extraction module is used to extract a plurality of target points included in the three-dimensional point cloud model after determining the three-dimensional point cloud model to be processed, wherein the three-dimensional point cloud model includes a plurality of points distributed in a three-dimensional space; An establishment module is used to establish a global weighted local coordinate system corresponding to each of the target points; A generating module, used for generating a plurality of global anchor points corresponding to each of the target points, wherein different target points correspond to the same number of global anchor points; A first determination module is used to determine a plurality of neighborhood points corresponding to each of the target points from all the points included in the three-dimensional point cloud model; A projection module, used for projecting all the neighborhood points and all the global anchor points corresponding to each of the target points into a globally weighted local coordinate system corresponding to each of the target points, to obtain a plurality of projected global anchor points and a plurality of projected neighborhood points corresponding to each of the target points; A second extraction module is used to extract the convolution feature of each neighborhood point corresponding to each target point according to a plurality of projected global anchor points, a plurality of projected neighborhood points and a predetermined convolution feature extraction formula corresponding to each target point; A second determination module is used to determine the rotation invariant convolution operator corresponding to each target point according to the convolution feature of each neighborhood point corresponding to each target point; wherein the rotationally invariant convolution operators corresponding to all the target points are used to input into the determined convolutional neural network to perform matching operations on the three-dimensional point cloud model; The specific manner in which the second extraction module extracts the convolution feature of each neighborhood point corresponding to each target point according to the multiple projected global anchor points, the multiple projected neighborhood points and the pre-determined convolution feature extraction formula corresponding to each target point is: In a globally weighted local coordinate system corresponding to each of the target points, constructing a relationship feature between each projected neighborhood point corresponding to each of the target points and each projected global anchor point; Extracting the convolution feature of each neighborhood point corresponding to each target point according to the relationship feature between each projected neighborhood point corresponding to each target point and each projected global anchor point and a pre-determined convolution feature extraction formula; And, the second determination module determines the rotation invariant convolution operator corresponding to each target point according to the convolution feature of each neighborhood point corresponding to each target point in a specific manner as follows: Performing a pooling operation on the convolution features of all the neighborhood points corresponding to each of the target points to extract significant features from the convolution features of all the neighborhood points corresponding to each of the target points; Determine the significant features among the extracted convolution features of all neighborhood points corresponding to each of the target points as the rotation invariant convolution operator corresponding to each of the target points; Among them, for any of the target points, the relationship characteristics between each projected neighborhood point corresponding to the target point and each projected global anchor point are: h(x’ l ,a’ n )=(x’ l -a’ n ,||x’ l -a’ n ||); Among them, x' l is the coordinate of the lth projection neighborhood point corresponding to the target point, a' n is the coordinate of the nth projection global anchor point corresponding to the target point, h(x' l ,a' n ) is the feature relationship between the lth projected neighborhood point corresponding to the target point and the nth projected global anchor point corresponding to the target point; and the pre-determined convolution feature extraction formula is: Among them, K is a 1x8 convolution kernel, It is the convolution feature of the neighborhood point that matches the lth projected neighborhood point corresponding to the target point.

Images