A method and device for extracting point cloud features based on complex network representation

Abstract

A method and device for extracting point cloud features based on complex network representation. The method includes: First step, using Hilbert transform to obtain the processing of the real and imaginary parts in the complex representation of the point cloud; Second step, using the result of the first step to perform convolutions on the real and imaginary parts in the real number domain respectively, and then calculating the final feature representations of the real and imaginary parts through the complex convolution formula, and multiple layers can be iterated to achieve the extraction of features with more complex and definite number of outputs; Third step, the complex representation combining the real and imaginary features of the point cloud can be calculated through the result in the second step, and obtained through modulus calculation; Fourth step, further perform operations such as normalization or feature encoding dimensionality reduction on the result of the third step to obtain a compact and representative global feature, providing a basic guarantee for subsequent three-dimensional point cloud processing and analysis tasks.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional processing and analysis, and more particularly to a method and apparatus for extracting point cloud features based on complex network representation. Background Art

[0002] Due to the continuous emergence of three-dimensional data acquisition devices in recent years and the decreasing cost, the three-dimensional data has been increasing rapidly, and there is an urgent need to process and analyze the three-dimensional data. The most critical part is its feature representation. Taking three-dimensional point cloud as an example, how to extract the feature representation of the three-dimensional point cloud has received much attention, because it is a basic feature extraction operation and can lay a foundation for subsequent point cloud processing and analysis tasks. However, existing point cloud feature networks are all based on the real number expression framework and do not use complex number expression networks for feature extraction. Mathematically, we know that real numbers are special cases where the imaginary part in complex numbers is zero, so it can be seen that complex number expression is more general and data retention is more comprehensive. Physically, through the Hilbert transform, we can obtain the imaginary part of its expression from its analytic expression. Therefore, existing networks are all simplified versions of feature extraction and do not obtain the point cloud feature representation with all information. Therefore, how to design a more general and more comprehensive point cloud feature is particularly important. The related background art documents are: Trabelsi C, Bilaniuk O, Zhang Y, et al. Deep complex networks[J]. arXiv preprint arXiv:1705.09792, 2017.

[0003] Through the above analysis, the problems and defects existing in the prior art are: existing point cloud feature extraction networks all operate based on the real number domain, lacking a more substantial and complete feature representation in the complex number domain, and a general complex number point cloud feature extraction network framework.

[0004] The difficulty in solving the above problems and defects is: at the present stage, there is no network based on point cloud complex numbers. Therefore, it is difficult to design a reasonable network framework, design the complex number expression of the point cloud, and construct the structure of the complex convolution kernel.

[0005] The significance of solving the above problems and defects is: complex number point cloud feature extraction covers only real number domain point cloud feature extraction, is a more general version of the point cloud feature extraction framework, and covers the information part lost by real numbers. It provides a more complete and substantial feature extraction method for subsequent three-dimensional point cloud processing and analysis tasks. Summary of the Invention

[0006] In view of the above problems and the deficiencies of related methods, the present invention proposes a method and device for extracting point cloud features based on complex network representation, and divides the construction of the point cloud complex feature extraction network into four steps. The point cloud features extracted through complex network representation contain more comprehensive information and require fewer iterations. They are applied to various tasks of point cloud processing and analysis, such as point cloud classification, point cloud segmentation, point cloud retrieval, etc., providing a basic feature guarantee for downstream tasks of point cloud processing and analysis.

[0007] To solve the above technical problems, the present invention adopts the following technical solutions to achieve the purpose:

[0008] A method and device for extracting point cloud features based on complex network representation. The method and device include the following steps:

[0009] S1. Point cloud preprocessing;

[0010] This step is used to represent the input point cloud as two parts: a real number representation part and an imaginary number representation part, which can be obtained by various methods, such as Hilbert transform, or Fourier transform e exponential representation, etc.

[0011] S2. Construction of the convolutional operation layer;

[0012] This step is used to perform convolutional operation calculations on the point cloud complex representation obtained in S1, which can be a single layer or multiple layers, and the output feature dimension size depends on the task requirements. The operation of each layer can be implemented by building a multi-layer perceptron, or point cloud convolutional operation, etc.;

[0013] S3. Modulus calculation;

[0014] This step is used to transform the point cloud complex features extracted in S2 into a global feature representation. There are various methods for modulus calculation, and the square root of the sum of the squares of the imaginary number and the real number can be used to obtain it.

[0015] S4. Feature post-processing;

[0016] This step is used to further process the point cloud feature representation obtained in S3. It can be normalization processing, such as the softmax function, etc., or further feature encoding to obtain a global representation, such as using the NetVLAD method, etc., or directly output the required size.

[0017] This method proposes a method and device for extracting point cloud features based on complex network representation. The key lies in the construction of a point cloud complex feature extraction network, which can be implemented in four steps. The point cloud features extracted through complex network representation contain more comprehensive information and require fewer iterations. They can be applied to various tasks in point cloud processing and analysis, such as point cloud classification, point cloud segmentation, point cloud retrieval, etc., providing a basic feature extraction guarantee for downstream tasks in point cloud processing and analysis.

[0018] Compared with the background technology, this method for extracting point cloud features based on complex network representation has the following beneficial effects:

[0019] 1. The present invention is a more generalized feature extraction method for point clouds, providing a method for more generalized, complete, and compact point cloud feature extraction;

[0020] 2. The present invention can be applied to the feature extraction part of various point cloud processing and analysis tasks, and the feature performance is better than that with only the real number domain. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a flowchart of the present invention,

[0022] Figure 2 is a block diagram for describing the main framework of the present invention,

[0023] Figure 3 is a flowchart of the implementation process of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0024] To more clearly illustrate the purpose, technical solution, and advantages of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the scope of protection of the present invention.

[0025] Embodiment:

[0026] As shown in Figure 1 and 2 and 3, this embodiment discloses a method and device for extracting point cloud features based on complex network representation, which specifically includes the following steps:

[0027] Step S1, point cloud preprocessing;

[0028] This step is used to represent the input point cloud as two parts: a real number representation part and an imaginary number representation part, which can be obtained by various methods, such as Hilbert transform, or Fourier transform exponential representation (z = r iθ) etc. Here, the Hilbert transform is used for illustration, as shown in formula (1).

[0029] Among them, this formula indicates that physically realizable signals are all real signals, and the spectrum of a real signal has conjugate symmetry, that is, the amplitudes of the positive and negative spectra are the same, and the phases are opposite. If only the complex expression p(t) of the positive frequency part of the signal is taken, then p(t) is the analytic representation of the point cloud signal s(t). And H[s(t)] is called the Hilbert transform of the point cloud signal s(t). Therefore, by inputting the point cloud signal s(t), its imaginary part, that is, the Hilbert transform, can be calculated according to formula (1).

[0030]

[0031] Then, the real and imaginary parts of the complex expression p(t) of the point cloud can be obtained: s(t) and H[s(t)]. In formula (1), i refers to the imaginary part. Here, since the Hilbert transform expression is actually the result of convolving the original signal with a signal, it can essentially be regarded as performing a special convolution integral, and the convolution signal is: That is, the impulse response. For the sake of simple calculation, a single layer of convolution can be used to simulate the calculation process of the Hilbert transform, which can simplify the calculation.

[0032] Step S2: Construction of the convolution operation layer;

[0033] This step is used to perform convolution operation calculations on the complex expressions of the point cloud obtained from S1 respectively. It can be composed of a single layer or multiple layers of iteration, and the output feature dimension size depends on the task requirements. The operation of each layer can be implemented by building a multi-layer perceptron, or point cloud convolution operation, etc. Here, the process of implementing with point cloud convolution operation is exemplified. First, define the complex convolution kernel W, as shown in formula (2), where A is the convolution kernel of the real part and B is the convolution kernel of the imaginary part.

[0034] W = A + iB (2)

[0035] Then, after defining the complex convolution kernel, we simplify the complex expression f of the point cloud obtained from S1 as shown in formula (3), where x and y are the real and imaginary parts of the point cloud respectively.

[0036] f = x + iy (3)

[0037] Therefore, the complex convolution process of the point cloud is defined by Equation (4). Correspondingly, the result X after real convolution is the difference between the convolution of the real part and the convolution of the imaginary part, as shown in Equation (5), while the result Y after imaginary convolution is the sum of the convolution of the imaginary part and the convolution of the real part, as shown in Equation (6-1). For simplicity, the feature extraction of the imaginary part is designed as shown in Equation (6-2), which can reduce the computational complexity while achieving good results. The difference between Equation (6-2) and Equation (6-1) lies only in the difference between the input and the convolution kernel weights. The size of the convolution kernel and the output are determined by the specific task, and one or more layers of iterative modes can be set to output the required feature size of the output.

[0038] W * f = (A * x - B * y) + i(B * x + A * y) (4)

[0039] X = (A * x - B * y) (5)

[0040] Y = (B * x + A * y) (6-1)

[0041] Y = (A * x + B * y) (6-2)

[0042] Step S3, modulus calculation;

[0043] This step is used to transform the complex features of the point cloud extracted in S2 into a global feature representation. There are various ways to calculate the modulus. It can be obtained by taking the square root of the sum of the squares of the imaginary and real numbers, as shown in Equation (7). The calculated feature is F. Since the square root operation may cause the network to be non-differentiable, it is simplified to use a convolution operation to simulate Equation (7).

[0044]

[0045] Step S4, feature post-processing;

[0046] This step is used to further process the point cloud feature representation obtained in S3. It can be a normalization process, such as using the softmax function to express the feature as a 1xD representation, or further feature encoding to obtain a global representation, such as using the NetVLAD method, etc., or directly output the required size. For example, in the point cloud retrieval task, when using NetVLAD for further encoding, the feature needs to be first L2-normalized and then transformed, as shown in Equation (8).

[0047] F = NetVLAD(|F|2) (8)

[0048] In summary, this embodiment discloses a point cloud feature extraction method and device based on complex network expression. Most existing methods are based on the extraction of real number features and cannot fully display the information contained in the data itself. Therefore, the present invention proposes a point cloud feature extraction method and device based on complex network expression, which can be divided into four steps to build a point cloud complex feature extraction network. In the first step, the Hilbert transform is used to obtain the processing of the real and imaginary parts of the point cloud complex expression; in the second step, the result of the first step is used to convolve the real domain and imaginary parts respectively, and then the complex convolution formula is used to calculate the final feature expression of the real part and the imaginary part, and multiple layers can be iterated to achieve the extraction of more complex features with a certain number of outputs; in the third step, the result of the second step can be used to calculate the complex expression combining the real and imaginary feature expressions of the point cloud, and it can be obtained by modulus calculation; in the fourth step, the result of the third step is further normalized, or feature encoding dimensionality reduction is performed to obtain compact and representative global features, which provides a basic guarantee for subsequent three-dimensional point cloud processing and analysis tasks.

[0049] In order to prove the advanced nature of the real-time technical solution of the present invention, actual effect tests were carried out in multiple projects, and actual efficiency comparisons were made with manual coding.

[0050] Table 1: Comparison of the effects of the present invention and PointNetVLAD

[0051]

[0052] As shown in Table 1, compared with the network that does not use complex feature expression, the results are divided into baseline and optimized results. Compared with the network that does not use complex feature expression, it can be clearly seen that, especially the baseline, the methods proposed in this patent have the best effects.

[0053] [1]Uy MA, Lee G H.PointNetVLAD: Deep Point Cloud Based Retrieval for Large-Scale Place Recognition[C] / / 2018IEEE / CVF Conference on Computer Vision and Pattern Recognition.IEEE, 2018.

[0054] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented in whole or in part in the form of a computer program product, the computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on a computer, the processes or functions described in the embodiments of the present invention are generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wire (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.). The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated therein. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)), etc.

[0055] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, simplifications, etc. made without departing from the spirit and principle of the present invention shall be equivalent replacement methods and are all included in the protection scope of the present invention.

Claims

1. A method for extracting point cloud features based on complex network representation, characterized in that, It includes the following steps: S1. Point cloud preprocessing. In point cloud preprocessing, the input point cloud needs to be represented as two parts: a real number representation part and an imaginary number representation part, which can be obtained by Hilbert transform or Fourier transform exponential expression; S2. Construction of the convolution operation layer. In the construction of the convolution operation layer, complex convolution operation is performed on the point cloud represented by complex numbers respectively; S3. Modulus calculation. In modulus calculation, the calculation of the complex number features of the overall point cloud is specifically the calculation of the modulus; S4. Feature post-processing. In feature post-processing, the point cloud feature post-processing method is used to obtain the final global point cloud complex number representation.

2. The method for extracting point cloud features based on complex network expression according to claim 1, wherein The Hilbert transform mentioned above is shown in formula (1); wherein, this formula indicates that physically realizable signals are all real signals, and the spectrum of real signals has conjugate symmetry, that is, the amplitudes of positive and negative spectra are the same, and the phases are opposite. If only the complex number representation p(t) of the positive frequency part of the signal is taken, then p(t) is the analytic representation of the point cloud signal s(t); and H[s(t)] is called the Hilbert transform of the point cloud signal s(t); therefore, by inputting the point cloud signal s(t), its imaginary part can be calculated according to formula (1). （1） Then, the real and imaginary parts of the point cloud complex number representation p(t) can be obtained: s(t) and H[s(t)].

3. The method for extracting point cloud features based on complex network expression according to claim 1, wherein, The specific construction of the convolution operation layer in S2 is as follows: First, define a complex convolution kernel as shown in formula (2), where A is the convolution kernel of the real part and B is the convolution kernel of the imaginary part; （2） Then, after defining the complex convolution kernel, the point cloud complex number representation obtained from S1 is simplified and expressed as shown in formula (3), where x and y are the real part and the imaginary part of the point cloud respectively; （3） Therefore, the complex convolution process of the point cloud is defined as formula (4); correspondingly, the result X after real number convolution is the difference between the convolution of the real part and the convolution of the imaginary part, as shown in formula (5), while the result Y after imaginary number convolution is the sum of the convolution of the imaginary part and the convolution of the real part, as shown in formula (6-1); for simplicity, the feature extraction of the imaginary part is designed as shown in formula (6-2), which can reduce the calculation amount while obtaining good results; it only differs from 6-1 in the input and the weight values of the convolution kernel; the size of the convolution kernel and the output are set in one or more iterative modes, aiming to output the required feature size of the output; （4） （5） （6-1） （6-2） 4. The method for extracting point cloud features based on complex network expression according to claim 1, wherein In step S3, the modulus calculation in modulus calculation is as follows; It is used to change the complex number features of the point cloud extracted in S2 into a global feature representation. There are various ways of modulus calculation, which is obtained by taking the square root of the sum of the squares of the imaginary number and the real number, as shown in formula (7), and the calculated feature is F; here, since the square root operation may cause the network to be non-differentiable, it is simplified to use convolution operation to simulate formula (7); （7）。 5. A point cloud feature extraction device based on complex number network representation, which is used to run or implement the point cloud feature extraction method based on complex number network representation described in any one of claims 1-4.

Images