Point cloud reconstruction method simulating human thinking process
Through a point cloud reconstruction method that imitates human thinking process, combined with multi-layer hybrid perception network and category information, the problem of insufficient detail capture in the existing technology of point cloud filling method is solved, and more accurate object shape reconstruction and recognition is achieved .
Patent Information
- Application Number
- CN202411350420.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-26
- Publication Date
- 2025-01-10
AI Technical Summary
The existing point cloud filling method does not capture object details sufficiently, lack of interpretability and object category information participation, resulting in unsatisfactory reconstruction results.
Using a method that imitates human thinking process, a three-dimensional point cloud of single-view angle of object is obtained through deep image preprocessing. After voxelization, the point cloud and voxel grid are encoded using a multi-layer hybrid perception network, fuse the features and use categories. The decoder and pyramid decoder carry out multi-scale point cloud reconstruction, and combine category information for decoding and fusion.
It improves the accuracy of point cloud reconstruction results, fully captures object details, and provides object type cognition, improving recognition accuracy.
Smart Images
Figure CN119295657A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the fields of computer vision and robotics, and in particular to a point cloud reconstruction method that simulates human thinking processes. Background Art
[0002] In unknown and unstructured environments, robots use a range of perception techniques to build and improve their understanding of the world. Vision is one of the main ways robots obtain information about objects. Through depth cameras, robots can obtain 3D point clouds of objects or environments to build their relevant representations. Accurately estimating and characterizing the 3D shape of an unknown object or environment is one of the challenges in many engineering applications. Once the representation of an object is established, it can be used for subsequent tasks for different purposes. For example, accurate information about the shape of an object is the basis for subsequent operations, such as reaching, grasping, and manipulation.
[0003] In most cases, due to complex environments or the inherent properties of objects, the point cloud data obtained by depth cameras is usually incomplete. Point cloud completion methods can effectively complete the reconstruction of object shapes. However, most current point cloud completion methods do not capture sufficient object details.
[0004] Previous point cloud completion methods mostly focused on how to use learning methods to establish the mapping relationship between local point clouds and complete point clouds. A prediction model was established based on various encoders and decoders, but this process only relied on the structure and depth of the network. The reconstruction process lacked interpretability and the participation of object category information, resulting in unsatisfactory reconstruction results. Summary of the invention
[0005] The purpose of the present invention is to provide a point cloud reconstruction method that simulates the human thinking process in order to improve the accuracy of point cloud reconstruction results. In view of the situation where point cloud data is incomplete, the robot uses the point cloud completion method to complete the object shape recognition and improve the recognition accuracy of the object.
[0006] The purpose of the present invention can be achieved by the following technical solutions:
[0007] A point cloud reconstruction method that simulates human thinking process, the method specifically includes:
[0008] S1. Obtain a depth image of the scene where the robot is located through a camera, and preprocess the depth image to obtain a single-view three-dimensional point cloud P of the object;
[0009] S2, voxelize the three-dimensional point cloud P to obtain a voxel grid V;
[0010] S3, use two encoders based on multi-layer hybrid perception networks to encode the point cloud P and voxel grid V, and then obtain the point cloud feature F p and voxel feature Fv ;
[0011] S4, use the point-voxel fusion module to fusion the point cloud feature F p and voxel feature F v Fusion is performed to obtain the point-voxel fusion feature F f ;
[0012] S5. Use the category decoder to fusion feature F of point-voxel f Perform category decoding to obtain the classification result L;
[0013] S6. Use the category guidance module to classify the classification result L and the point-voxel fusion feature F f Fusion is performed to obtain the category fusion feature F c ;
[0014] S7. Use pyramid decoder to fusion feature F c Decode and obtain the multi-scale complete point cloud P c .
[0015] Furthermore, the specific steps of S3 are:
[0016] Two encoders based on multi-layer hybrid perception networks are used to encode the point cloud P and voxel grid V respectively, and the point cloud features F are obtained respectively. p and voxel feature F v The encoder includes two multi-layer hybrid perception networks, each of which is composed of 6 convolutional layers. The outputs of the last 4 convolutional layers are respectively concatenated after passing through the maximum pooling layer to obtain the encoded point cloud feature F p Or voxel feature F v , the input of the first convolutional layer is the point cloud P or voxel grid V, and the input of the next five convolutional layers is the output of the previous convolutional layer;
[0017] The calculation process in the convolutional layer is:
[0018] Output = Cov(Input)
[0019] Cov(n)=Σx(k)×h(nk)
[0020] Among them, Ouput is the output of the convolution layer, Input is the input of the convolution layer, Cov(n) is the nth element of the convolution result, x(k) is the kth element of the input sequence, and h(nk) is the nkth element after processing the convolution kernel sequence.
[0021] Furthermore, the specific steps of S4 are:
[0022] The point-voxel fusion module first transforms the point cloud feature F pand voxel feature F v The vector Fpv_1 is obtained by column splicing. After the vector Fpv_1 passes through the convolution layer of the point-voxel fusion module, the vector Fpv_2 is formed. The vector Fpv_2 passes through the self-attention layer of the point-voxel fusion module to obtain the point-voxel fusion feature F f .
[0023] Furthermore, the calculation process inside the self-attention layer is:
[0024]
[0025]
[0026] Among them, Q, K, V are query, key and value respectively, softmax is the normalized exponential function, K T is the transpose of K, is the length of the sequence, x i is the i-th element of the input vector.
[0027] Furthermore, the specific steps of S5 are:
[0028] Use the category decoder to fusion feature F of point-voxel f Perform category decoding to obtain the classification result L, where the category decoder contains a convolutional neural network and a fully connected neural network.
[0029] Furthermore, the specific steps of S6 are:
[0030] The category guidance module expands the dimension of the classification result L, and then the expanded classification result L and the point-voxel fusion feature F f splicing, and processed using a convolutional neural network to obtain the category fusion feature F containing category information c .
[0031] Furthermore, the specific steps of S7 are:
[0032] Use pyramid decoder to fuse the features F for the category c Decoding, the pyramid decoder consists of two fully connected networks and three convolutional networks. The two fully connected networks extract the category fusion feature F c , and get the feature F c_1 and feature F c_2 , the category fusion feature F c , Feature F c_1 and feature F c_2 The three convolutional networks are input respectively, and the three convolutional networks output the reconstructed point cloud results of three different scales, which are integrated to obtain the multi-scale complete point cloud P c .
[0033] Furthermore, the method further includes uniformly training the category decoder and the pyramid decoder based on an overall loss function, wherein the overall loss function Loss is:
[0034]
[0035] in, They are the category loss functions Loss class And the pyramid decoder loss function Loss gen The weight of .
[0036] Furthermore, the category loss function Loss class for:
[0037]
[0038]
[0039] Among them, y i is the probability distribution of the true sample label, To normalize the probability distribution of the predicted output, K represents the total number of output categories, x j Represents the j-th element of the input vector of the class loss function.
[0040] Furthermore, the pyramid decoder loss function Loss gen for:
[0041] Loss gen =d CD (Y1,Y GT_1 )+d CD (Y2,Y GT_2 )+d CD (Y3,Y GT_3 )
[0042]
[0043] Among them, d CD is the chamfer distance, Y1, Y2 and Y3 are the predicted point clouds of three scales, i.e., the reconstructed point clouds of three different scales, Y GT_1 , Y GT_2 and Y GT_3 These are real point clouds at three scales, where S1 and S2 represent two point cloud sets for which the chamfer distance is to be calculated.
[0044] Compared with the prior art, the present invention has the following beneficial effects:
[0045] The present invention obtains a single-view three-dimensional point cloud of an object, then obtains and fuses point cloud features and voxel features, identifies the category of the object, and processes the object category and point-voxel fusion features in combination with a category guidance module, obtains the category fusion features and decodes them to obtain a multi-scale complete point cloud. The present invention utilizes single-view point cloud information and adopts a human-like thinking process from single-view observation information to classification and then to three-dimensional reconstruction of the object, which fully captures the details of the object, completes the object reconstruction task, and can also provide object type recognition, thereby improving the accuracy of the point cloud reconstruction results. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] Figure 1 is a flow chart of the present invention;
[0047] Figure 2 It is a flowchart of each module used in the present invention. DETAILED DESCRIPTION
[0048] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.
[0049] The present invention proposes a point cloud reconstruction method that simulates the human thinking process, which can efficiently complete the object reconstruction task. The present invention can not only complete the object reconstruction task, but also provide object type recognition. The flowchart of the present invention is as follows Figure 1 As shown in the figure, the flow chart of each module is as follows Figure 2 shown.
[0050] The steps of the present invention include:
[0051] S1. Obtain a depth image of the scene where the robot is located through a camera, and preprocess the depth image to obtain a single-view three-dimensional point cloud P of the object;
[0052] S2, voxelize the three-dimensional point cloud P to obtain a voxel grid V;
[0053] S3, use two encoders based on multi-layer hybrid perception networks to encode the point cloud P and voxel grid V, and then obtain the point cloud feature F p and voxel feature F v ;
[0054] S4, use the point-voxel fusion module to fusion the point cloud feature F p and voxel feature F v Fusion is performed to obtain the point-voxel fusion feature F f ;
[0055] S5. Use the category decoder to fusion feature F of point-voxel fPerform category decoding to obtain the classification result L;
[0056] S6. Use the category guidance module to classify the classification result L and the point-voxel fusion feature F f Fusion is performed to obtain the category fusion feature F c ;
[0057] S7. Use pyramid decoder to fusion feature F c Decode and obtain the multi-scale complete point cloud P c .
[0058] In S1, the preprocessing is to process the depth image acquired by the camera to obtain the required spatial information of the object;
[0059] Preprocessing operations include: coordinate conversion, object segmentation, and downsampling. Coordinate conversion is to convert the image pixel coordinate system to the world coordinate system to obtain the real position of the point cloud in the real world. Object segmentation is to segment the part of the object that needs to be operated from the point cloud data of the entire scene. Downsampling converts the number of object point clouds obtained to the number of network inputs. Here, downsampling adopts the farthest point sampling method. The final output is set to 2048 point clouds. When the object point cloud is less than 2048, the existing point cloud data is repeated to make the final number 2048. The object single-view 3D point cloud P contains a total of 2048 points, and each point is represented by: p i =(x,y,z), where x, y, and z are the three-dimensional coordinates in space.
[0060] In S2, voxelization is the process of discretizing point clouds into voxels; each voxel can be regarded as a cube, representing a unit sample evenly spaced on a 3D grid. The voxel value is 0 or 1, where 0 represents an empty voxel and 1 represents the presence of a point cloud in the voxel. The voxel grid V is set to a resolution of 64×64×64.
[0061] In S3, the encoder of the multi-layer hybrid perception network is used to complete the feature processing operation of the data. The encoder contains two multi-layer hybrid perception networks, one for encoding the point cloud and the other for encoding the voxels, respectively obtaining the point cloud features F p and voxel feature F v ;
[0062] The multi-layer hybrid perception network consists of 6 convolutional layers. The output dimension of each layer is [64, 64, 128, 256, 512, 1024]. The outputs of the last 4 convolutional layers are concatenated after passing through the maximum pooling layer to obtain the encoded point cloud feature F. p Or voxel feature F v , where the point cloud feature F p and voxel feature F vThey are all 1920×1 vectors.
[0063] The formula for the convolutional layer is as follows:
[0064] Output=Cov(Input) (1)
[0065] Cov(n)=Σx(k)×h(nk) (2)
[0066] Where Ouput is the output of the convolutional network, Input is the input of the convolutional network, Cov(n) is the nth element of the convolution result, x(k) is the kth element of the input sequence, and h(nk) is the nkth element after the convolution kernel sequence is processed; the input of the first convolutional layer is point cloud data or voxel data, and the inputs of the following five convolutional layers are the outputs of the previous convolutional network.
[0067] The maximum pooling layer formula is as follows:
[0068] F pool =Pool max (F) (3)
[0069] Among them, Poolmax is the maximum pooling layer, and the input of the pooling layer here is the output of the last four layers of convolution.
[0070] In S4, the point-voxel fusion module is used to fuse the point cloud feature Fp and the voxel feature Fv. The module contains a convolutional network and a self-attention layer. The output of the module is the point-voxel fusion feature Ff.
[0071] The point-voxel fusion module first transforms the point cloud feature F p and voxel feature F v The columns are concatenated to obtain a 1920×2 vector Fpv_1, which is then processed by the convolutional network and the self-attention layer. After Fpv_1 passes through the convolutional layer, it forms a 1920×1 vector Fpv_2. The self-attention layer further processes Fpv_2 to obtain F f ; The self-attention layer is responsible for further extracting the connection between point cloud and voxel features. The query, key, and value of attention are all input Fpv_2; The formula of the self-attention layer is:
[0072]
[0073]
[0074] Among them, Q, K, V are query, key and value respectively, softmax is the normalized exponential function, K T is the transpose of K, is the length of the sequence, x iis the i-th element of the input vector. The role of the softmax function is to convert each element in the input vector into its form as a probability value.
[0075] In S5, the category decoder is used to decode the point-voxel fusion feature F f Perform category decoding to obtain the object classification result; the category decoder contains a convolutional neural network and a fully connected neural network, and the category decoder outputs the object category result L; the category loss uses the Softmax cross entropy loss, which represents the accuracy of point cloud classification based on probability and its cross entropy; the loss term is expressed by minimizing the negative log-likelihood:
[0076]
[0077]
[0078] In the formula, y i is the probability distribution of the true sample label, is the probability distribution of the normalized prediction output.
[0079] The category guidance module in S6 aims to fuse the object category results with the point-voxel fusion features. This module first expands the dimension of the object category, then concatenates it with the point-voxel fusion features, and processes it using a convolutional neural network to obtain the fusion feature F containing the category information. c .
[0080] 1. The pyramid decoder in S7 decodes the fused features and predicts the complete point cloud to obtain a multi-scale complete point cloud Pc. The pyramid decoder consists of two fully connected networks and three convolutional networks. The two fully connected networks further extract the fused features F c , and get F c_1 and F c_2 ; The inputs of the three convolutional networks are F c 、F c_1 and F c_2 , the output is the reconstructed point cloud results of three different scales, namely W1, W2, and W3. For single-view scenes, they are set to W1=128, W2=512, and W3=2048;
[0081] The loss function for pyramid decoder training is:
[0082] Loss gen = d CD (Y1,Y GT_1 )+d CD (Y2,Y GT_2 )+d CD (Y3,Y GT_3) (8)
[0083]
[0084] Among them, d CD is the chamfer distance, Y1, Y2 and Y3 are the predicted point clouds of three scales, Y GT_1 , Y GT_2 and Y GT_3 are the real point clouds of three scales, where Y1 and Y GT_1 The scale is consistent, Y2 and Y GT_2 The scale is consistent, Y3 and Y GT_3 The scale is consistent, S1 and S2 represent the two point cloud sets for calculating the chamfer distance;
[0085] The loss function of the progressive object recognition method that simulates the overall human thinking process is:
[0086]
[0087] are the weights of the two loss functions.
[0088] The present invention provides a feasible method for robots to complete object shape recognition using point cloud completion methods in the case of incomplete point cloud data. The present invention proposes a progressive object recognition method that imitates the human thinking process, using single-view point cloud information, adopting a human-like thinking process from single-view observation information to classification, and then to three-dimensional reconstruction of the object to complete the recognition of the category and appearance shape of the object. The method uses a depth camera to collect visual information of the target object in the environment, and pre-processes to obtain a single-view three-dimensional point cloud of the object; voxelizes the three-dimensional point cloud to obtain a voxel grid; uses two multi-layer hybrid perception networks to encode the point cloud and voxel grid respectively, and then obtains point cloud features and voxel features; uses a point-voxel fusion module based on a fully connected network and a self-attention layer to process the point cloud features and voxel features to obtain point-voxel fusion features; uses a category decoder to process the point-voxel fusion features to obtain the object category; uses a category guidance module to process the object category and the point-voxel fusion features to obtain the category fusion features; uses a pyramid decoder to decode the category fusion features to obtain a multi-scale complete point cloud.
[0089] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.
Claims
1. A point cloud reconstruction method that simulates human thinking process, characterized in that: The methods specifically include: S1. Obtain a depth image of the scene where the robot is located through a camera, and preprocess the depth image to obtain a single-view three-dimensional point cloud P of the object; S2, voxelize the three-dimensional point cloud P to obtain a voxel grid V; S3, use two encoders based on multi-layer hybrid perception networks to encode the point cloud P and voxel grid V, and then obtain the point cloud feature F p and voxel feature F v ; S4, use the point-voxel fusion module to fusion the point cloud feature F p and voxel feature F v Fusion is performed to obtain the point-voxel fusion feature F f ; S5. Use the category decoder to fusion feature F of point-voxel f Perform category decoding to obtain the classification result L; S6. Use the category guidance module to classify the classification result L and the point-voxel fusion feature F f Fusion is performed to obtain the category fusion feature F c ; S7. Use pyramid decoder to fusion feature F c Decode and obtain the multi-scale complete point cloud P c .
2. The point cloud reconstruction method simulating human thinking process according to claim 1, characterized in that: The specific steps of S3 are: Two encoders based on multi-layer hybrid perception networks are used to encode the point cloud P and voxel grid V respectively, and the point cloud features F are obtained respectively. p and voxel feature F v The encoder includes two multi-layer hybrid perception networks, each of which is composed of 6 convolutional layers. The outputs of the last 4 convolutional layers are respectively concatenated after passing through the maximum pooling layer to obtain the encoded point cloud feature F p Or voxel feature F v , the input of the first convolutional layer is the point cloud P or voxel grid V, and the input of the next five convolutional layers is the output of the previous convolutional layer; The calculation process in the convolutional layer is: Output = Cov(Input) Cov(n)=Σx(k)×h(nk) Among them, Ouput is the output of the convolution layer, Input is the input of the convolution layer, Cov(n) is the nth element of the convolution result, x(k) is the kth element of the input sequence, and h(nk) is the nkth element after processing the convolution kernel sequence.
3. The point cloud reconstruction method simulating human thinking process according to claim 1, characterized in that: The specific steps of S4 are: The point-voxel fusion module first transforms the point cloud feature F p and voxel feature F v The vector Fpv_1 is obtained by column splicing. After the vector Fpv_1 passes through the convolution layer of the point-voxel fusion module, the vector Fpv_2 is formed. The vector Fpv_2 passes through the self-attention layer of the point-voxel fusion module to obtain the point-voxel fusion feature F f .
4. The point cloud reconstruction method simulating human thinking process according to claim 3, characterized in that: in, The calculation process inside the self-attention layer is: Among them, Q, K, V are query, key and value respectively, softmax is the normalized exponential function, K T is the transpose of K, is the length of the sequence, x i is the i-th element of the input vector.
5. The point cloud reconstruction method simulating human thinking process according to claim 4, characterized in that: in, The specific steps of S5 are: Use the category decoder to fusion point-voxel feature F f Perform category decoding to obtain the classification result L, where the category decoder contains a convolutional neural network and a fully connected neural network.
6. The point cloud reconstruction method simulating human thinking process according to claim 5, characterized in that: in, The specific steps of S6 are: The category guidance module expands the dimension of the classification result L, and then the expanded classification result L and the point-voxel fusion feature F f splicing, and processed using a convolutional neural network to obtain the category fusion feature F containing category information c .
7. The point cloud reconstruction method simulating human thinking process according to claim 6, characterized in that: in, The specific steps of S7 are: Use pyramid decoder to fuse the features F for the category c Decoding, the pyramid decoder consists of two fully connected networks and three convolutional networks. The two fully connected networks extract the category fusion feature F c , and get the feature F c_1 and feature F c_2 , the category fusion feature F c , Feature F c_1 and feature F c_2 The three convolutional networks are input respectively, and the three convolutional networks output the reconstructed point cloud results of three different scales, which are integrated to obtain the multi-scale complete point cloud P c .
8. The point cloud reconstruction method simulating human thinking process according to claim 7, characterized in that: in, The method also includes uniformly training the category decoder and the pyramid decoder based on an overall loss function, wherein the overall loss function Loss is: in, They are the category loss functions Loss class And the pyramid decoder loss function Loss gen The weight of .
9. The point cloud reconstruction method simulating human thinking process according to claim 8, characterized in that: in, The category loss function Loss class for: Among them, y i is the probability distribution of the true sample label, To normalize the probability distribution of the predicted output, K represents the total number of output categories, x j Represents the j-th element of the input vector of the class loss function.
10. The point cloud reconstruction method simulating human thinking process according to claim 9, characterized in that: in, Pyramid decoder loss function Loss gen for: Loss gen =d CD (Y1,Y GT_1 )+d CD (Y2,Y GT_2 )+d CD (Y3,Y GT_3 ) Among them, d CD is the chamfer distance, Y1, Y2 and Y3 are the predicted point clouds of three scales, i.e., the reconstructed point clouds of three different scales, Y GT_1 , Y GT_2 and Y GT_3 These are real point clouds at three scales, where S1 and S2 represent two point cloud sets for which the chamfer distance is to be calculated.