A point cloud upsampling method, device and storage medium based on feedback mechanism

Abstract

The present invention provides a point cloud upsampling method, device, and storage medium based on a feedback mechanism. The method includes: inputting the original point cloud into a trained point cloud upsampling model; the point cloud upsampling model performs time-step iterative processing on the initial point cloud based on a cascade feedback network, and then outputs an upsampled point cloud; the cascade feedback network includes multiple cascaded feedback upsampling modules, and an FPS module is connected between adjacent feedback upsampling modules, the first input of the FPS module is connected to the output of the feedback upsampling module of the previous stage, and the second input of the FPS module is connected to the original point cloud; in each time step iteration, the output of each feedback upsampling module in the previous time step is used as the feedback input of the feedback upsampling module in the current time step. By fusing high-level information with low-level information through time steps and feedforward information flow, the sparse point cloud can be refined at the time step, thereby obtaining a representative and fine point cloud.

Description

Technical Field

[0001] The present invention relates to the technical field of point cloud processing in computer vision, and in particular to a point cloud upsampling method, device and storage medium based on a feedback mechanism. Background Art

[0002] Point clouds are widely used in 3D reconstruction and computer vision applications, including autonomous driving, augmented reality, and robotics. However, point clouds acquired by scanning devices often suffer from limitations and noise, resulting in irregular and sparse data with varying density and sampling intervals. As observed in public benchmark datasets such as KITTI and ScanNet, these irregularities can severely hinder the effectiveness of point cloud processing and analysis. Therefore, processing sparse, irregular point clouds into dense, regular point sets is key to improving point cloud quality.

[0003] Point cloud upsampling is a technology that converts irregular point clouds into regular point clouds. Point cloud processing methods have accelerated the development of point cloud upsampling technology. PU-Net proposed a classic framework, which includes three parts: feature extraction, feature expansion, and coordinate reconstruction. Most of the subsequent work related to point cloud upsampling adopted this framework. For example, PU-GCN uses graph convolutional networks for feature extraction, and its upsampling module is based on NodeShuffle. Experiments have shown that its performance is better than PU-GAN. PU-EVA introduces an approximate upsampling module based on edge vectors, which uses neighborhood points and maximum pooling to ensure the uniformity of points. Recently, PUCRN proposed a cascaded refinement network for point cloud upsampling, which uses a transformer-based feature extraction module to learn global and local shape context.

[0004] However, these point cloud upsampling methods are all based on feed-forward networks, in which information only flows from low-level to high-level, resulting in the loss of a large amount of high-level useful information. Summary of the Invention

[0005] The present invention aims to at least solve the technical problems existing in the prior art and provide a point cloud upsampling method, device and storage medium based on a feedback mechanism.

[0006] In order to achieve the above-mentioned purpose of the present invention, according to the first aspect of the present invention, the present invention provides a point cloud upsampling method based on a feedback mechanism, including: obtaining an original point cloud, inputting the original point cloud into a trained point cloud upsampling model, the point cloud upsampling model performing time step iterative processing on the initial point cloud based on a cascade feedback network and outputting an upsampled point cloud; the cascade feedback network includes multiple cascaded feedback upsampling modules, and an FPS module is further connected between adjacent feedback upsampling modules, the first input end of the FPS module is connected to the output end of the feedback upsampling module of the previous stage, and the second input end of the FPS module is connected to the original point cloud; in each time step iteration, the output of each feedback upsampling module in the previous time step is used as the feedback input of the feedback upsampling module in the current time step.

[0007] In order to achieve the above-mentioned purpose of the present invention, according to the second aspect of the present invention, the present invention provides a feedback-based point cloud upsampling device, including: an input module for obtaining an original point cloud; a point cloud upsampling module, inputting the original point cloud into a trained point cloud upsampling model, the point cloud upsampling model outputs an upsampled point cloud after time step iterative processing on the initial point cloud based on a cascade feedback network; the cascade feedback network includes multiple cascaded feedback upsampling modules, and an FPS module is further connected between adjacent feedback upsampling modules, the first input end of the FPS module is connected to the output end of the feedback upsampling module of the previous stage, and the second input end of the FPS module is connected to the original point cloud; in each time step iteration, the output of each feedback upsampling module in the previous time step is used as the feedback input of the feedback upsampling module in the current time step.

[0008] In order to achieve the above-mentioned objectives of the present invention, according to the third aspect of the present invention, the present invention provides a computer-readable storage medium, wherein the storage medium stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, the at least one program, the code set or instruction set is loaded and executed by a processor to implement the point cloud upsampling method based on the feedback mechanism as described in the first aspect of the present invention.

[0009] The above technical solution: This article adopts multiple cascaded feedback upsampling modules to obtain the final upsampled point cloud through time step iteration. The cascade feedback network used in the point cloud upsampling model not only feeds back the feedforward information between the upsampling modules, but also contains feedback information flowing from the upper layer to the lower layer. The feedback mechanism is introduced through time iteration. As time increases, the feedback information flows back from the tth step to the tth step. The output of the high-level layer in the t-1 step has fine information. Fusing the high-level information with the low-level information can help the low-level features produce a finer-grained super-resolution point cloud in the tth step. Through the time step and the feedforward information flow, the feedback upsampling module gradually refines its output, which can easily refine the sparse point cloud in the time step to obtain a representative fine point cloud. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a flow chart of the point cloud upsampling method based on the feedback mechanism in Example 1 of the present invention;

[0011] Figure 2 1 is a schematic diagram of the structure of the cascade feedback network in Example 1 of the present invention;

[0012] Figure 3 2 is a schematic structural diagram of the feedback upsampling module in Example 1 of the present invention;

[0013] Figure 4 is the quantitative measurement result of the point cloud upsampling model of Example 1 of the present invention and the existing model on the PU1K dataset;

[0014] Figure 5 Quantitative measurement results of the point cloud upsampling model of Example 1 of the present invention and the existing model on the PU-GAN dataset;

[0015] Figure 6 This is the visualization result of the point cloud upsampling model of Example 1 of the present invention and the existing model on the PU-GAN dataset with 1024 input points. DETAILED DESCRIPTION

[0016] The following describes embodiments of the present invention in detail. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended only to explain the present invention and are not to be construed as limiting the present invention.

[0017] In the description of the present invention, it should be understood that the terms "longitudinal", "transverse", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention.

[0018] In the description of the present invention, unless otherwise specified and limited, it should be noted that the terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a mechanical connection or an electrical connection, or it can be the internal communication between two components. It can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to the specific circumstances.

[0019] Example 1

[0020] This embodiment discloses a point cloud upsampling method based on a feedback mechanism, such as Figure 1 As shown, including:

[0021] Step S1: Obtaining a raw point cloud. The raw point cloud is preferably, but not limited to, from the fields of 3D reconstruction or computer vision. Point cloud data is obtained by capturing surface points on an object using a device such as a laser scanner. The point cloud data includes the three-dimensional coordinates of each point and, preferably, also includes information such as laser reflection intensity or color at each point.

[0022] Step S2: Input the original point cloud into the trained point cloud upsampling model. The point cloud upsampling model performs time step iterative processing on the initial point cloud based on the cascade feedback network and then outputs the upsampled point cloud. Figure 2 As shown in Figure 1, the cascaded feedback network (CFnet) includes multiple cascaded feedback upsampling modules (FUMs), with FPS modules connected between adjacent FUMs. The first input of the FPS module is connected to the output of the previous level's feedback upsampling module, and the second input of the FPS module is connected to the original point cloud. In each time step iteration, the output of each feedback upsampling module in the previous time step is used as the feedback input of the feedback upsampling module in the current time step.

[0023] In this embodiment, for the convenience of observation, Figure 2 In , input represents the sparse original point cloud input (such as Figure 2 Preferably, the total number of time step iterations is preset in advance. Figure 2In , let the total number of steps of the time step iteration be T, T is preferably but not limited to be greater than or equal to 2, and information feedback is performed on each step except the 0th step, starting from the 1st step. Figure 2 In the example, for the first step, the output features of FUM_0 of the 0th step are fed back to FUM_0 of the current 1st step. The feedback information flows back from the t-1th step to the tth step, and from the upper layer to the lower layer. The output of the high layer in the t-1th step has fine information, which can help the low-level features produce a finer-grained super-resolution point cloud in the tth step.

[0024] In this embodiment, the point cloud upsampling model is designed to reconstruct a dense point cloud from a sparse input across time steps, and the feedback upsampling module FUM parameters in the cascade feedback network (CFnet) are shared. Each feedback upsampling module FUM takes the output of the previous feedback upsampling module FUM as part of the input, that is, the feedforward information. i represents the index of the feedback upsampling module FUM, and i is a positive integer. In particular, the FPS module (FPS is the abbreviation of the farthest point sampling algorithm) splices the original point cloud and the feedforward output of the i-1th feedback upsampling module, and samples the spliced point cloud to a fixed scale through the farthest point sampling algorithm. In this way, the input of the current module is fused with the prior information of the original data, which is more conducive to obtaining a favorable output. Among them, the input of the 0th module is only the original point cloud.

[0025] In this embodiment, preferably, Figure 2 As shown in Figure 1, the cascade feedback network includes three cascaded feedback upsampling modules, namely FUM_0, FUM_1 and FUM_2, which use three feedback upsampling modules to generate feedforward feature information. In addition to the feedforward feature information, the cascade feedback network CFNet also contains feedback information flowing from the upper layer to the lower layer, and introduces a feedback mechanism through time iteration, such as Figure 2 As shown (from top to bottom), as time increases, feedback information flows back from step t-1 to step t.

[0026] In this embodiment, if Figure 3 As shown in Figure 1, the feedback upsampling module includes a sequentially connected feature extraction submodule (Feature Extraction), feature fusion submodule (Feature Fusion), feature expansion submodule (Feature Expansion), and coordinate generation submodule (Coordinate Generation), as well as a jumper unit that connects the input point cloud of the feature extraction submodule to the coordinate generation submodule. The feedback upsampling module is also provided with an FB module set, which represents the points and features returned from step t-1.

[0027] In this embodiment, it is further preferred that Figure 3As shown in the figure, the feature extraction submodule includes a first multi-layer perceptron pooling layer, a point-transformer unit, and a second multi-layer perceptron pooling layer connected in sequence. Multi-layer perceptron pooling layers, or MLPs, specifically include:

[0028] Phase 1: Previous methods usually use MLP-based feature extraction to learn input features. However, MLP-based methods cannot capture local and contextual features. Therefore, this application introduces the transformer in 3D vision into the network of the feature extraction submodule. Specifically, the input of the i-th feedback upsampling module FUM (3 channels) at the t-th time step is The output of feature extraction is the corresponding point cloud feature (C channels) Where N is the number of points and j is the index of the point. First, a multi-layer perceptron (MLP) is performed on the original point cloud j to obtain the features of the original points. Then, a maximum pooling layer is used on the obtained point features to obtain the global features. The final feature F′ can be obtained by fusion operation κ of the point features and the global features. Therefore, F′ obtained by the first multi-layer perceptron pooling layer can be expressed as:

[0029]

[0030] Where M() and A() represent the MLP operation and the max pooling operation respectively. [·] is the concatenation operation.

[0031] Second stage: Use the point-transformer unit to extract local and contextual features F″ by establishing relationships between points. The F″ obtained by the point-transformer unit is:

[0032]

[0033] Among them, θ(,) is the point-transformer function.

[0034] The third stage: Like the first stage, the second multi-layer perception pooling layer is used again to aggregate the global features to generate the final point feature point-transformer. This process is roughly the same as formula (1) and will not be repeated here.

[0035] In this embodiment, it is further preferred that Figure 2 As shown, the feature fusion submodule includes a cross transform unit (Cross Transformer), which is used to fuse the output features (feedback features) of the feedback upsampling module to which the cross transform unit belongs in the previous time step and the output features (feedforward features) of the feature extraction submodule of the feedback upsampling module to which the cross transform unit belongs in the current time step.

[0036] The feature fusion method is used to aggregate the feedforward features and feedback features. In the field of point cloud processing, cross-layer feature fusion strategies are widely used in point cloud completion methods. Feedforward information flows from the lower layer to the upper layer, and the output flows back to the next step as feedback information for fusion. However, the premise of cross-layer feature fusion is that the resolution of the two is consistent. In the field of point cloud processing, point cloud completion and upsampling both involve upsampling rates, which will result in unequal resolutions of features that need to be fused. The FBNet of this application proposes a cross transformation method to fuse the features of two point clouds with different resolutions. FBNet has also demonstrated its effectiveness through a series of experiments. Therefore, the cross transformation unit (Cross Transformer) is directly applied to the feature fusion module of the network.

[0037] The feedforward point cloud and its features of the i-th step feedback upsampling module FUM are denoted as P i t and F i t , the corresponding feedback point clouds and their features are and The fusion operation based on the cross transform unit (Cross Transformer) can be expressed by the formula as follows:

[0038]

[0039] Where R() represents the Cross Transformer. r is the upsampling rate of the current feedback upsampling module FUM.

[0040] Specifically, Cross Transformer establishes a feedforward feature F through the attention mechanism i t and feedback characteristics Therefore, Cross Transformer can query useful information from the feedback point cloud to enrich the current point cloud and obtain more valuable features. Note that when the feedback information is None (for example, t = 0), Cross Transformer degenerates into point-transformer.

[0041] In this embodiment, it is further preferred that Figure 3 As shown, the feature expansion submodule includes a point confusion unit pointshuffle, which uses point shuffle to expand the aggregated point features. First, a series of fully connected layers are applied to obtain high-dimensional features of scale rC×N. Then, the expanded features are reconstructed into low-dimensional high-resolution features The scale is C×Nr. The feature expansion submodule of this application saves time and saves some computing resources.

[0042] In this embodiment, it is further preferred that the coordinate generation submodule includes a third multi-layer perception pooling layer, a repeated operation unit and a fusion unit; the third multi-layer perception pooling layer further performs feature extraction on the features output by the feature extension submodule; the repeated operation unit uses the upsampling rate of the current feedback upsampling module to repeatedly operate on the input point cloud of the feature extraction submodule introduced by the jumper unit; the fusion unit is used to fuse the output features of the third multi-layer perceptron and the output features of the repeated operation unit to obtain the output point cloud of the feedback upsampling module.

[0043] The purpose of coordinate reconstruction is to extract the features from the upsampled points Generate a new point set A commonly used coordinate reconstruction method is to directly regress the 3D point coordinates, but it is difficult to generate high-fidelity coordinates in a noise-free environment. To solve this problem, this application chooses to use both the original point cloud and the upsampled point features for coordinate generation. The output of the coordinate generation module can be calculated as:

[0044]

[0045] In Equation (4), D(,) represents repeated operations. This method alleviates the learning conflicts between multiple stages in a coarse-to-fine framework. In this way, we can generate reliable point coordinates through the stacked feedback upsampling module FUM.

[0046] In this embodiment, it is further preferred that a joint loss function is used during the training of the point cloud upsampling model:

[0047]

[0048] Where L represents the joint loss, t represents the number of time step iterations, T represents the total number of preset time step iterations, n represents the number of feedback upsampling modules in the point cloud upsampling model, i represents the index of the feedback upsampling module, and P i t represents the output point cloud of the i-th feedback upsampling module at the t-th time step, and GT represents P i t The corresponding real point cloud, L CD (P i t , GT) represents the loss of the i-th feedback upsampling module at the t-th time step, L CD (P i t , GT) is calculated according to the following formula:

[0049]

[0050] x represents P i t The value of the midpoint cloud, y represents the value of the point cloud in GT, |P i t | indicates P i t The number of point clouds of GT, |GT| represents the number of point clouds of GT.

[0051] The loss function of the cascade feedback network is joint because each feedback upsampling module has its own prediction output. Let n = 3, T = 3, so the total loss function consists of 9 parts.

[0052] The upsampling effect of the point cloud upsampling model provided in this embodiment is experimentally verified, and the effectiveness of the network proposed by us is demonstrated by conducting experiments on the publicly available datasets PU1K and PU-GAN in the field of point cloud upsampling. During the experiment, the number of time steps in the cascade feedback network CFNet proposed in this application is set to 3, the upsampling rate of the first feedback upsampling module FUM is set to 1, and the remaining feedback upsampling modules FUM are set to 2 to achieve ×4 upsampling. The method compared with CFNet includes 7 existing point cloud upsampling methods, namely PU-Net, MPU, PU-GAN, di-pu, PU-GCN, PU-EVA and PUCRN. The performance of the model is evaluated by chamfer distance (CD) and Hausdorff distance (HD). The smaller the values of these indicators, the better the performance.

[0053] Figure 4 Quantitative results on CD and HD for the PU1K dataset (×4 upsampling) are shown in

[15] . The number of input points varies, including 512 (sparse), 1024 (medium), and 2048 (dense).

[0054] Figure 5 Quantitative results on CD and HD are shown for the PU-GAN dataset (×4 upsampling rate). The number of input points is 1024.

[0055] Figure 6 The visualization results on the PU-GAN dataset (×4 upsampling) with 1024 input points are shown.

[0056] From the experimental results, it can be seen that the cascade feedback network CFNet proposed in this application outputs a more refined point cloud.

[0057] Example 2

[0058] This embodiment also discloses a feedback-based point cloud upsampling device, comprising: an input module for acquiring an original point cloud. The input module is preferably, but not limited to, a data acquisition port.

[0059] The point cloud upsampling module inputs the original point cloud into a trained point cloud upsampling model. The point cloud upsampling model performs time-step iterative processing on the initial point cloud based on a cascaded feedback network and outputs an upsampled point cloud. The cascaded feedback network includes multiple cascaded feedback upsampling modules, and an FPS module is connected between adjacent feedback upsampling modules. The first input of the FPS module is connected to the output of the previous feedback upsampling module, and the second input of the FPS module is connected to the original point cloud. In each time step iteration, the output of each feedback upsampling module in the previous time step is used as the feedback input of the feedback upsampling module in the current time step. The point cloud upsampling module is preferably, but not limited to, a computer processor.

[0060] The detailed structure of the point cloud upsampling model of this embodiment refers to Example 1 and will not be repeated here.

[0061] Example 3

[0062] This embodiment discloses a computer-readable storage medium, which stores at least one instruction, at least one program, a code set, or an instruction set. The at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the point cloud upsampling method based on the feedback mechanism as described in Example 1.

[0063] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0064] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

Claims

1. A point cloud upsampling method based on feedback mechanism, characterized in that: include: Obtaining an original point cloud, and inputting the original point cloud into a trained point cloud upsampling model, wherein the point cloud upsampling model performs time step iterative processing on the initial point cloud based on a cascade feedback network and then outputs an upsampled point cloud; The cascade feedback network includes multiple cascaded feedback upsampling modules, and an FPS module is further connected between adjacent feedback upsampling modules, the first input end of the FPS module is connected to the output end of the feedback upsampling module of the previous stage, and the second input end of the FPS module is connected to the original point cloud; In each time step iteration, the output of each feedback upsampling module in the previous time step is used as the feedback input of the feedback upsampling module in the current time step; During the training of the point cloud upsampling model, a joint loss function is used: Where L represents the joint loss, t represents the number of time step iterations, T represents the total number of preset time step iterations, n represents the number of feedback upsampling modules in the point cloud upsampling model, i represents the index of the feedback upsampling module, and P i t represents the output point cloud of the i-th feedback upsampling module at the t-th time step, and GT represents P i t The corresponding real point cloud, L CD (P i t ,GT) represents the loss of the i-th feedback upsampling module at the t-th time step, L CD (P i t ,GT) is calculated according to the following formula: x represents P i t The value of the midpoint cloud, y represents the value of the point cloud in GT, |P i t | indicates P i t The number of point clouds of GT, |GT| represents the number of point clouds of GT.

2. The point cloud upsampling method based on the feedback mechanism according to claim 1, characterized in that: The cascade feedback network includes three cascaded feedback upsampling modules.

3. The point cloud upsampling method based on feedback mechanism according to claim 1 or 2, characterized in that: The feedback upsampling module includes a feature extraction submodule, a feature fusion submodule, a feature expansion submodule and a coordinate generation submodule connected in sequence, and also includes a jumper unit that connects the input point cloud of the feature extraction submodule to the coordinate generation submodule.

4. The point cloud upsampling method based on feedback mechanism according to claim 3, characterized in that: The feature extraction submodule includes a first multi-layer perception pooling layer, a point-transformer unit, and a second multi-layer perception pooling layer connected in sequence.

5. The point cloud upsampling method based on feedback mechanism according to claim 4, characterized in that: The feature fusion submodule includes a cross transformation unit, which is used to fuse the output features of the feedback upsampling module to which the cross transformation unit belongs in the previous time step and the output features of the feature extraction submodule of the feedback upsampling module to which the cross transformation unit belongs in the current time step.

6. The point cloud upsampling method based on feedback mechanism according to claim 4 or 5, characterized in that: The feature expansion submodule includes a point confusion unit.

7. The point cloud upsampling method based on feedback mechanism according to claim 6, characterized in that: The coordinate generation submodule includes a third multi-layer perception pooling layer, a repeated operation unit and a fusion unit; The third multi-layer perception pooling layer further extracts features from the output of the feature expansion submodule; The repetitive operation unit uses the upsampling rate of the current feedback upsampling module to repeatedly operate on the input point cloud of the feature extraction submodule introduced by the jumper unit; The fusion unit is used to fuse the output features of the third multi-layer perceptron and the output features of the repeated operation unit to obtain the output point cloud of the feedback upsampling module.

8. A point cloud upsampling device based on feedback, used to implement the point cloud upsampling method based on feedback mechanism according to any one of claims 1 to 7, characterized in that: include: Input module, used to obtain the original point cloud; The point cloud upsampling module inputs the original point cloud into the trained point cloud upsampling model. The point cloud upsampling model performs time step iterative processing on the initial point cloud based on the cascade feedback network and outputs the upsampled point cloud. The cascade feedback network includes multiple cascaded feedback upsampling modules, and an FPS module is further connected between adjacent feedback upsampling modules, the first input end of the FPS module is connected to the output end of the feedback upsampling module of the previous stage, and the second input end of the FPS module is connected to the original point cloud; In each time step iteration, the output of each feedback upsampling module in the previous time step is used as the feedback input of the feedback upsampling module in the current time step.

9. A computer-readable storage medium, characterized in that The storage medium stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the point cloud upsampling method based on the feedback mechanism as described in any one of claims 1 to 7.

Images