Colorful three-dimensional point cloud restoration method and device based on supervised learning, and medium

Abstract

The invention discloses a colored three-dimensional point cloud repairing method and device based on supervised learning and a medium. The method comprises the steps that 1, a colored three-dimensional point cloud containing holes and noise is acquired to serve as an original point cloud; 2, identifying and positioning a target hole area in the original point cloud by analyzing the density of the original point cloud; step 3, repairing the detected target cavity area by using a first multi-dimensional collaborative optimization network to obtain a repaired local point cloud; step 4, fusing the repaired local point cloud with the original point cloud to obtain a preliminarily repaired complete point cloud; and step 5, performing global optimization on the complete point cloud by using a second multi-dimensional collaborative optimization network, and correcting noise of a non-cavity area. According to the method, two main defects, namely holes and noise, in the point cloud can be effectively processed at the same time, and position information and color information of the repaired point cloud are taken into account.

Description

Technical Field

[0001] The present invention relates to the technical field of point cloud restoration, and in particular to a method, device and medium for color three-dimensional point cloud restoration based on supervised learning. Background Art

[0002] With the rapid development of 3D scanning and computer vision technology, 3D point cloud data is increasingly used in various fields, including but not limited to autonomous driving, robot vision, augmented reality (AR) and virtual reality (VR). However, due to various factors, such as equipment accuracy limitations and environmental occlusion, the acquired raw point cloud data often has problems such as holes and noise, which greatly limits the application value of point cloud data.

[0003] At present, the research on point cloud repair and completion mainly focuses on the following directions:

[0004] 1. Geometry-based methods

[0005] Early point cloud restoration methods were mainly based on geometric principles, such as surface reconstruction, interpolation, etc. These methods work well when dealing with simple geometric shapes, but have difficulty dealing with complex three-dimensional structures and details.

[0006] 2. Machine Learning-Based Methods

[0007] With the development of deep learning technology, point cloud restoration methods based on neural networks have gradually become a research hotspot. These methods can better handle complex three-dimensional structures by learning the features of a large amount of point cloud data. However, most existing methods focus on the restoration of the spatial coordinates of points, and the processing of color point clouds needs to be improved.

[0008] 3. Multimodal Fusion Method

[0009] Some studies have attempted to combine point cloud data with data from other modalities (such as images) to improve the accuracy of restoration. However, such methods usually require additional data sources, increasing the complexity of data acquisition and processing. Summary of the invention

[0010] In view of this, the purpose of the present invention is to propose a color 3D point cloud restoration method based on supervised learning, which can effectively deal with the two main defects in the point cloud: holes and noise, and take into account the position information and color information of the repaired point cloud.

[0011] In order to achieve the above technical objectives, the technical solution adopted by the present invention is:

[0012] The present invention provides a color three-dimensional point cloud restoration method based on supervised learning, comprising the following steps:

[0013] Step 1, obtaining a colored 3D point cloud containing holes and noise as the original point cloud;

[0014] Step 2: identifying and locating the target cavity area in the original point cloud by analyzing the density of the original point cloud;

[0015] Step 3: Use the first multi-dimensional collaborative optimization network to repair the detected target hole area to obtain a repaired local point cloud;

[0016] Step 4: Fuse the repaired local point cloud with the original point cloud to obtain a complete point cloud that has been initially repaired;

[0017] Step 5: Use the second multi-dimensional collaborative optimization network to globally optimize the complete point cloud and correct the noise in the non-hole area.

[0018] Furthermore, the step 2 specifically includes:

[0019] Step 21, dividing the original point cloud into a number of voxels; for each voxel, calculating the number of points contained therein to obtain the density of the voxel;

[0020] Step 22: Based on the average density of the entire point cloud, a density threshold is set;

[0021] Step 23, select the first voxel as the current voxel and go to step 24;

[0022] Step 24, determine whether the density of the current voxel is lower than the density threshold. If so, mark the voxel as a potential hole area and go to step 25; if not, select the next voxel as the current voxel and return to step 24;

[0023] Step 25: taking an area within a set range of the edge of the potential hole area as an extended area, and taking the potential hole area and the extended area as areas to be repaired;

[0024] Step 26: Repeat steps 24 and 25 until all voxels are traversed to obtain multiple areas to be repaired as target hole areas in the original point cloud.

[0025] Furthermore, the step 3 specifically includes:

[0026] Step 31: input the target hole region in the original point cloud into the trained first multi-dimensional collaborative optimization network;

[0027] Step 32: repair the target hole area through the trained first multi-dimensional collaborative optimization network, and output the repaired target hole area, that is, the local point cloud.

[0028] Furthermore, before step 31, the method further includes: training the first multi-dimensional collaborative optimization network to obtain a trained first multi-dimensional collaborative optimization network; specifically including:

[0029] Step a1: creating a first multi-dimensional collaborative optimization network based on the AdaPoinTr architecture, and setting six channels in the first multi-dimensional collaborative optimization network, including an x ​​channel, a y channel, and a z channel for obtaining position information, and an R channel, a G channel, and a B channel for obtaining color information;

[0030] Step a2: A hole repair model is also set in the first multi-dimensional collaborative optimization network, and the hole repair model includes an encoder and a decoder;

[0031] Step a3: Input the target hole regions of multiple original point clouds as training sets into the encoder of the first multi-dimensional collaborative optimization network; the point set of the target hole region is represented as: X local =p1,p2,...,p n ; Among them, X local represents the point set of the target hole area, p i is a single point in the target hole area; n represents the number of points in the target hole area;

[0032] Step a4: The encoder extracts the position information and color information of each point in the input target hole area as local features through six channels; the extraction process is expressed as: local =f local (X local );

[0033] Among them, F local represents the local features, f local represents the feature extraction function;

[0034] Step a5: using the local features output by the encoder as input to the decoder, and generating a coarse-grained local prediction point cloud through the decoder;

[0035] Step a6: Calculate the position difference and color difference between the current local prediction point cloud and the same part of the target point cloud through the first loss function, and back-propagate to the first multi-dimensional collaborative optimization network;

[0036] Step a7, the first multi-dimensional collaborative optimization network adjusts network parameters according to the difference; the decoder regenerates a local prediction point cloud according to the adjusted network parameters;

[0037] Step a8, repeat steps a6 to a7 to iterate until the value of the first loss function is minimized and the optimal local prediction point cloud is obtained; at this time, the training of the first multi-dimensional collaborative optimization network is completed.

[0038] Furthermore, the formula of the first loss function is specifically:

[0039]

[0040] Where L represents the first loss function; N represents the number of points in the target point cloud; i is the subscript of the point in the target point cloud; w i is the detail richness weight, which is calculated according to the degree of change of the normal vector in the neighborhood of point i; CD is the improved chamfer distance loss, which is used to optimize the overall shape of the point cloud; PT is the target point cloud; P is the local predicted point cloud; Color_loss is the color reconstruction loss, which optimizes the color information of the point cloud; Normal_loss is the normal vector loss; λ1, λ2 and λ3 are the global weight coefficients of each loss term;

[0041] Detail richness weight w i The calculation method is:

[0042] w i =f(var(normal j )),j∈theneighborhoodofpointi;

[0043] Among them, var(normal j ) represents the variance of the normal vectors of all points j in the neighborhood of point i, and f is a monotonically increasing function used to map the variance to an appropriate weight range.

[0044] Furthermore, the step 4 specifically includes:

[0045] Step 41: For each point p in the repaired local point cloud i , set a neighborhood radius r;

[0046] Step 42: Search for p in the original point cloud i r neighborhood, determine whether there is a point in the original point cloud in the r neighborhood;

[0047] Step 43: If there is no point in the original point cloud in the neighborhood of r, it means that p i Located in the potential void area, directly i Add to the original point cloud;

[0048] Step 44: If there is a point in the original point cloud in the neighborhood of r, it means that p i Located in the extended area, calculate p i The nearest original point p in its r neighborhood o The distance d and the color difference c;

[0049] Step 45: If d is less than the preset threshold d t And c is less than the preset threshold ct , then use p i Replace p o Otherwise, keep p o and discard p i .

[0050] Furthermore, the step 5 specifically includes:

[0051] Step 51: input the complete point cloud into the trained second multi-dimensional collaborative optimization network;

[0052] Step 52: globally optimize the complete point cloud through the trained second multi-dimensional collaborative optimization network, and output the optimized target point cloud.

[0053] Furthermore, before step 51, the method further includes: training the second multi-dimensional collaborative optimization network to obtain a trained second multi-dimensional collaborative optimization network; specifically including:

[0054] Step b1, creating a second multi-dimensional collaborative optimization network based on the AdaPoinTr architecture, and setting six channels in the first multi-dimensional collaborative optimization network, including an x ​​channel, a y channel, and a z channel for obtaining position information, and an R channel, a G channel, and a B channel for obtaining color information;

[0055] Step b2: Input multiple complete point clouds as training sets into the second multi-dimensional collaborative optimization network, wherein the complete point clouds are represented as: global =p1,p2,...,p N ,N>n; where X global is the point set of the complete point cloud; p i is a single point in the complete point cloud; N represents the number of points in the complete point cloud;

[0056] Step b3: From the complete point cloud X through six channels global Extract the position and color information of each point as the global feature F global , the extraction process is expressed as: F global =f θ (X global ), where f θ is the feature extraction function;

[0057] Step b4, the global optimization process is expressed as: Among them, g θ is to optimize the network, is the optimized global prediction point cloud, g θ Minimize noise in the inpainted point cloud by correcting geometric discontinuities and ensuring global consistency of position and color;

[0058] Step b5: Calculate the current complete point cloud X through the second loss function global With global prediction point cloud The position and color differences of the same part are back-propagated to the second multi-dimensional collaborative optimization network;

[0059] Step b6: the second multi-dimensional collaborative optimization network adjusts network parameters according to the difference; and regenerates a global prediction point cloud according to the adjusted network parameters;

[0060] Step b7, repeat steps b5 to b6 for iteration until the value of the second loss function is minimized and the optimal total predicted point cloud is obtained; at this point, the second multi-dimensional collaborative optimization network training is completed;

[0061] The formula of the second loss function is specifically:

[0062]

[0063] Among them, L chamfer represents the second loss function.

[0064] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a color three-dimensional point cloud repair method based on supervised learning as described above is implemented.

[0065] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the method for repairing a color three-dimensional point cloud based on supervised learning as described above is implemented.

[0066] By adopting the above technical solution, the present invention has the following beneficial effects compared with the prior art:

[0067] 1. A step-by-step repair strategy is adopted to first locally repair the target hole area and then globally optimize the complete point cloud to reduce noise, effectively balancing the computational efficiency and repair quality.

[0068] 2. An automatic hole detection mechanism is introduced to automatically identify and locate the target hole area in the original point cloud by analyzing the density of the point cloud, thereby improving the pertinence and efficiency of repair.

[0069] 3. Introducing the normal vector information into the first loss function effectively retains the detailed features of the point cloud.

[0070] 4. Taking both position information and color information into consideration, comprehensive restoration of color point cloud is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0072] Figure 1 It is an execution flow chart of a color three-dimensional point cloud restoration method based on supervised learning provided by an embodiment of the present invention.

[0073] Figure 2 It is a technical route framework diagram of the present invention.

[0074] Figure 3 It is a schematic diagram of an electronic device provided by an embodiment of the present invention.

[0075] Figure 4 It is a schematic diagram of a computer-readable storage medium provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0076] The present invention will be further described in detail below in conjunction with the accompanying drawings and examples. It is particularly noted that the following examples are only used to illustrate the present invention, but are not intended to limit the scope of the present invention. Similarly, the following examples are only partial embodiments of the present invention rather than all embodiments, and all other embodiments obtained by those of ordinary skill in the art without creative work are within the scope of protection of the present invention.

[0077] See also Figure 1 and Figure 2 The technical route of the present invention mainly includes the following four stages:

[0078] Cavity detection module: automatically identifies and locates the cavity areas in the point cloud by analyzing the point cloud density.

[0079] Local hole repair module: Use the first multi-dimensional collaborative optimization network (Multi-Dimensional Collaborative Optimization Network, MDCON) to locally repair the detected hole area.

[0080] Restoration result fusion module: intelligently fuse the repaired local point cloud with the original point cloud to obtain a complete point cloud that has been initially repaired.

[0081] Global optimization module: Use the second multi-dimensional collaborative optimization network to globally optimize the complete point cloud, correct the noise in the non-hole area and further improve the restoration quality.

[0082] A color 3D point cloud restoration method based on supervised learning of the present invention comprises the following steps:

[0083] Step 1, obtaining a colored 3D point cloud containing holes and noise as the original point cloud;

[0084] Step 2: identifying and locating the target cavity area in the original point cloud by analyzing the density of the original point cloud;

[0085] In three-dimensional point cloud data, holes (or "voids") refer to areas where point cloud data is missing due to occlusion, scanning angle limitations, or insufficient equipment accuracy. These areas appear as abnormally low density or no points at all in the point cloud, which seriously affects the integrity of the point cloud data and the effect of subsequent applications. Void detection is the first step in the entire repair process. Its purpose is to accurately locate the area that needs to be repaired and provide a precise target for subsequent local repair. The present invention uses a method based on point cloud density analysis to achieve automatic void detection, and its specific steps are as follows:

[0086] Step 21, dividing the original point cloud into a number of voxels; for each voxel, calculating the number of points contained therein to obtain the density of the voxel; this step is to calculate the density of the point cloud, so as to facilitate subsequent determination by density;

[0087] Step 22: Based on the average density of the entire point cloud, a density threshold is set; the density threshold can be adjusted according to the specific application scenario and point cloud characteristics; this step is to set the density threshold as a criterion for subsequent determination of whether it is a hole;

[0088] Step 23, select the first voxel as the current voxel and go to step 24;

[0089] Step 24, determine whether the density of the current voxel is lower than the density threshold. If so, mark the voxel as a potential hole area and go to step 25; if not, select the next voxel as the current voxel and return to step 24; this step is to determine the hole and find the potential hole area;

[0090] Step 25: The area within the set range of the edge of the potential hole area is used as the expansion area, and the potential hole area and the expansion area are used as the area to be repaired. This step is to appropriately expand the potential hole area and include a certain range of the hole edge in the area to be repaired, so as to ensure the continuity and smoothness of subsequent repair.

[0091] Step 26: Repeat steps 24 and 25 until all voxels are traversed to obtain multiple areas to be repaired as target hole areas in the original point cloud.

[0092] Step 3: Use the first multi-dimensional collaborative optimization network to repair the detected target hole area to obtain a repaired local point cloud;

[0093] After obtaining the target hole area to be repaired, each area to be repaired is repaired separately. This strategy has two main advantages:

[0094] Network attention: By processing the hole areas separately, the network can pay more attention to the parts that need to be repaired, thus achieving better repair results.

[0095] Computational efficiency: Only processing the hole area that needs to be repaired, rather than the entire point cloud, can significantly reduce the amount of calculation and improve the repair efficiency.

[0096] For each detected hole region, the first Multi-Dimensional Co-Optimization Network (MDCON) is used for restoration. MDCON is a deep learning network specially designed for color point cloud restoration, which can process the location and color information of points simultaneously and pays special attention to detail preservation.

[0097] Before using the first multi-dimensional collaborative optimization network, it is necessary to train the first multi-dimensional collaborative optimization network to obtain a trained first multi-dimensional collaborative optimization network; the training process specifically includes the following steps:

[0098] Step a1: creating a first multi-dimensional collaborative optimization network based on the AdaPoinTr architecture, and setting six channels in the first multi-dimensional collaborative optimization network, including an x ​​channel, a y channel, and a z channel for obtaining position information, and an R channel, a G channel, and a B channel for obtaining color information;

[0099] Step a2: A hole repair model is also set in the first multi-dimensional collaborative optimization network, and the hole repair model includes an encoder and a decoder;

[0100] Step a3: input the target hole regions of the multiple original point clouds as training sets into the encoder of the first multi-dimensional collaborative optimization network; the target hole regions include potential hole regions and extended regions, which can provide sufficient context information; the point set of the target hole region is represented as:

[0101] X local =p1,p2,...,p n ; Among them, X local represents the point set of the target hole area, p i is a single point in the target hole area; n represents the number of points in the target hole area;

[0102] Step a4: The encoder extracts the position information and color information of each point in the input target hole area as local features through six channels; the extraction process is expressed as: local =f local (X local ), where F local represents the local features, f local represents the feature extraction function;

[0103] Step a5: using the local features output by the encoder as input to the decoder, and generating a coarse-grained local prediction point cloud through the decoder;

[0104] Step a6: Calculate the position difference and color difference between the current local prediction point cloud and the same part of the target point cloud through the first loss function, and back-propagate to the first multi-dimensional collaborative optimization network;

[0105] Step a7, the first multi-dimensional collaborative optimization network adjusts network parameters according to the difference; the decoder regenerates a local prediction point cloud according to the adjusted network parameters;

[0106] Step a8, repeat steps a6 to a7 to iterate until the value of the first loss function is minimized and the optimal local prediction point cloud is obtained; at this time, the training of the first multi-dimensional collaborative optimization network is completed.

[0107] In this embodiment, the formula of the first loss function is specifically:

[0108]

[0109] Where L represents the first loss function; N represents the number of points in the target point cloud; i is the subscript of the point in the target point cloud; w i is the detail richness weight, which is calculated according to the degree of change of the normal vector in the neighborhood of point i; CD is the improved chamfer distance loss, which is used to optimize the overall shape of the point cloud; PT is the target point cloud; P is the local predicted point cloud; Color_loss is the color reconstruction loss, which optimizes the color information of the point cloud;

[0110] Normal_loss is the normal vector loss; λ1, λ2 and λ3 are the global weight coefficients of each loss term;

[0111] Detail richness weight w i The calculation method is:

[0112] w i =f(var(normal j )),j∈theneighborhoodofpointi;

[0113] Among them, var(normal j) represents the variance of the normal vectors of all points j in the neighborhood of point i, and f is a monotonically increasing function used to map the variance to an appropriate weight range.

[0114] The overall processing process of the first multi-dimensional collaborative optimization network:

[0115] 1. Input: target hole region (including potential hole region and expansion region to provide sufficient contextual information).

[0116] 2. Hole repair model: A deep learning convolutional network is used as the base model for hole repair. It mainly has the following core modules: A). Encoding module: The incomplete point cloud is input into the encoding area composed of the convolutional neural network to extract the feature representation of the input point cloud. In this encoding module, after each convolution operation, a fixed-length feature representation is generated through point-by-point feature extraction and global pooling. B). Decoding module: The output in the encoding module is used as the input of the decoding module to generate a new predicted point cloud. The decoding module uses a hierarchical decoding structure to gradually generate more and more refined point clouds. Initially, a rough point cloud is generated, and then a more detailed completed point cloud is generated through gradual refinement. C). Shape generation module: The network first generates a coarse-grained predicted point cloud, compares the output with the target point cloud, and back-propagates the error between the two (missing information of the point cloud) to the network. This transmission mechanism uses the first loss function as a guide to update the weight parameters in the network. The optimized parameters can more accurately predict the optimal solution based on the target point cloud, that is, the color and position of the missing point cloud. By iteratively minimizing the gradient (or second-order derivative) of the loss function, the predicted point cloud will continue to approach the global optimal solution until the gradient is zero, that is, the target point cloud. Thereby improving a more complete 3D shape. This hierarchical generation strategy can effectively improve the quality of point cloud completion. D). First loss function: In order to optimize the network, the first loss function is used to measure the difference between the generated point cloud and the real point cloud.

[0117] 3. Output: The point cloud N(x) repaired by the decoding module and the shape generation module, including the filled holes.

[0118] 4. Optimization: The network is optimized by calculating the difference between the output point cloud and the target point cloud (ground truth) at the same part. This difference includes the error of position information and the error of color information.

[0119] After the first multi-dimensional collaborative optimization network training is completed, perform the following steps:

[0120] Step 31: input the target hole region in the original point cloud into the trained first multi-dimensional collaborative optimization network;

[0121] Step 32: repair the target hole area through the trained first multi-dimensional collaborative optimization network, and output the repaired target hole area, that is, the local point cloud.

[0122] Step 4: Fuse the repaired local point cloud with the original point cloud to obtain a complete point cloud that has been initially repaired; after the local holes have been repaired, the repaired local point cloud needs to be fused with the original point cloud to obtain a complete repaired point cloud. This process needs to be handled with caution to avoid problems such as point cloud duplication, discontinuity, or unnatural edges. An intelligent fusion method based on neighborhood analysis is proposed, and its steps are as follows:

[0123] Step 41: For each point p in the repaired local point cloud i , set a neighborhood radius r;

[0124] Step 42: Search for p in the original point cloud i r neighborhood, determine whether there is a point in the original point cloud in the r neighborhood;

[0125] Step 43: If there is no point in the original point cloud in the neighborhood of r, it means that p i Located in the potential void area, directly i Add to the original point cloud;

[0126] Step 44: If there is a point in the original point cloud in the neighborhood of r, it means that p i Located in the extended area, calculate p i The nearest original point p in its r neighborhood o The distance d and the color difference c;

[0127] Step 45: If d is less than the preset threshold d t And c is less than the preset threshold c t , then use p i Replace p o Otherwise, keep p o and discard p i .

[0128] This method can achieve smooth fusion of the restoration results and avoid obvious seams or discontinuities.

[0129] Step 5: Use the second multi-dimensional collaborative optimization network to globally optimize the complete point cloud and correct the noise in the non-hole area.

[0130] Global repair goal: After local repair is completed, the overall shape of the point cloud is initially completed, but there are still some global problems, such as geometric discontinuity, noise in non-hole areas, and color or texture inconsistency. These problems cannot be solved by local repair alone. Therefore, a global optimization network is introduced to focus on further adjustment of the overall point cloud.

[0131] Expansion of receptive field: The key to global optimization is a larger receptive field, that is, the network must be able to process the entire point cloud to capture the global context. By allowing the network to directly process the complete point cloud rather than the local area, it can learn the spatial relationship between points in a larger range, make global corrections, and ensure overall consistency.

[0132] Motivation for network reuse: The MDCON network structure used in local restoration is reused because its multi-scale architecture has the ability to handle complex geometry and texture details. MDCON can extract point cloud features at multiple levels, so that it can handle local details and effectively capture global features. Reusing this architecture not only reduces the design complexity, but also can well meet the needs of global optimization.

[0133] Advantages of multi-scale feature extraction: The MDCON network structure can process point cloud data from coarse to fine. It can not only capture global geometric features at a larger scale to ensure the overall shape of the point cloud is coordinated, but also refine texture and color at a smaller scale to correct inconsistent details. This multi-scale learning mode enables the network to simultaneously handle large-scale geometric adjustments and local texture corrections, ensuring the accuracy and comprehensiveness of global optimization.

[0134] Before using the second multi-dimensional collaborative optimization network, it is necessary to train the second multi-dimensional collaborative optimization network to obtain a trained first multi-dimensional collaborative optimization network; the training process specifically includes the following steps:

[0135] Step b1, creating a second multi-dimensional collaborative optimization network based on the AdaPoinTr architecture, and setting six channels in the first multi-dimensional collaborative optimization network, including an x ​​channel, a y channel, and a z channel for obtaining position information, and an R channel, a G channel, and a B channel for obtaining color information;

[0136] Step b2: Input multiple complete point clouds as training sets into the second multi-dimensional collaborative optimization network, wherein the complete point clouds are represented as: global =p1,p2,...,p N ,N>n; where X global is the point set of the complete point cloud; p i is a single point in the complete point cloud; N represents the number of points in the complete point cloud;

[0137] Step b3: From the complete point cloud X through six channels global Extract the position and color information of each point as the global feature F global , the extraction process is expressed as: F global =f θ (X global ), where f θ is the feature extraction function;

[0138] Step b4, the global optimization process is expressed as: Among them, g θ is to optimize the network, is the optimized global prediction point cloud, g θ Minimize noise in the inpainted point cloud by correcting geometric discontinuities and ensuring global consistency of position and color;

[0139] Step b5: Calculate the current complete point cloud X through the second loss function global With global prediction point cloud The position and color differences of the same part are back-propagated to the second multi-dimensional collaborative optimization network;

[0140] Step b6: the second multi-dimensional collaborative optimization network adjusts network parameters according to the difference; and regenerates a global prediction point cloud according to the adjusted network parameters;

[0141] Step b7, repeat steps b5 to b6 for iteration until the value of the second loss function is minimized and the optimal total predicted point cloud is obtained; at this point, the second multi-dimensional collaborative optimization network training is completed;

[0142] In this embodiment, the formula of the second loss function is specifically:

[0143]

[0144] Among them, L chamfer represents the second loss function.

[0145] The overall processing process of the second multi-dimensional collaborative optimization network:

[0146] 1. Complete point cloud as input: In the global optimization stage, the network input is expanded from the local area in the local repair to the entire point cloud. This input method enables the network to capture contextual information in a global scope, analyze the global distribution of geometry and color, and then perform global correction more effectively. In the local repair stage, the repair of the point cloud is mainly concentrated in a specific local area. The input of the network can be expressed as: X local =p1,p2,...,p n Among them, X localrepresents the point set in the local area, p i is a single point in the point cloud. The local restoration network extracts the local features f of these points local (X local ), repairing the missing geometry and color information. In the global optimization stage, the entire point cloud needs to be processed, and the input can be expressed as: global =

[0147] p1,p2,...,p N ,N>>n, where X global is the point set of the complete point cloud. Compared with the local input, X global Including more points expands the network's receptive field from local to global, allowing it to learn global geometry and texture distribution.

[0148] 2. Global information fusion and correction: By reusing the multi-scale structure of MDCON, the network can fuse the global features of the entire point cloud with the results of local restoration, eliminate residual noise, correct geometric discontinuities, and ensure the uniformity of color and texture. In this way, not only can potential problems in local restoration be corrected, but the overall consistency of the point cloud can also be enhanced. The core of the MDCON network structure is multi-scale feature extraction, and the network will process point clouds at multiple scales. Let the feature extraction function be f θ , whose input is the point set X, then the multi-scale feature extraction process can be described as:

[0149]

[0150] in, Represents point cloud representations at different scales, f θ is the feature extraction function, and m is the number of layers in the network. Through this multi-scale feature extraction, the network can capture the global geometry and local detail features of the point cloud at different scales. In the local restoration stage, the local point cloud input X is used. local , the generated features are: The core idea of ​​the global optimization stage is to reuse this feature extraction structure, but the input is from the local point cloud X local Expanded to global point cloud X global , enabling the network to capture a wider range of contextual information.

[0151] 3. Global optimization and mathematical implementation: In the global optimization stage, the input of the network is the entire point cloud X global The network uses the global feature F global To optimize the geometry and color consistency of the entire point cloud. Assume that the point cloud after the network output is optimized The global optimization process can be expressed as:

[0152] Among them, gθ is to optimize the network, is the point cloud output after optimization. Since the global network covers more global context information (point cloud without noise), g θ The noise in the repaired point cloud is minimized by correcting the discontinuity of the geometric shape and ensuring the global consistency of geometry and color. The training goal of the global optimization network is to minimize the input point cloud X global And optimized point cloud The difference between them. Commonly used loss functions include Chamfer distance or Earth Mover Distance (EMD). The second loss function uses Chamfer distance for calculation. The formula is:

[0153]

[0154] By minimizing L chamfer ,The network is able to generate restoration results that are closer to the real point cloud while ensuring the consistency of the global geometric structure and color.

[0155] After the second multi-dimensional collaborative optimization network training is completed, perform the following steps:

[0156] Step 51: input the complete point cloud into the trained second multi-dimensional collaborative optimization network;

[0157] Step 52: globally optimize the complete point cloud through the trained second multi-dimensional collaborative optimization network, and output the optimized target point cloud.

[0158] The advantages of global optimization are:

[0159] 1. Model generalization and consistency guarantee: The network architecture that reuses MDCON ensures the global consistency of point cloud geometry and color features during the global optimization process. By learning global features, the network can handle different types of point cloud data and has stronger generalization capabilities. This global optimization further improves the quality of point cloud restoration. The generated point cloud is not only reasonable in the local area, but also coherent and consistent at the global level.

[0160] 2. Noise processing and consistency correction: The global optimization network can effectively detect and remove noise in non-missing areas by integrating a wider range of spatial features, and use the global context to perform consistency correction on the geometric shape and texture features of the point cloud. Ultimately, the point cloud generated by the network is more coherent and natural.

[0161] The multi-dimensional collaborative optimization network (MDCON) proposed in this paper is a deep learning network designed specifically for color 3D point cloud restoration tasks. The implementation of MDCON is based on the AdaPoinTr architecture, but a lot of improvements and innovations have been made to it to better adapt to the color point cloud restoration task. The main features and innovations of MDCON are as follows:

[0162] 1. Multi-dimensional feature extraction:

[0163] (1) Multi-scale feature extraction: Combine multiple convolution kernel sizes to better capture the local and global geometric structures of the point cloud.

[0164] (2) Color feature extraction: When adding the color channel to the original XYZ channel, the goal is to add the color information of each point in each point cloud, so that the network can learn the features of geometric structure (xyz) and color distribution (RGB) at the same time. The mutual enhancement of geometric structure and color information is achieved. The detailed steps are described as follows: i). Input feature expansion. Usually, the input features of point cloud data only contain spatial coordinate information (position information), that is, each point P i Expressed in three-dimensional coordinates: P i =[x i ,y i ,z i ], by converting the color information [r i ,g i ,b i ] Combined with the original xyz coordinate information, the dimension of the input point is expanded from 3 dimensions to 6 dimensions: P′ i =[x i ,y i ,z i ,r i ,g i ,b i ] Therefore, every point P′ in the point cloud i Contains spatial coordinates and color features and is input into the network. ii). Adjustment of network hierarchy. In the convolution layer, the traditional 3D point cloud processing network only processes spatial coordinates and extracts local spatial geometric features through 3D convolution kernels. Now, the input channel of the convolution layer is expanded to 6 dimensions, including spatial and color information. In each layer of convolution calculation, it can be expressed as: C j =f1(W j *P′ i +b j )W j is the convolution kernel, acting on P′ i (6-dimensional features including spatial coordinates and color), b jis the bias term, and f1 is the activation function (e.g., ReLU). In this process, the convolution kernel operates in both feature space and color space to capture local and global geometric and color features of the point cloud. This can be achieved through multi-scale feature extraction, such as combining convolution kernels of different sizes to process different levels of geometric and color information.

[0165] (3) Joint prediction of space and color: Modify the dimensions of the output layer to six dimensions of XYZ and RGB, and simultaneously predict the spatial coordinates and color information of the point to ensure the consistency of the restoration results.

[0166] 2. Innovative loss function design: A comprehensive loss function based on detail richness weight is proposed. The core idea of ​​the loss function is to dynamically adjust the weight of each point in the loss calculation according to the detail richness of the area where the point is located. The specific form is as follows

[0167]

[0168] in:

[0169] L represents the first loss function;

[0170] N represents the number of points in the target point cloud;

[0171] i is the subscript of the midpoint of the target point cloud;

[0172] w i is the detail richness weight, which is calculated based on the degree of change of the normal vector in the neighborhood of point i;

[0173] CD is the improved Chamfer Distance loss, which is used to optimize the overall shape of the point cloud;

[0174] PT is the target point cloud;

[0175] P is the local prediction point cloud;

[0176] Color_loss is the color reconstruction loss, which optimizes the color information of the point cloud. Similar to the chamfer distance, first, for each point p in the predicted point cloud, i Find its nearest neighbor point q in the target point cloud j . Then calculate the mean square error of their RGB colors;

[0177] Normal_loss is the normal vector loss, which is used for multi-faceted supervision to ensure the accuracy of the geometric structure. Similarly, for each point p in the predicted point cloud i , find its nearest neighbor point q in the target point cloud j Then calculate the angle between their normal vectors. The smaller the angle, the more accurate the prediction and the smaller the loss.

[0178] λ1, λ2 and λ3 are the global weight coefficients of each loss term;

[0179] Detail richness weight w i The calculation of is a key innovation of this method. For each point in the point cloud, the degree of change of the normal vector in its local neighborhood is calculated. The area with more dramatic changes is considered to be the place with richer details, and the corresponding w i The larger the value, the greater the i The specific calculation method is:

[0180] w i =f(var(normal j )),j∈theneighborhoodofpointi;

[0181] Among them, var(normal j ) represents the variance of the normal vectors of all points j in the neighborhood of point i, and f is a monotonically increasing function used to map the variance to an appropriate weight range.

[0182] This detail-aware weighted loss function design has the following advantages:

[0183] 1. Preserve details: By increasing the weights of detail-rich areas, the network is guided to1 pay more attention to these areas, thereby better preserving and reconstructing complex geometric structures.

[0184] 2. Avoid over-smoothing: Traditional methods often lead to over-smoothing, but this method effectively avoids this problem by emphasizing detail areas.

[0185] 3. Balance the global and local: Although the attention to details is increased, the optimization of the overall shape is still guaranteed by adjusting the global weight coefficient.

[0186] 4. Through this carefully designed network structure and loss function, MDCON can effectively retain and reconstruct the detailed features of the point cloud while repairing point cloud holes and removing noise, thereby achieving high-quality color 3D point cloud restoration.

[0187] like Figure 3 As shown, an embodiment of the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the above-mentioned color three-dimensional point cloud repair method based on supervised learning is implemented.

[0188] like Figure 4 As shown, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned color three-dimensional point cloud restoration method based on supervised learning.

[0189] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0190] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0191] The above descriptions are only some embodiments of the present invention, and are not intended to limit the protection scope of the present invention. Any equivalent device or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A color 3D point cloud restoration method based on supervised learning, characterized in that: The steps include: Step 1, obtaining a colored 3D point cloud containing holes and noise as the original point cloud; Step 2: identifying and locating the target cavity area in the original point cloud by analyzing the density of the original point cloud; Step 3: Use the first multi-dimensional collaborative optimization network to repair the detected target hole area to obtain a repaired local point cloud; Step 4: Fuse the repaired local point cloud with the original point cloud to obtain a complete point cloud that has been initially repaired; Step 5: Use the second multi-dimensional collaborative optimization network to globally optimize the complete point cloud and correct the noise in the non-hole area.

2. A color 3D point cloud restoration method based on supervised learning as claimed in claim 1, characterized in that: The step 2 specifically includes: Step 21, dividing the original point cloud into a number of voxels; for each voxel, calculating the number of points contained therein to obtain the density of the voxel; Step 22: Based on the average density of the entire point cloud, a density threshold is set; Step 23, select the first voxel as the current voxel and go to step 24; Step 24, determine whether the density of the current voxel is lower than the density threshold. If so, mark the voxel as a potential hole area and go to step 25; if not, select the next voxel as the current voxel and return to step 24; Step 25: taking an area within a set range of the edge of the potential hole area as an extended area, and taking the potential hole area and the extended area as areas to be repaired; Step 26: Repeat steps 24 and 25 until all voxels are traversed to obtain multiple areas to be repaired as target hole areas in the original point cloud.

3. A color 3D point cloud restoration method based on supervised learning as claimed in claim 2, characterized in that: The step 3 specifically includes: Step 31: input the target hole region in the original point cloud into the trained first multi-dimensional collaborative optimization network; Step 32: repair the target hole area through the trained first multi-dimensional collaborative optimization network, and output the repaired target hole area, that is, the local point cloud.

4. A color 3D point cloud restoration method based on supervised learning as claimed in claim 3, characterized in that: Before step 31, the step further includes: training the first multi-dimensional collaborative optimization network to obtain a trained first multi-dimensional collaborative optimization network; specifically including: Step a1: creating a first multi-dimensional collaborative optimization network based on the AdaPoinTr architecture, and setting six channels in the first multi-dimensional collaborative optimization network, including an x ​​channel, a y channel, and a z channel for obtaining position information, and an R channel, a G channel, and a B channel for obtaining color information; Step a2: A hole repair model is also set in the first multi-dimensional collaborative optimization network, and the hole repair model includes an encoder and a decoder; Step a3: Input the target hole regions of multiple original point clouds as training sets into the encoder of the first multi-dimensional collaborative optimization network; the point set of the target hole region is represented as: X local =p1,p2,...,p n ; Among them, X local represents the point set of the target hole area, p i is a single point in the target hole area; n represents the number of points in the target hole area; Step a4: The encoder extracts the position information and color information of each point in the input target hole area as local features through six channels; the extraction process is expressed as: local =f local (X local ), where F local represents the local features, f local represents the feature extraction function; Step a5: using the local features output by the encoder as input to the decoder, and generating a coarse-grained local prediction point cloud through the decoder; Step a6: Calculate the position difference and color difference between the current local prediction point cloud and the same part of the target point cloud through the first loss function, and back-propagate to the first multi-dimensional collaborative optimization network; Step a7, the first multi-dimensional collaborative optimization network adjusts network parameters according to the difference; the decoder regenerates a local prediction point cloud according to the adjusted network parameters; Step a8, repeat steps a6 to a7 iteratively until the value of the first loss function is minimized and the optimal local prediction point cloud is obtained; at this time, the first multi-dimensional collaborative optimization network training is completed.

5. The method for color 3D point cloud restoration based on supervised learning as claimed in claim 4, characterized in that: The formula of the first loss function is specifically: Where L represents the first loss function; N represents the number of points in the target point cloud; i is the subscript of the point in the target point cloud; w i is the detail richness weight, which is calculated according to the degree of change of the normal vector in the neighborhood of point i; CD is the improved chamfer distance loss, which is used to optimize the overall shape of the point cloud; PT is the target point cloud; P is the local predicted point cloud; Color_loss is the color reconstruction loss, which optimizes the color information of the point cloud; Normal_loss is the normal vector loss; λ1, λ2 and λ3 are the global weight coefficients of each loss term; Detail richness weight w i The calculation method is: w i =f(var(normal j )),j∈theneighborhoodofpointi; Among them, var(normal j ) represents the variance of the normal vectors of all points j in the neighborhood of point i, and f is a monotonically increasing function used to map the variance to an appropriate weight range.

6. The method for color 3D point cloud restoration based on supervised learning as claimed in claim 2, characterized in that: The step 4 specifically includes: Step 41: For each point p in the repaired local point cloud i , set a neighborhood radius r; Step 42: Search for p in the original point cloud i r neighborhood, determine whether there is a point in the original point cloud in the r neighborhood; Step 43: If there is no point in the original point cloud in the neighborhood of r, it means that p i Located in the potential void area, directly i Add to the original point cloud; Step 44: If there is a point in the original point cloud in the neighborhood of r, it means that p i Located in the extended area, calculate p i The nearest original point p in its r neighborhood o The distance d and the color difference c; Step 45: If d is less than the preset threshold d t And c is less than the preset threshold c t , then use p i Replace p o Otherwise, keep p o and discard p i .

7. The method for color 3D point cloud restoration based on supervised learning according to claim 1, characterized in that: The step 5 specifically includes: Step 51: input the complete point cloud into the trained second multi-dimensional collaborative optimization network; Step 52: globally optimize the complete point cloud through the trained second multi-dimensional collaborative optimization network, and output the optimized target point cloud.

8. The method for color 3D point cloud restoration based on supervised learning as claimed in claim 7, characterized in that: Before step 51, the step further includes: training the second multi-dimensional collaborative optimization network to obtain a trained second multi-dimensional collaborative optimization network; specifically including: Step b1, creating a second multi-dimensional collaborative optimization network based on the AdaPoinTr architecture, and setting six channels in the first multi-dimensional collaborative optimization network, including an x ​​channel, a y channel, and a z channel for obtaining position information, and an R channel, a G channel, and a B channel for obtaining color information; Step b2: Input multiple complete point clouds as training sets into the second multi-dimensional collaborative optimization network, wherein the complete point clouds are represented as: global =p1,p2,...,p N ,N>n; where X global is the point set of the complete point cloud; p i is a single point in the complete point cloud; N represents the number of points in the complete point cloud; Step b3: From the complete point cloud X through six channels global Extract the position and color information of each point as the global feature F global , the extraction process is expressed as: F global =f θ (X global ), where f θ is the feature extraction function; Step b4, the global optimization process is expressed as: Among them, g θ is to optimize the network, is the optimized global prediction point cloud, g θ Minimize noise in the inpainted point cloud by correcting geometric discontinuities and ensuring global consistency of position and color; Step b5: Calculate the current complete point cloud X through the second loss function global With global prediction point cloud The position and color differences of the same part are back-propagated to the second multi-dimensional collaborative optimization network; Step b6: the second multi-dimensional collaborative optimization network adjusts network parameters according to the difference; and regenerates a global prediction point cloud according to the adjusted network parameters; Step b7, repeat steps b5 to b6 for iteration until the value of the second loss function is minimized and the optimal total predicted point cloud is obtained; at this point, the second multi-dimensional collaborative optimization network training is completed; The formula of the second loss function is specifically: Among them, L chamfer represents the second loss function.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the color three-dimensional point cloud repair method based on supervised learning as described in any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, a color three-dimensional point cloud restoration method based on supervised learning as described in any one of claims 1 to 8 is implemented.

Images