Point cloud optimization method, system, electronic device, and storage medium

Abstract

The application discloses a point cloud optimization method, system, electronic equipment and storage medium, and the point cloud optimization method comprises the following steps: obtaining an initial point cloud and an RGB image of a target object; preprocessing the initial point cloud to obtain a pretreated point cloud; meshing the pretreated point cloud to obtain a meshed point cloud; calculating vertex coordinates and normal vectors of the meshed point cloud in the RGB image coordinate system; performing differentiable rendering on the meshed point cloud according to the vertex coordinates and the normal vectors, and outputting a rendering result; and calculating a loss function according to the rendering result and the RGB image to iteratively optimize the meshed point cloud. The point cloud optimization method provided by the application takes the RGB image information of the target object as the system input to guide the direction of point cloud optimization, so that additional equipment is not needed, and the redundancy of system construction and the cost increase are avoided.

Description

Technical Field

[0001] This invention relates to the field of image processing, and in particular to a method, system, electronic device, and storage medium for optimizing structured light point clouds. Background Technology

[0002] Structured light-based 3D reconstruction, characterized by its non-contact nature and high precision, is finding increasingly widespread applications in industrial inspection, cultural relic digitization, and smart terminals. The development of machine learning has further facilitated its deeper application in fields with higher security requirements, such as medical and cosmetic procedures. In various applications, the accuracy and density of the reconstructed points have a crucial impact on the efficiency and accuracy of the application; more precise point clouds lead to higher positioning efficiency and reconstruction results; denser point clouds often significantly improve the resolution of the application.

[0003] In practical applications, due to the influence of natural environment, equipment, and inherent signal properties, the point clouds obtained by structured light 3D reconstruction are often contaminated by noise, leading to a decrease in reconstruction quality and ultimately affecting the application effect. To optimize the point clouds generated by structured light 3D reconstruction, point cloud optimization based on depth cameras, photometric stereo methods, and maximum likelihood methods have been proposed. However, due to limitations such as the need for additional equipment and noise reduction principles, these methods are often not adopted in practical applications.

[0004] Therefore, to address the above problems, it is necessary to provide a new point cloud optimization method, system, electronic device, and storage medium. Summary of the Invention

[0005] The purpose of this invention is to provide a point cloud optimization method, system, electronic device, and storage medium, which can improve point cloud accuracy, increase point cloud density, and improve the reconstruction effect of objects.

[0006] To achieve the above objectives, the technical solution provided by the present invention is as follows: In a first aspect, the present invention provides a point cloud optimization method, comprising: Acquire the initial point cloud and RGB image of the target object; The initial point cloud is preprocessed to obtain a preprocessed point cloud; The preprocessed point cloud is meshed to obtain a meshed point cloud; Calculate the vertex coordinates and normal vectors of the meshed point cloud in the RGB image coordinate system; The meshed point cloud is rendered using the vertex coordinates and the normal vector, and the rendering result is output. The loss function is calculated based on the rendering results and the RGB image to iteratively optimize the meshed point cloud.

[0007] In one or more embodiments, acquiring the initial point cloud and RGB image of the target object includes: The initial point cloud of the target object is obtained from the reference camera of the structured light module, and the RGB image of the target object is obtained from the RGB camera of the structured light module.

[0008] In one or more embodiments, the preprocessing of the initial point cloud to obtain a preprocessed point cloud includes: The initial point cloud is downsampled using the voxel grid method, and then smoothed using the moving least squares method to obtain a preprocessed point cloud.

[0009] In one or more embodiments, the step of meshing the preprocessed point cloud to obtain a meshed point cloud includes: The preprocessed point cloud is meshed into triangular facets using the greedy projection method to obtain a triangular facet meshed point cloud.

[0010] In one or more embodiments, calculating the vertex coordinates and normal vectors of the meshed point cloud in the RGB image coordinate system includes: Based on the transformation relationship between the preprocessed point cloud coordinate system and the RGB image coordinate system, the vertex coordinates and normal vectors of the meshed point cloud in the RGB image coordinate system are calculated.

[0011] In one or more embodiments, the step of performing differentiable rendering of the meshed point cloud based on the vertex coordinates and the normal vector, and outputting the rendering result, includes: The meshed point cloud is projected and transformed based on the vertex coordinates and the normal vector. The vertices and normal vectors of the projected and transformed meshed point cloud are rasterized, and the results are shading and output as coarse rendering. Based on the vertex coordinates and the normal vector, the vertices and normal vectors of the meshed point cloud are rasterized, and displacement mapping is performed on the rasterized vertices and normal vectors. The shading process is then performed, and a fine rendering result is output.

[0012] In one or more embodiments, the step of calculating a loss function based on the rendering result and the RGB image to iteratively optimize the meshed point cloud includes: The coarse rendering result and the fine rendering result are respectively compared with the RGB image to calculate the L1 norm loss function, and the illumination, albedo and displacement map information in the differentiable rendering process are backpropagated to iteratively optimize the meshed point cloud.

[0013] Secondly, the present invention provides a point cloud optimization system, characterized in that it includes: Structured light module is used to acquire the initial point cloud and RGB image of the target object; The preprocessing module is used to preprocess the initial point cloud to obtain a preprocessed point cloud; A meshing module is used to mesh the preprocessed point cloud to obtain a meshed point cloud. The calculation module is used to calculate the vertex coordinates and normal vectors of the meshed point cloud in the RGB image coordinate system; A differentiable rendering module is used to perform differentiable rendering of the meshed point cloud based on the vertex coordinates and the normal vector, and output the rendering result; An optimization module is used to calculate a loss function based on the rendering results and the RGB image to iteratively optimize the meshed point cloud.

[0014] Thirdly, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the point cloud optimization method as described above.

[0015] Fourthly, the present invention provides a computer storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the point cloud optimization method as described above.

[0016] Compared with existing technologies, the point cloud optimization method provided by this invention uses the RGB image information of the target object as system input to guide the direction of point cloud optimization. This eliminates the need for additional equipment, avoiding redundancy in system construction and increased costs. Moreover, during the differentiable rendering process, the original point cloud can be upsampled. The higher the resolution of the RGB camera, the more point cloud details are obtained, greatly increasing the upper limit of point cloud optimization and making the reconstruction effect more realistic. Attached Figure Description

[0017] Figure 1 This is a flowchart of a point cloud optimization method according to one embodiment of the present invention; Figure 2 This is a schematic diagram illustrating a usage scenario of the structured light module according to one embodiment of the present invention; Figure 3 This is a schematic diagram of a gridding method in one embodiment of the present invention; Figure 4 This is a flowchart of coarse rendering in one embodiment of the present invention; Figure 5 This is a flowchart of fine rendering in one embodiment of the present invention; Figure 6 This is a structural block diagram of a point cloud optimization system according to one embodiment of the present invention; Figure 7 This is a structural block diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0018] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings, but it should be understood that the scope of protection of the present invention is not limited to the specific embodiments.

[0019] Unless otherwise expressly stated, throughout the specification and claims, the term "comprising" or its variations such as "including" or "comprises" shall be understood to include the stated elements or components without excluding other elements or other components.

[0020] Please refer to Figure 1 The diagram shown is a flowchart of a point cloud optimization method according to an embodiment of the present invention. The point cloud optimization method includes the following steps: S101: Obtain the initial point cloud and RGB image of the target object.

[0021] In one exemplary embodiment, an initial point cloud of the target object is obtained from the reference camera of the structured light module, and an RGB image of the target object is obtained from the RGB camera of the structured light module.

[0022] It should be noted that, please refer to Figure 2 As shown, the structured light module includes a reference camera (such as...) for acquiring point clouds of the target object. Figure 2 The system includes camera 1 and camera 2 (usually CCD cameras), a projector for projecting specific light information onto the surface of the target object, and an RGB camera for acquiring RGB images of the target object.

[0023] In this embodiment, the specific method by which the structured light module acquires the point cloud data of the target object is not limited, nor is it specifically limited whether the reference camera in the structured light module is monocular or binocular, as long as it can acquire the point cloud data of the target object. Of course, in other embodiments, the point cloud data of the target object can also be acquired through other point cloud data acquisition modules besides the structured light module.

[0024] S102: Preprocess the initial point cloud to obtain a preprocessed point cloud.

[0025] In one exemplary embodiment, the specific method for preprocessing the initial point cloud to obtain a preprocessed point cloud includes: downsampling the initial point cloud according to the voxel grid method, and smoothing the downsampled initial point cloud according to the moving least squares method to obtain a preprocessed point cloud.

[0026] It should be noted that using the voxel mesh method to downsample the initial point cloud can remove redundant points in the point cloud data while preserving the spatial geometric features of the point cloud model.

[0027] Specifically, the initial point cloud after downsampling is smoothed using the moving least squares method, including: Determine the fitting function: (1) in, As basis functions, The coefficients are to be determined, and the coefficients are the corresponding spatial points. The function.

[0028] Weighted Discrete L2 Normal Form: (2) For nodes The corresponding value, As the weight function, when equation (2) takes its minimum value, it can be calculated that... Equation (2) for Differentiate: (3) (4) in : (5) (6) Let the shape function for: (7) The fitted function is obtained as follows: (8) Finally, we use the commonly used weighting function: (9) After downsampling and smoothing the initial point cloud, the resulting preprocessed point cloud is smoother and still retains its original features. This allows for optimization to begin with a low-noise initial value, thus reducing optimization time.

[0029] S103: The preprocessed point cloud is meshed to obtain a meshed point cloud.

[0030] Since subsequent rendering involves converting a continuous three-dimensional space into a two-dimensional image for display, the preprocessed point cloud needs to be meshed first.

[0031] Specifically, the preprocessed point cloud is meshed into triangular facets using the greedy projection method to obtain a meshed point cloud.

[0032] Specifically, when the initial point cloud of the target object is obtained through a structured light module, since the initial point cloud obtained by the structured light module corresponds to the pixels in the image captured by the reference camera and has a uniform distribution, it can be used as follows: Figure 3 The simplified greedy projection triangulation method shown directly performs triangulation on the image, connecting the nearest three pixels and triangulating them in a uniform direction. After completing all triangulations on the image, the point cloud corresponding to each pixel is preprocessed into a 3D spatial triangular mesh using the same connection method.

[0033] in, Figure 3 (a) The gray area represents the pixel points on the image taken by the reference camera that correspond to the point cloud. Figure 3 (b) is Figure 3 (a) is a triangular mesh obtained by triangulating the corresponding pixels in the point cloud.

[0034] S104: Calculate the vertex coordinates and normal vectors of the meshed point cloud in the RGB image coordinate system.

[0035] Since the subsequent optimization is based on the RGB image, it is necessary to calculate the vertex coordinates and normal vectors of the meshed point cloud in the RGB image coordinate system according to the transformation relationship between the preprocessed point cloud coordinate system and the RGB image coordinate system.

[0036] In one exemplary embodiment, when the initial point cloud of the target object is obtained through a structured light module, the origin coordinates of the initial point cloud are the optical center coordinates of the reference camera. Therefore, it is necessary to calculate the vertex coordinates and normal vectors of the meshed point cloud in the RGB image coordinate system based on the transformation relationship between the reference camera coordinate system and the RGB camera coordinate system.

[0037] The following example, using a structured light module with a binocular camera as the reference camera, illustrates the specific method for calculating the vertex coordinates and normal vectors of the meshed point cloud in the RGB image coordinate system.

[0038] Since the stereo system undergoes epipolar correction during point cloud generation, the corrected point cloud needs to be rotated back to the original coordinate system first. Let the correction transformation matrix after epipolar correction be... The coordinates of a certain three-dimensional point in space are Then, the coordinates of the point after rotating back to the original coordinate system are: (10) Let the rotation and translation matrix of the RGB camera relative to the reference camera be... Then rotate the obtained point cloud to the RGB camera coordinate system: (11) Set the intrinsic parameters of the RGB camera to The distortion coefficient is Then the RGB camera pixel coordinates corresponding to the point cloud satisfy: (12) (13) (14) Because the origin of the texture coordinates generated by meshing is the bottom left corner of the image and is normalized to... Between these points, the origin of the RGB pixel plane obtained from equation (14) is the upper left corner of the image, so assuming the width of the texture image is... Height is Therefore, the coordinates converted to mesh textures are: (15) When the reference camera in the structured light module is a monocular camera, epipolar correction is not required, and only equation (10) needs to be skipped for subsequent calculations.

[0039] The aforementioned coordinate transformation process is called perspective projection transformation, denoted as . ,in All are three-dimensional vectors.

[0040] S105: Perform differentiable rendering on the meshed point cloud based on the vertex coordinates and the normal vector, and output the rendering result.

[0041] It's important to note that rendering is the process of converting points in 3D space into pixels. Differentiable rendering means that this rendering process is differentiable, meaning all functions in this process have gradients. This satisfies the condition for backpropagation, which involves optimizing the input based on the results. Converting a 3D triangular mesh into a 2D image typically involves ray tracing and rasterization. This invention uses PyTorch3D-based rasterization to better adapt to the accelerated computation process of the GPU.

[0042] In an exemplary embodiment, the differentiable rendering method in step S105 includes: performing a projection transformation on the meshed point cloud based on the vertex coordinates and the normal vector, rasterizing the vertices and normal vectors of the projected meshed point cloud, performing shading processing, and outputting a coarse rendering result; rasterizing the vertices and normal vectors of the meshed point cloud based on the vertex coordinates and the normal vector, performing displacement mapping on the rasterized vertices and normal vectors, performing shading processing, and outputting a fine rendering result.

[0043] Please refer to Figure 4 and Figure 5 The diagram shown is a flowchart of a differentiable rendering process according to an embodiment of the present invention. Wherein, Figure 4 This is a rough rendering flowchart of the aforementioned differentiable rendering process. Figure 5 This is a flowchart of the detailed rendering process in the aforementioned differentiable rendering process.

[0044] Specifically, the rendered image is a combination of the albedo of the target object's surface and the shadows cast by the lighting. The surface albedo is the original color of the target object, denoted as... The shadows produced by light are denoted as The rendered output will then be: (16) The shadow shading is calculated from the rasterized normal vector and the lighting. This invention uses a spherical harmonic lighting (SH) model to simulate global illumination. Therefore, for a normal vector... The vertex of the given point has the following shadow coloring: (17) in It is the normal vector The calculated spherical harmonic basis functions These are spherical harmonic coefficients; in this embodiment, second-order spherical harmonic basis functions are used. To represent light.

[0045] Different methods of obtaining rasterized normal vectors will result in different shadow shading, and thus different rendering outputs. For example... Figure 4 The rough rendering process shown above, if keeping the initial number of triangles constant, directly uses the vertices in the initial triangle mesh. With normal vector Compared with the result obtained through projection transformation and If the rasterization is performed together, the number of vertices and normals in the rasterized result will remain unchanged. This process can actually be seen as the triangular facets being displayed on a two-dimensional plane according to the projection transformation relationship in step S104.

[0046] and Figure 5The detailed rendering process shown directly rasterizes the vertices and normals without performing the projection transformation in step S104. This is equivalent to re-triangulating each pixel on the two-dimensional plane of the RGB image, which in three dimensions is represented by re-meshing the triangular facets corresponding to each pixel. Since the original triangular facet mesh has been downsampled, and the resolution of the RGB camera used for texture mapping is generally higher than the initial point cloud, the rasterized re-division effectively upsamples the triangular facet mesh according to the resolution of the RGB texture image, resulting in a denser mesh.

[0047] A displacement map is a texture that can change the shape of a triangular mesh, i.e., change the vertex positions of the triangular mesh. In this embodiment, the displacement map and the rasterized vertex and normal vector have the same first two dimensions, meaning that each pixel stores the proportion by which the corresponding vertex moves along its normal vector. Let the dense vertices after rasterization be... Its corresponding normal vector is The displacement map value at this point is After the displacement mapping, the change at this point is as follows: (18) By continuously optimizing the displacement map, the vertices of the meshed point cloud can be continuously optimized.

[0048] S106: Calculate the loss function based on the rendering result and the RGB image to iteratively optimize the meshed point cloud.

[0049] In one exemplary embodiment, the coarse rendering result and the fine rendering result are respectively compared with the RGB image to calculate the L1 norm loss function, and the illumination, albedo and displacement map information in the differentiable rendering process are backpropagated to iteratively optimize the meshed point cloud.

[0050] Specifically, the entire rendering process involves three unknowns: the albedo of the target object's surface, illumination, and displacement map. This invention, based on the PyThorch framework, sets these three unknowns to initial values ​​of random numbers approaching zero, and then iteratively optimizes them using the Adam (Adaptive Moment Estimation) optimizer. This results in a coarse rendering output. and detailed rendering results Compared with RGB images respectively calculate loss function based on norm: (19) in The coefficients of the loss function are based on the final global loss. Backpropagation iteratively optimizes the variables (albedo, illumination, displacement map) in the Adam optimizer until the loss is reduced to a certain size or a certain number of iterations are completed.

[0051] In summary, the point cloud optimization method provided by this invention uses the RGB image information of the target object as system input to guide the direction of point cloud optimization. This eliminates the need for additional equipment, avoiding redundancy in system setup and increased costs. Furthermore, during differentiable rendering, upsampling of the original point cloud can be achieved. The higher the resolution of the RGB camera, the more detail is obtained in the point cloud, significantly increasing the upper limit of point cloud optimization and resulting in more realistic reconstruction effects.

[0052] Based on the same inventive concept as the aforementioned point cloud optimization method, the present invention also provides a point cloud optimization system 600, which includes a structured light module 601, a preprocessing module 602, a meshing module 603, a calculation module 604, a differentiable rendering module 605, and an optimization module 606.

[0053] The structured light module 601 is used to acquire the initial point cloud and RGB image of the target object, and the specific acquisition method can be found in step S101. The preprocessing module 602 is used to preprocess the initial point cloud to obtain a preprocessed point cloud, and the specific preprocessing method can be found in step S102. The meshing module 603 is used to mesh the preprocessed point cloud to obtain a meshed point cloud, and the specific meshing method can be found in step S103. The calculation module 604 is used to calculate the vertex coordinates and normal vectors of the meshed point cloud in the RGB image coordinate system, and the specific calculation method can be found in step S104. The differentiable rendering module 605 is used to perform differentiable rendering on the meshed point cloud based on the vertex coordinates and the normal vectors, and output the rendering result, and the specific differentiable rendering method can be found in step S105. The optimization module 606 is used to calculate a loss function based on the rendering result and the RGB image to iteratively optimize the meshed point cloud, and the specific optimization method can be found in step S106.

[0054] Please refer to Figure 7 As shown, embodiments of the present invention also provide an electronic device 700, which includes at least one processor 701, a memory 702 (e.g., non-volatile memory), a main memory 703, and a communication interface 704, and the at least one processor 701, the memory 702, the main memory 703, and the communication interface 704 are connected together via a bus 705. The at least one processor 701 is used to invoke at least one program instruction stored or encoded in the memory 702 to cause the at least one processor 701 to perform various operations and functions of the methods described in the various embodiments of this specification.

[0055] In the embodiments of this specification, the electronic device 700 may include, but is not limited to: personal computer, server computer, workstation, desktop computer, laptop computer, notebook computer, mobile electronic device, smartphone, tablet computer, cellular phone, personal digital assistant (PDA), handheld device, messaging device, wearable electronic device, consumer electronic device, etc.

[0056] This invention also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement various operations and functions of the methods described in the various embodiments of this specification.

[0057] The computer-readable storage medium can be any available medium or data storage device that a computer can access, including but not limited to magnetic storage (e.g., floppy disks, hard disks, magnetic tapes, magneto-optical disks (MOs), etc.), optical storage (e.g., CDs, DVDs, BDs, HVDs, etc.), and semiconductor storage (e.g., ROMs, EPROMs, EEPROMs, non-volatile memory (NAND flash), solid-state drives (SSDs), etc.).

[0058] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0059] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

Claims

1. A point cloud optimization method, characterized in that, include: The initial point cloud of the target object is obtained from the reference camera of the structured light module, and the RGB image of the target object is obtained from the RGB camera of the structured light module. The initial point cloud is downsampled using the voxel grid method, and then smoothed using the moving least squares method to obtain a preprocessed point cloud. The preprocessed point cloud is meshed into triangular facets using the greedy projection method to obtain a triangular facet meshed point cloud. Based on the transformation relationship between the preprocessed point cloud coordinate system and the RGB image coordinate system, calculate the vertex coordinates and normal vectors of the triangular mesh point cloud in the RGB image coordinate system; The triangular mesh point cloud is rendered using the vertex coordinates and the normal vector, and the rendering result is output. The differentiable rendering includes coarse rendering and fine rendering; The rough rendering includes: performing a projection transformation on the triangular mesh point cloud based on the vertex coordinates and the normal vector, rasterizing the vertices and normal vectors of the projection-transformed triangular mesh point cloud, performing shading processing, and outputting the rough rendering result. The fine rendering includes: rasterizing the vertices and normal vectors of the triangular mesh point cloud according to the vertex coordinates and the normal vectors, re-triangulating each pixel on the two-dimensional plane of the RGB image, and upsampling the triangular mesh in three-dimensional space according to the resolution of the RGB texture image; performing displacement mapping on the rasterized vertices and normal vectors, color processing, and outputting the fine rendering result; The displacement map and the rasterized vertex and normal vector have the same first two dimensions. Each pixel stores the proportion of the corresponding vertex moving along its normal vector. Let v be the dense vertex in the rasterized map. i Its corresponding normal vector is n i The displacement map value at this point is z. i After the displacement mapping, the point changes to v′. i =v i +z i *n i ; The coarse rendering result and the fine rendering result are respectively compared with the RGB image to calculate the L1 norm loss function. The lighting, albedo and displacement map information in the differentiable rendering process are then backpropagated to iteratively optimize the triangular mesh point cloud.

2. A point cloud optimization system for the point cloud optimization method of claim 1, characterized in that, include: Structured light module is used to acquire the initial point cloud and RGB image of the target object; The preprocessing module is used to preprocess the initial point cloud to obtain a preprocessed point cloud; A meshing module is used to mesh the preprocessed point cloud to obtain a meshed point cloud. The calculation module is used to calculate the vertex coordinates and normal vectors of the meshed point cloud in the RGB image coordinate system; A differentiable rendering module is used to perform differentiable rendering of the meshed point cloud based on the vertex coordinates and the normal vector, and output the rendering result; An optimization module is used to calculate a loss function based on the rendering results and the RGB image to iteratively optimize the meshed point cloud.

3. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the point cloud optimization method as described in claim 1.

4. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the point cloud optimization method as described in claim 1.

Images