Real-time 3D point cloud fusion and rendering methods, devices and systems

Abstract

This application provides a real-time 3D point cloud fusion and rendering method, apparatus, and system. The method includes real-time acquisition of M frames of first 3D point clouds, where the m-th frame of the first 3D point cloud is the point cloud data corresponding to the scan data of the m-th frame, and M is a positive integer, 1 ≤ m ≤ M. The voxels related to the m-th frame of the first 3D point cloud in the 3D mesh structure are determined as m-th frame incremental voxels. The m-th frame incremental voxels are traversed, and the first parameter information included in the m-th frame of the first 3D point cloud is fused into the m-th frame incremental voxels to obtain the m-th frame incremental fusion result corresponding to the m-th frame incremental voxels. The M-frame incremental fusion result is extracted to obtain an incremental point cloud model, where the M-frame incremental fusion result includes the m-th frame incremental fusion result. Configuration information is acquired, which is determined according to user requirements. The incremental point cloud model is rendered according to the configuration information. This reduces computing power and memory consumption, enabling consumer-grade devices to obtain high-quality point cloud models through real-time fusion.

Description

Technical Field

[0001] This application relates to the field of image processing, and more specifically, to a real-time three-dimensional point cloud fusion and rendering method, apparatus, and system. Background Technology

[0002] In computer graphics within computer vision, real-time 3D point cloud fusion and rendering is the process of determining the shape and / or appearance of a real-world object. The process of real-time 3D point cloud fusion and rendering can be broadly divided into three parts: image scanning, image information fusion, and model rendering.

[0003] While industrial-grade scanning equipment and processors can achieve high-quality real-time 3D point cloud fusion and rendering, they are costly. With the advantages of low cost and convenience, consumer-grade devices can also achieve real-time 3D point cloud fusion and rendering. However, the image quality acquired by consumer-grade scanning equipment is poor, which can negatively impact the quality of the reconstructed 3D model.

[0004] Currently, real-time 3D point cloud fusion and rendering can be performed through full fusion and rendering, but this method requires a large amount of computing power and memory.

[0005] Therefore, how to ensure the quality of real-time 3D point cloud fusion and rendering on consumer devices while reducing computing power and memory requirements has become an urgent problem to be solved. Summary of the Invention

[0006] This application provides a real-time 3D point cloud fusion and rendering method, apparatus and system that can reduce computing power and memory consumption, enabling consumer-grade devices to obtain high-quality point cloud models in real time.

[0007] Firstly, a three-dimensional image reconstruction method is provided. This method includes: acquiring M frames of first 3D point clouds, where the m-th frame of first 3D point cloud is the point cloud data corresponding to the scan data of the m-th frame, and M is a positive integer, 1 ≤ m ≤ M. The voxels related to the m-th frame of first 3D point clouds in the three-dimensional mesh structure are identified as m-th frame incremental voxels. The m-th frame incremental voxels are traversed, and the first parameter information included in the m-th frame of first 3D point clouds is fused into the m-th frame incremental voxels to obtain the m-th frame incremental fusion result corresponding to the m-th frame incremental voxels. The M-frame incremental fusion result is extracted to obtain an incremental point cloud model, where the M-frame incremental fusion result includes the m-th frame incremental fusion result. Configuration information is acquired, which is determined according to user requirements. The incremental point cloud model is rendered according to the configuration information.

[0008] Through the technical solution of the embodiments of this application, the fusion process of the real-time 3D point cloud fusion and rendering method does not traverse all voxels in the 3D mesh structure, but traverses the incremental voxels corresponding to the scan data. This can reduce computing resources and memory resources, thereby helping to obtain the real-time fused incremental point cloud model more efficiently, and then perform real-time 3D point cloud fusion and rendering to improve the frame rate.

[0009] Secondly, a real-time 3D point cloud fusion and rendering apparatus is provided, including an acquisition unit and a processing unit. The acquisition unit is used to acquire M frames of first 3D point clouds, where the m-th frame of first 3D point cloud is the point cloud data corresponding to the scan data of the m-th frame, and M is a positive integer, 1≤m≤M. The processing unit is used to determine the voxels related to the m-th frame of first 3D point clouds in the 3D mesh structure as the m-th frame incremental voxels. The processing unit is also used to traverse the m-th frame incremental voxels and fuse the first parameter information included in the m-th frame of first 3D point clouds into the m-th frame incremental voxels to obtain the m-th frame incremental fusion result corresponding to the m-th frame incremental voxels.

[0010] The technical effects of the device involved in the second aspect are similar to those in the first aspect, and will not be elaborated here.

[0011] Thirdly, a real-time 3D point cloud fusion and rendering apparatus is provided, comprising: a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to perform: the method as described in the first aspect or any possible implementation thereof.

[0012] Fourthly, a real-time 3D point cloud fusion and rendering system is provided, including a scanning device and a 3D image reconstruction apparatus for implementing the method of the first aspect.

[0013] Fifthly, a computer-readable storage medium is provided for storing a computer program that causes a computer to perform a method as described in the first aspect or any possible implementation thereof. Attached Figure Description

[0014] Figure 1 This is a schematic diagram of a real-time 3D point cloud fusion and rendering system provided in an embodiment of this application;

[0015] Figure 2 This is a schematic diagram of another real-time 3D point cloud fusion and rendering system provided in the embodiments of this application;

[0016] Figure 3 This is a schematic diagram of a three-dimensional mesh structure provided in an embodiment of this application;

[0017] Figure 4 This is a schematic diagram of a real-time 3D point cloud fusion and rendering method provided in an embodiment of this application;

[0018] Figure 5 This is a schematic diagram of another real-time 3D point cloud fusion and rendering method provided in the embodiments of this application;

[0019] Figure 6 This is a schematic diagram of a real-time 3D point cloud fusion and rendering device provided in an embodiment of this application;

[0020] Figure 7 This is a schematic diagram of the hardware structure of a real-time 3D point cloud fusion and rendering device provided in this application. Detailed Implementation

[0021] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0022] To facilitate understanding of the embodiments of this application, the following points are made:

[0023] First, in this application, unless otherwise specified or there is a logical conflict, the terms and / or descriptions between different embodiments are consistent and can be referenced by each other. The technical features in different embodiments can be combined to form new embodiments according to their inherent logical relationship.

[0024] Second, in this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, or B exists alone, where A and B can be singular or plural. In the textual description of this application, the character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, and c can mean: a, or, b, or, c, or, a and b, or, a and c, or, b and c, or, a, b, and c. Here, a, b, and c can each be single or multiple.

[0025] Third, in this application, the terms "first," "second," and various numerical designations (e.g., #1, #2, etc.) indicate distinctions made for ease of description and are not intended to limit the scope of the embodiments of this application. For example, they distinguish different 3D point clouds, rather than describing a specific order or sequence. It should be understood that such described objects can be interchanged where appropriate to describe solutions other than those in the embodiments of this application.

[0026] Fourth, in this application, descriptions such as "when," "under the circumstances," and "if" all refer to situations where corresponding actions will be taken under certain objective circumstances, and are not time-limited. They do not require a judgment action at the time of implementation, nor do they imply any other limitations. Furthermore, they do not mean that the judgment action following these conditional conjunctions is the only condition for achieving the result; other additional conditions may also be included to achieve the result.

[0027] Fifth, in this application, the terms “comprising” and “having” and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such process, method, product or device.

[0028] The real-time 3D point cloud fusion and rendering method, apparatus and system proposed in this application can be applied to scenarios such as augmented reality, virtual reality, mixed reality and 3D printing.

[0029] To better understand the solutions of the embodiments of this application, the following will first combine... Figure 1 A brief introduction to the possible system architectures of embodiments of this application is provided.

[0030] Figure 1 This is a schematic diagram of a real-time 3D point cloud fusion and rendering system provided in an embodiment of this application.

[0031] like Figure 1 As shown, the real-time 3D point cloud fusion and rendering system includes a scanning device 110, a computing processing device 120, and a display device 130. The scanning device 110 and the computing processing device 120 can transmit data via wired or wireless connections, as can the computing processing device 120 and the display device 130. For example, the scanning device 110 may include a camera 111, a LiDAR 112, a mobile phone 113, a 3D scanner 114, and a handheld 3D scanner 115. It should be understood that the scanning device 110 is consumer-grade, and the camera 111 may be a combination of a depth camera and an RGB camera. The depth camera can be based on structured light technology, binocular vision technology, direct time-of-flight (DTOF) technology, or indirect time-of-flight (ITOF) technology. The display device 130 may include a monitor 131, a projector, or other display devices used to display the reconstructed 3D model.

[0032] It should be understood that the scanning device 110 and the computing processing device 120 can be independent devices, or the computing processing device 120 can also be the processor of the scanning device 110. This application embodiment does not limit this.

[0033] Figure 2 This is a schematic diagram of another real-time 3D point cloud fusion and rendering system provided in the embodiments of this application.

[0034] like Figure 2 As shown, the real-time 3D point cloud fusion and rendering system may include a scanning device 110, a cloud processing device 140, and a display device 130. The scanning device 110 uploads the scanned images to the cloud processing device 140, and the cloud processing device 140 sends the reconstructed 3D model to the display device 130.

[0035] To facilitate understanding of the solution, the technical terms involved in the embodiments of this application will be briefly explained first.

[0036] 1. Three-dimensional mesh structure

[0037] The three-dimensional mesh structure can also be called a fusion field, which can be a global data cube. Figure 3 This is a schematic diagram of a three-dimensional mesh structure provided in an embodiment of this application. For example... Figure 3 As shown, the global data cube is uniformly divided into n×n×n voxels with a certain precision. A voxel is the basic unit of three-dimensional space. The three-dimensional mesh structure can also be implemented using spatial "hash" indexing. Furthermore, the three-dimensional mesh structure can be implemented using a multi-level spatial structure.

[0038] Consumer-grade real-time 3D scanners typically include depth and color cameras. However, they suffer from the following problems: Firstly, due to the relatively poor depth quality of consumer-grade depth cameras, real-time scanning can easily lead to surface delamination, flying spots, and noise near the model. Secondly, consumer-grade color cameras are generally rolling shutter cameras, which can easily cause color blurring and motion blur during real-time scanning.

[0039] The image information acquired by consumer-grade scanning devices has the aforementioned problems. Therefore, there are currently some solutions in the image fusion process and model rendering process to address these issues.

[0040] For example, the full-scale fusion patch rendering method involves traversing all voxels in the fusion field, updating the voxels of the fusion field based on the acquired point cloud of the current frame, then traversing all voxels in the updated fusion field to extract the fusion model, and finally rendering the fusion model using polygon patches. While full-scale fusion effectively addresses the aforementioned problems, it consumes significant computational power and memory, sometimes requiring acceleration via a graphics processing unit (GPU). Furthermore, polygon patch rendering easily introduces polygonal edge problems, leading to model distortion.

[0041] Therefore, another example is the full-volume fusion point cloud rendering method. This method is similar to the full-volume fusion patch rendering method in the process of reconstructing the 3D model; each step requires traversing all voxels of the fusion field. However, the difference lies in using dense point cloud rendering instead of polygon patch rendering, which allows for high-fidelity model rendering. Nevertheless, the full-volume fusion point cloud rendering method also consumes a significant amount of computing power and memory during implementation, sometimes even requiring GPU acceleration.

[0042] To address the aforementioned issues, this application proposes a real-time 3D point cloud fusion and rendering method, apparatus, and system that can ensure the quality of the reconstructed 3D model while reducing computational and memory requirements. A detailed description will follow with reference to the accompanying drawings.

[0043] Figure 4 This is a schematic diagram of a real-time 3D point cloud fusion and rendering method provided in an embodiment of this application.

[0044] S410, obtain the first 3D point cloud of frame M. The first 3D point cloud of frame m is the point cloud data corresponding to the scan data of frame m. M is a positive integer, 1≤m≤M.

[0045] As one possible implementation, the first 3D point cloud of M frames is acquired using a consumer-grade LiDAR.

[0046] The first 3D point cloud of M frames can be transmitted from the lidar to the computing processing device via wired or wireless network.

[0047] As one possible implementation, an M-frame first 2D image is acquired, the first 2D image including depth information; based on the depth information, the M-frame first 2D image is converted into an M-frame first 3D point cloud.

[0048] For example, a depth camera can be used to acquire the first 2D image in M ​​frames.

[0049] It should be understood that the types of depth cameras have already... Figure 1 The details are described in detail in the text, so I will not repeat them here.

[0050] Specifically, images can be acquired based on user-triggered operations on the scanner, or by obtaining a sequence of scanned images at a preset scanning frequency.

[0051] For example, a depth camera and an RGB camera can be used to acquire the first 2D image of frame M. In this case, the first 2D image of frame m includes not only depth information but also color information. This will be discussed below. Figure 5 The following is a detailed explanation using the case where the first 2D image includes both depth and color information.

[0052] S420, the voxels in the 3D mesh structure that are related to the first 3D point cloud in the m-th frame are determined as the incremental voxels in the m-th frame.

[0053] As one possible implementation, the incremental voxel of the m-th frame is the voxel corresponding to the surface of the first 3D point cloud in the m-th frame in the three-dimensional mesh structure.

[0054] It should be understood that the region formed by the incremental voxel in the m-th frame is not the entire region of the 3D mesh structure, but rather the region within the voxel range corresponding to the first 3D point cloud of the m-th scan data frame. The specific implementation will be combined with... Figure 5 Detailed description.

[0055] S430, traverse the incremental voxels of the m-th frame, and fuse the first parameter information included in the first 3D point cloud of the m-th frame into the incremental voxels of the m-th frame to obtain the incremental fusion result of the m-th frame corresponding to the incremental voxels of the m-th frame.

[0056] As one possible implementation, the first parameter information may include at least one of the following: the point cloud position, color, normal vector, confidence level, and update time of the first 3D point cloud in the m-th frame.

[0057] The process of traversing the incremental voxel of the m-th frame can also be referred to as traversing the voxel corresponding to the region formed by the incremental voxel of the m-th frame.

[0058] It should be understood that when the first parameter information includes the color of the first 3D point cloud in the m-th frame, the first 2D image in the m-th frame also includes color information obtained through an RGB camera. The following will combine... Figure 5 The fusion method is explained in detail.

[0059] In the above technical solution, the fusion process of the real-time 3D point cloud fusion and rendering method does not traverse all voxels in the 3D mesh structure, but traverses the incremental voxels corresponding to the scan data. This can reduce computing resources and memory resources, thereby helping to obtain the real-time fused point cloud model more efficiently.

[0060] S440, extract the incremental fusion results of M frames to obtain the incremental point cloud model. The incremental fusion results of M frames include the incremental fusion results of the m-th frame.

[0061] For example, the incremental voxels of the m-th frame are traversed, and the incremental fusion result of the m-th frame is extracted by interpolation to obtain the incremental fusion result of the M-th frame, thereby obtaining the incremental point cloud model.

[0062] It should be understood that the incremental fusion result of the m-th frame may include one or more attributes such as the point cloud position, normal vector, color, and confidence level corresponding to the incremental voxel. It should also be understood that the type and number of attributes included in the fused voxel depend on the first parameter information used in the fusion process.

[0063] It should also be understood that in the subsequent rendering process of real-time 3D point cloud fusion and rendering, only the surface portion is rendered, that is, the portion with opposite TSDF values ​​or a TSDF value of 0. Therefore, although the resolution of the incremental point cloud model extracted above is relatively coarse, it can still ensure the basic effect of the reconstructed 3D model.

[0064] In the above technical solution, compared to traversing all voxels in the 3D mesh structure for each extraction and fusion result, the above solution only needs to traverse the incremental voxels corresponding to the scan data of that frame. This also reduces computation and memory usage during the point cloud model extraction process, thereby saving the total time for real-time 3D point cloud fusion and rendering. Although the incremental point cloud model obtained by the above extraction has a coarser resolution, the extraction speed of the incremental point cloud model is faster, thus improving the reconstruction speed of the real-time 3D model.

[0065] The size of a voxel determines the output resolution. Due to computational limitations, a large voxel size can lead to a sparse point cloud model when directly rendered, resulting in a coarse-grained resolution and potentially introducing polygonal edge issues that distort the model. Therefore, to further ensure rendering quality, step S440 can be improved as follows.

[0066] As one possible approach, the extracted M-frame incremental fusion results are upsampled to obtain an incremental point cloud model.

[0067] It should be understood that the incremental point cloud model obtained after upsampling is a high-fidelity point cloud model with more refined and denser resolution.

[0068] In the above technical solution, by upsampling the extracted M-frame incremental fusion results, a higher resolution and higher fidelity incremental point cloud model can be obtained, thereby reducing model distortion in subsequent rendering processes.

[0069] Similar to the steps in S430, real-time scanning of the target object generally requires multiple frames of scan data. That is, a point cloud model can be obtained from the multiple frames of real-time scan data of the target object.

[0070] As one possible implementation, when the number of scanned data frames reaches a first threshold, the voxels corresponding to the 3D point clouds of these scanned data frames are traversed, and the fusion results of the fused voxels are extracted to obtain an incremental point cloud model.

[0071] This reduces the number of times the incremental point cloud model is extracted, but it does not require traversing all voxels in the 3D mesh structure each time. This reduces the computational power and memory usage during the traversal process to a certain extent, while also reducing the extraction frequency of the point cloud model, thereby reducing computational power and memory usage in the point cloud model extraction process.

[0072] S450 obtains configuration information and renders the incremental point cloud model of S440 based on the configuration information, where the configuration information is determined according to user requirements.

[0073] It should be understood that the rendering object is the incremental point cloud model of S440, that is, the incremental point cloud model corresponding to one frame of incremental voxels is rendered and shaded. Alternatively, if the incremental point cloud model is formed by the fusion of multiple frames of incremental data, the incremental point cloud model corresponding to multiple frames of incremental voxels can also be rendered and shaded.

[0074] It should also be understood that the rendering object is an incremental point cloud model of S440, which can be a coarse-resolution incremental point cloud model or an incremental point cloud model that has been upsampled to obtain high-fidelity and high-resolution models. This application embodiment does not limit this.

[0075] As one possible implementation, the configuration information can be received from the display device.

[0076] It should be understood that configuration information can be obtained through interaction between the user and the display device, and then sent to the computing processing device via the display device.

[0077] In this way, by acquiring configuration information related to user needs in real time through the display device, it is possible to obtain the real-time 3D point cloud fusion and rendering effect required by the user, thereby improving the user experience.

[0078] As one possible implementation, the configuration information can also be pre-set.

[0079] For example, when a user needs to observe the fusion quality of an incremental point cloud model, the rendering object included in the configuration information is the confidence attribute of each voxel of the incremental model.

[0080] Specific configuration information could be as follows: voxels with confidence levels below the second threshold are configured as red; voxels with confidence levels above the third threshold are configured as green; and voxels with confidence levels between the second and third thresholds are configured with a uniform transition.

[0081] For example, when a user needs to observe the details of a point cloud model, the rendering objects included in the configuration information are the normal vector and point cloud position of each voxel of the incremental point cloud model.

[0082] Specific configuration information can be used to configure voxels whose normal vectors are aligned with the direction of ambient light as light colors, and voxels whose normal vectors are at too large an angle with the direction of ambient light or whose point cloud positions are occluded as dark colors.

[0083] In this way, when users need to observe the details of the point cloud model, if the uneven surface is uniformly rendered as bright white, the uneven geometric surface will not be visible. By using the above method, the interference of color on the geometric surface can be avoided, and users can better observe the details of the incremental point cloud model, thus improving the user experience.

[0084] For example, when a user needs to observe the color of a point cloud model, the rendering object included in the configuration information is the color attribute of each voxel in the incremental point cloud model.

[0085] Specifically, when the target object is a geometrically symmetrical object, it is impossible to distinguish the scan quality solely through geometric information. For example, for a sphere, the color attributes of each voxel in the incremental point cloud model can be used for rendering and coloring, making it easier for users to observe the quality of the incremental point cloud model.

[0086] In the above technical solution, the incremental point cloud model, that is, each voxel in the incremental point cloud model, is rendered and colored. This means that rendering and coloring is performed only on the incremental voxels corresponding to the 3D point cloud of each frame of scanned data, without needing to render and color regions corresponding to non-incremental voxels. This reduces computational power and memory usage during real-time 3D point cloud fusion and rendering, while effectively suppressing the problems of high noise and color modeling in real-time rendering. Furthermore, due to the efficiency of incremental updates and incremental upsampling, and the intuitive rendering and coloring of multiple fusion attributes, a high-fidelity, dense, intuitive, and easily identifiable point cloud model can be obtained in real time.

[0087] The following will combine Figure 5 The real-time 3D point cloud fusion and rendering method according to embodiments of this application is described in detail. Figure 5 This will be explained in detail using the case where the first 2D image of frame M includes depth and color information.

[0088] Figure 5 This is a schematic diagram of another real-time 3D point cloud fusion and rendering method provided in the embodiments of this application.

[0089] S501, acquire the first 2D image of M frames. The first 2D image of M frames includes depth information and color information, where M is a positive integer.

[0090] For example, M frames of the first 2D image of the target object are acquired by a depth camera, where the m-th frame of the first 2D image corresponds to the m-th scan data frame. It should be understood that the m-th scan data frame represents any frame of scan data acquired by the depth camera.

[0091] As one possible implementation, M frames of the first 2D image are received from a scanning device, such as a depth camera.

[0092] It should be understood that the first 2D image of the M-frame with depth information differs from the brightness values ​​stored in pixels in a grayscale image. The depth information of the first 2D image of the M-frame is the distance from the target point to the depth camera stored in each pixel, which is the depth value. The depth image with depth information is also a two-dimensional image, so it is also referred to here as the first 2D image of the M-frame.

[0093] S502, based on the first 2D image of frame M, estimate the first parameter information of each pixel to obtain the first 3D point cloud of frame M. The first parameter information includes three-dimensional coordinates, normal vector and confidence level.

[0094] Specifically, based on the first 2D image of the m-th frame, the parameter information of each pixel in the first 2D image of the m-th frame is estimated to obtain the point cloud of the first 3D image of the m-th frame.

[0095] In other words, the depth information of the first 2D image in the m-th frame is converted into a 3D point cloud with attributes such as 3D coordinates, normal vectors, and confidence level.

[0096] Specifically, the first 2D image of the m-th frame is transformed into the first 3D point cloud of the m-th frame by pose projection.

[0097] It should be understood that since the first 2D image of the m-th frame includes not only depth information but also color information, the first parameter information of the first 3D point cloud of the m-th frame used in the subsequent fusion process includes at least one of the following: the point cloud position, color, normal vector, confidence level, and fusion time corresponding to the fusion of the incremental voxels of the m-th frame. The specific fusion of the incremental voxels of the m-th frame will be explained in detail later.

[0098] For example, during the fusion process, the point cloud position, i.e., the three-dimensional coordinates, of the first 3D point cloud in the m-th frame can be used for fusion. This application embodiment does not limit the number of first parameter information of the 3D point cloud used in the fusion process; it can be a single parameter, or multiple or all of the aforementioned information.

[0099] S503, the voxels in the 3D mesh structure that are related to the first 3D point cloud in the m-th frame are determined as the incremental voxels in the m-th frame.

[0100] Specifically, the first 3D point cloud of the m-th frame is transformed into a coordinate system of a three-dimensional mesh structure. In different coordinate systems of three-dimensional mesh structures, there can be different ways to determine the incremental voxels of the m-th frame.

[0101] The three-dimensional mesh structure can be a global data cube, a spatial "hash" index, or a multi-level spatial structure.

[0102] The following section will explain in detail the possible implementations of the three-dimensional structure as a global data cube and the determination of the spatial "hash" index for the incremental voxel of the m-th frame.

[0103] S503a, voxels in the three-dimensional mesh structure that satisfy the preset truncation distance are determined as incremental voxels of the m-th frame. The preset truncation distance is related to the surface of the first 3D point cloud of the m-th frame.

[0104] The incremental voxel of the m-th frame can also be understood as the voxel corresponding to the surface of the first 3D point cloud in the three-dimensional mesh structure of the m-th frame.

[0105] For example, the three-dimensional mesh structure can be a predefined global data cube, which is uniformly divided into n×n×n voxels with a certain precision, such as... Figure 3 As shown.

[0106] In this context, any point in the first 3D point cloud of the m-th frame can be mapped to a corresponding voxel in the global data cube. Each voxel, relative to the m-th scan data frame, yields a truncated signed distance function (TSDF) value. The typical range of TSDF values ​​is [-1, 1]. The TSDF value is defined as the truncated result of the directed distance between voxels at the corresponding depth values.

[0107] Since the scanning device acquires the surface information of the target object, each point in each frame of 3D point cloud data is a point on the reconstructed object's surface. Therefore, the TSDF value can also be understood as the minimum directed distance from a voxel to the real-time 3D point cloud fusion and rendering surface. When the TSDF value is less than 0, it indicates that the voxel is outside the real-time 3D point cloud fusion and rendering object, i.e., in front of the real-time 3D point cloud fusion and rendering surface; when the TSDF value is equal to 0, it indicates that the voxel coincides with that point on the 3D model surface, i.e., the voxel is a point on the reconstructed object's surface; when the TSDF value is greater than 0, it indicates that the voxel is inside the reconstructed object, i.e., behind the real-time 3D point cloud fusion and rendering surface. In other words, the closer a voxel is to the real-time 3D point cloud fusion and rendering surface, the closer its TSDF value is to 0.

[0108] Specifically, the TSDF value of each voxel corresponding to each point in the first 3D point cloud of the m-th frame is calculated relative to the m-th scan data frame; voxels whose absolute TSDF value is less than a preset truncation distance are determined as incremental voxels of the m-th frame.

[0109] For example, the preset cutoff distance can be 1. That is, voxels other than those with TSDF values ​​of 1 and -1 are determined as incremental voxels in the m-th frame.

[0110] S503b: In the 3D mesh structure, the voxel corresponding to the first 3D point cloud in the m-th frame, which satisfies the preset radius, is determined as the incremental voxel of the m-th frame.

[0111] Specifically, taking the voxel corresponding to the first 3D point cloud in the m-th frame as the center, voxels that meet the preset radius can be indexed by spatial "hash".

[0112] S504, traverse the incremental voxels of the m-th frame, and fuse the first parameter information included in the first 3D point cloud of the m-th frame into the incremental voxels of the m-th frame to obtain the incremental fusion result of the m-th frame corresponding to the incremental voxels of the m-th frame.

[0113] It should be understood that the first parameter information of the 3D point cloud corresponding to the scan data of the m-th frame is fused to obtain the incremental fusion result of the m-th frame, thereby updating the three-dimensional mesh structure, and only incrementally updating the incremental voxel of the m-th frame.

[0114] In other words, iterate through the incremental voxels of the m-th frame, and update the data information of the voxels corresponding to the incremental voxels of the m-th frame based on the first parameter information included in the first 3D point cloud of the m-th frame. The first parameter information is already in... Figure 4 The details are described in detail elsewhere, so they will not be repeated here.

[0115] The above process is only for real-time 3D point cloud fusion and rendering. It can be based on a single frame of scan data to obtain the fusion result, or it can be based on real-time scanning of multiple frames of scan data of the target object to obtain the scan result corresponding to the multiple frames of scan data.

[0116] In the above technical solution, for each frame of scan data obtained, only the voxels corresponding to that scan data are traversed to realize the fusion process. Compared with traversing all voxels in the three-dimensional mesh structure for each frame of scan data obtained, unnecessary computation and memory consumption can be avoided, and the computation time can be reduced, thus obtaining the real-time fused scan model more efficiently.

[0117] S505, extract the incremental fusion results of M frames to obtain the incremental point cloud model. The incremental fusion results of M frames include the incremental fusion results of the m-th frame.

[0118] S506 obtains configuration information and renders the incremental point cloud model of S505 based on the configuration information, where the configuration information is determined according to user requirements.

[0119] It should be understood that S505 and S506 can be referred to in detail in S440 and S450 respectively, and will not be elaborated here.

[0120] S507 sends the rendered incremental point cloud model to the display device.

[0121] As one possible implementation, after the user sees the rendered incremental point cloud model through the display device, the display device can obtain the user's new configuration information and send the new configuration information to the computing processing device to re-render the incremental point cloud model.

[0122] Specifically, the user interacts with the device through buttons on the user interface (UI). During the data acquisition process, the device can obtain the user's configuration information based on the user's button operations on the UI.

[0123] In the above technical solution, the real-time rendered point cloud model is displayed on the display device in real time. This helps users determine the area that needs to be scanned to obtain more data frames based on the surface quality of the incremental point cloud model displayed on the display device, thereby improving scanning efficiency and quality, and providing intuitive and visual feedback on the scanning data.

[0124] The above provides a detailed description of real-time 3D point cloud fusion and rendering methods. The following section will combine... Figure 6 and Figure 7 A brief description of the real-time 3D point cloud fusion and rendering apparatus is provided. It should be understood that the descriptions of the apparatus embodiments and the method embodiments correspond to each other. Therefore, any content not described in detail can be found in the method embodiments above, and for the sake of brevity, will not be repeated here.

[0125] Figure 6 This is a schematic diagram of a real-time 3D point cloud fusion and rendering apparatus provided in an embodiment of this application. The real-time 3D point cloud fusion and rendering apparatus includes: an acquisition unit 601 and a processing unit 602.

[0126] The acquisition unit 601 is used to acquire the first 3D point cloud of the Mth frame. The first 3D point cloud of the mth frame is the point cloud data corresponding to the scan data of the mth frame. M is a positive integer, 1≤m≤M.

[0127] Processing unit 602 is used to determine the voxels in the three-dimensional mesh structure that are related to the first 3D point cloud of the m-th frame as incremental voxels of the m-th frame.

[0128] The processing unit 602 is also used to traverse the incremental voxels of the m-th frame and fuse the first parameter information included in the first 3D point cloud of the m-th frame into the incremental voxels of the m-th frame to obtain the incremental fusion result of the m-th frame corresponding to the incremental voxels of the m-th frame.

[0129] The processing unit 602 is specifically used to extract the incremental fusion results of M frames to obtain an incremental point cloud model. The incremental fusion results of M frames include the incremental fusion results of the m-th frame.

[0130] The acquisition unit 601 is also used to acquire configuration information, which is determined according to user requirements.

[0131] The processing unit 602 is also used to render the incremental point cloud model according to the configuration information.

[0132] Optionally, the incremental voxel of the m-th frame is the voxel corresponding to the first 3D point cloud in the m-th frame in the three-dimensional mesh structure.

[0133] Optionally, the processing unit 602 is specifically used for

[0134] Voxels within a preset truncation distance in the 3D mesh structure are defined as incremental voxels for the m-th frame. The preset truncation distance is related to the surface of the first 3D point cloud in the m-th frame. Alternatively, voxels within the 3D mesh structure with a preset radius centered on the voxel corresponding to the first 3D point cloud in the m-th frame are defined as incremental voxels for the m-th frame.

[0135] Optionally, the acquisition unit 601 is specifically used to acquire M frames of first 2D images, the M frames of first 2D images including depth information. The processing unit 602 is specifically used to convert the M frames of first 2D images into M frames of first 3D point clouds based on the depth information.

[0136] Optionally, the processing unit 602 is specifically used to upsample the extracted M-frame incremental fusion result to obtain an incremental point cloud model.

[0137] Optionally, the real-time 3D point cloud fusion and rendering apparatus further includes a receiving unit 603. The receiving unit 603 is used to send the rendered incremental point cloud model to the display device.

[0138] Optionally, the receiving unit 603 is used to receive configuration information from the display device.

[0139] Optionally, the first parameter information includes at least one of the following: the point cloud position, color, normal vector, confidence level of the first 3D point cloud, and the fusion time corresponding to the fusion of the incremental voxel of the m-th frame.

[0140] It should be understood that the above description is merely illustrative. The real-time 3D point cloud fusion and rendering apparatus is used to execute the methods or steps mentioned in the foregoing method embodiments. Therefore, the apparatus corresponds to the foregoing method embodiments. For details, please refer to the description of the foregoing method embodiments, which will not be repeated here.

[0141] Figure 7 This is a schematic diagram of the hardware structure of a real-time 3D point cloud fusion and rendering device provided in this application. Figure 7 The agricultural trajectory point identification device 700 shown may include a memory 710, a processor 720, and a communication interface 730. The memory 710, processor 720, and communication interface 730 are connected via an internal connection path. The memory 710 stores instructions, and the processor 720 executes the instructions stored in the memory 720 to control at least some parameters of the input / output interface 730 to receive / send configuration information. Optionally, the memory 710 may be coupled to the processor 720 via an interface, or it may be integrated with the processor 720.

[0142] It should be noted that the aforementioned communication interface 730 uses a transceiver device, such as, but not limited to, a transceiver, to enable communication between the communication device 700 and other devices or communication networks. The aforementioned communication interface 730 may also include an input / output interface.

[0143] In implementation, each step of the above method can be completed by the integrated logic circuitry of the hardware in the processor 720 or by instructions in software form. The method disclosed in the embodiments of this application can be directly implemented by the hardware processor, or by a combination of hardware and software modules in the processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory 710, and the processor 720 reads the information in memory 710 and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, detailed descriptions are not provided here.

[0144] It should be understood that in the embodiments of this application, the processor can be a central processing unit (CPU), a GPU, or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0145] As one possible implementation, the above-mentioned real-time 3D point cloud fusion and rendering method can be performed using a single CPU thread.

[0146] As one possible implementation, the above-mentioned real-time 3D point cloud fusion and rendering method can be computed in multiple threads using computing devices such as CPUs or GPUs.

[0147] It should also be understood that, in the embodiments of this application, the memory in any of the above embodiments can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM). By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0148] As one possible implementation, the memory region corresponding to the incremental point cloud model is extracted and the memory region corresponding to the incremental point cloud model is rendered separately.

[0149] It should be understood that a memory copy is required during the rendering of the incremental point cloud model.

[0150] In the above technical solution, only the voxels of the corresponding regions after the fusion process need to be copied, thus reducing resource consumption.

[0151] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0152] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0153] This application embodiment also provides a real-time 3D point cloud fusion and rendering system including a scanning device 110 and a computing processing device 120, wherein the computing processing device 120 may be the aforementioned real-time 3D point cloud fusion and rendering device 600 or real-time 3D point cloud fusion and rendering device 700.

[0154] Alternatively, the computing processing device 120 may be located in a cloud-based processing device 140.

[0155] Optionally, if the computing processing device 120 is located in the cloud processing device 140, the system further includes means for receiving instructions from the computing processing device 120.

[0156] Optionally, the real-time 3D point cloud fusion and rendering system may also include a display device 130.

[0157] This application also provides an apparatus comprising a processing unit and a storage unit, wherein the storage unit is used to store instructions, and the processing unit executes the instructions stored in the storage unit to enable the apparatus to perform the above-described real-time 3D point cloud fusion and rendering method.

[0158] This application also provides a computer-readable medium storing program code that, when executed on a computer, causes the computer to perform the above-described actions. Figure 4 or Figure 5 The method in the middle.

[0159] This application also provides a chip, including: at least one processor and a memory, wherein the at least one processor is coupled to the memory and is used to read and execute instructions in the memory to perform the above-mentioned... Figure 4 or Figure 5 The method in the middle.

[0160] This application also provides an agricultural vehicle, including: at least one processor and a memory, wherein the at least one processor is coupled to the memory and is used to read and execute instructions in the memory to perform the above-mentioned tasks. Figure 4 or Figure 5 The method in the middle.

[0161] The terms “component,” “unit,” “system,” etc., used in this specification are used to refer to computer-related entities, hardware, firmware, combinations of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, and / or a computer. As illustrated, applications running on computing devices and computing devices can both be components. One or more components may reside in a process and / or an execution thread, and components may be located on a single computer and / or distributed among two or more computers. Furthermore, these components can be executed from various computer-readable media on which various data structures are stored. Components can communicate, for example, via local and / or remote processes based on signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system, and / or a network, such as the Internet interacting with other systems via signals).

[0162] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0163] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0164] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0165] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0166] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0167] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0168] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A real-time 3D point cloud fusion and rendering method, characterized in that, include: Real-time acquisition of the first 3D point cloud of M frames, where the first 3D point cloud of the m-th frame is the point cloud data corresponding to the scan data of the m-th frame, M is a positive integer, 1≤m≤M; The voxels in the 3D mesh structure that are related to the first 3D point cloud in the m-th frame are determined as the incremental voxels in the m-th frame; Traverse the incremental voxels of the m-th frame, and fuse the first parameter information included in the first 3D point cloud of the m-th frame into the incremental voxels of the m-th frame to obtain the incremental fusion result of the m-th frame corresponding to the incremental voxels of the m-th frame. Extract the incremental fusion results of M frames to obtain an incremental point cloud model, wherein the incremental fusion results of M frames include the incremental fusion results of the m-th frame; Obtain configuration information, which is determined based on user requirements; The incremental point cloud model is rendered based on the configuration information.

2. The method according to claim 1, characterized in that, The incremental voxel of the m-th frame is the voxel corresponding to the first 3D point cloud in the m-th frame within the three-dimensional mesh structure.

3. The method according to claim 2, characterized in that, The step of determining the voxels in the 3D mesh structure that are related to the first 3D point cloud in the m-th frame as the incremental voxels in the m-th frame includes: The voxels within the three-dimensional mesh structure that satisfy a preset truncation distance are determined as the incremental voxels of the m-th frame, where the preset truncation distance is related to the surface of the first 3D point cloud in the m-th frame; or... In the three-dimensional mesh structure, the voxel corresponding to the first 3D point cloud in the m-th frame, with a preset radius as its center, is determined as the incremental voxel of the m-th frame.

4. The method according to any one of claims 1 to 3, characterized in that, The acquisition of the first 3D point cloud of M frames includes: Acquire M frames of the first 2D image, wherein the M frames of the first 2D image include depth information; Based on the depth information, the first 2D image of the M-frame is converted into the first 3D point cloud of the M-frame.

5. The method according to any one of claims 1 to 3, characterized in that, The extraction of M-frame incremental fusion results to obtain the incremental point cloud model includes: The extracted M-frame incremental fusion results are upsampled to obtain the incremental point cloud model.

6. The method according to any one of claims 1 to 3, characterized in that, The method further includes: Send the rendered incremental point cloud model to the display device.

7. The method according to claim 6, characterized in that, The acquisition of configuration information includes: Receive the configuration information from the display device; The configuration information includes one or more of the following: confidence attribute, normal vector, and point cloud position or color attribute for each voxel.

8. The method according to any one of claims 1 to 3, characterized in that, The first parameter information includes at least one of the following: The point cloud position, color, normal vector, confidence level of the first 3D point cloud in frame m, and the fusion time corresponding to the fusion of the incremental voxels in frame m.

9. A real-time 3D point cloud fusion and rendering device, characterized in that, include: A processor and a memory, the memory being used to store a computer program, the processor being used to call and run the computer program stored in the memory to execute: the real-time three-dimensional point cloud fusion and rendering method as described in any one of claims 1 to 8.

10. A real-time 3D point cloud fusion and rendering system, characterized in that, The system includes a scanning device and an apparatus for implementing the real-time three-dimensional point cloud fusion and rendering method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, Used to store a computer program that causes a computer to execute: the real-time 3D point cloud fusion and rendering method as described in any one of claims 1 to 8.

Images