Real-time three-dimensional (3D) point cloud fusion and rendering method, apparatus and system

Abstract

Embodiments of the present application provide a real-time 3D point cloud fusion and rendering method, apparatus and system. The method comprises: obtaining M frames of first 3D point cloud in real time, an m-th frame of the first 3D point cloud being point cloud data corresponding to an m-th frame of scanning data, M being a positive integer, and 1 ≤ m ≤ M; determining voxels, in a 3D grid structure, related to the m-th frame of the first 3D point cloud as an m-th frame of incremental voxels; traversing the m-th frame of incremental voxels, and fusing first parameter information comprised in the m-th frame of the first 3D point cloud into the m-th frame of incremental voxels, so as to obtain an m-th frame incremental fusion result corresponding to the m-th frame of incremental voxels; extracting M frames of incremental fusion results, to obtain an incremental point cloud model, the M frames of incremental fusion results comprising the m-th frame incremental fusion result; obtaining configuration information, the configuration information being determined on the basis of a user requirement; and rendering the incremental point cloud model on the basis of the configuration information. Therefore, the consumption of computing power and memory can be reduced, and consumer-grade devices can be fused in real time to obtain a high-quality point cloud model.

Description

Real-time 3D point cloud fusion and rendering method, device and system

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on September 15, 2023, with application number 202311200056.7 and titled “Real-time three-dimensional point cloud fusion and rendering method, device and system”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of image processing, and more specifically, to a real-time three-dimensional point cloud fusion and rendering method, device and system. Background Art

[0003] In computer graphics within computer vision, real-time 3D point cloud fusion and rendering is the process of determining the shape and / or appearance of real-world objects. This process can be broadly divided into image scanning, image information fusion, and model rendering.

[0004] While industrial-grade scanning equipment and processors can achieve high-quality, real-time 3D point cloud fusion and rendering, they are expensive. With the emergence of consumer-grade devices, which offer advantages such as low cost and convenience, real-time 3D point cloud fusion and rendering can also be achieved using these devices. However, the image quality captured by consumer-grade scanning devices is poor, which in turn affects the quality of the reconstructed 3D model.

[0005] Currently, real-time 3D point cloud fusion and rendering can be performed through full fusion and rendering, but this method requires a lot of computing power and memory.

[0006] Therefore, how to ensure the quality of real-time 3D point cloud fusion and rendering on consumer-grade devices while reducing computing power and memory requirements has become an urgent problem to be solved.

[0007] Summary of the Invention

[0008] The embodiments of the present application provide a real-time three-dimensional point cloud fusion and rendering method, device and system, which can reduce the consumption of computing power and memory, and enable consumer-grade devices to obtain high-quality point cloud models through real-time fusion.

[0009] In a first aspect, a three-dimensional image reconstruction method is provided, the method comprising: obtaining M frames of first 3D point cloud, wherein the first 3D point cloud of the mth frame is point cloud data corresponding to the scan data of the mth frame, M is a positive integer, 1≤m≤M. The voxels related to the first 3D point cloud of the mth frame in the three-dimensional grid structure are determined as the incremental voxels of the mth frame. The incremental voxels of the mth frame are traversed, and the first parameter information included in the first 3D point cloud of the mth frame is fused into the incremental voxels of the mth frame to obtain the incremental fusion result of the mth frame corresponding to the incremental voxels of the mth frame. The incremental fusion results of the M frames are extracted to obtain an incremental point cloud model, and the incremental fusion results of the M frames include the incremental fusion result of the mth frame. Configuration information is obtained, and the configuration information is determined according to user needs. The incremental point cloud model is rendered according to the configuration information.

[0010] Through the technical solution of the embodiment of the present application, the fusion process of the real-time three-dimensional point cloud fusion and rendering method does not traverse all voxels in the three-dimensional grid structure, but traverses the incremental voxels corresponding to the scanning data, which can reduce computing resources and memory resources, thereby helping to more efficiently obtain the real-time fused incremental point cloud model, and then perform real-time three-dimensional point cloud fusion and rendering to improve the frame rate.

[0011] In a second aspect, a real-time three-dimensional point cloud fusion and rendering device is provided, comprising an acquisition unit and a processing unit. The acquisition unit is configured to acquire M frames of first 3D point clouds, where the first 3D point cloud of the mth frame is point cloud data corresponding to the scan data of the mth frame, where M is a positive integer, 1≤m≤M. The processing unit is configured to determine voxels in a three-dimensional grid structure associated with the first 3D point cloud of the mth frame as incremental voxels of the mth frame. The processing unit is further configured to traverse the incremental voxels of the mth frame and fuse first parameter information included in the first 3D point cloud of the mth frame into the incremental voxels of the mth frame to obtain an incremental fusion result of the mth frame corresponding to the incremental voxels of the mth frame.

[0012] The technical effects corresponding to the device involved in the second aspect are similar to those of the first aspect and will not be elaborated here.

[0013] In a third aspect, a real-time three-dimensional point cloud fusion and rendering device is provided, comprising: a processor and a memory, the memory being used to store a computer program, and the processor being used to call and run the computer program stored in the memory to execute: a method as in the first aspect or any possible implementation of the first aspect.

[0014] In a fourth aspect, a real-time three-dimensional point cloud fusion and rendering system is provided, comprising a scanning device and a three-dimensional image reconstruction device for implementing the method described in the first aspect.

[0015] In a fifth aspect, a computer-readable storage medium is provided for storing a computer program, wherein the computer program enables a computer to execute: a method as in the first aspect or any possible implementation of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] FIG1 is a schematic diagram of a real-time 3D point cloud fusion and rendering system provided in an embodiment of the present application;

[0017] FIG2 is a schematic diagram of another real-time 3D point cloud fusion and rendering system provided in an embodiment of the present application;

[0018] FIG3 is a schematic diagram of a three-dimensional grid structure provided in an embodiment of the present application;

[0019] FIG4 is a flow chart of a real-time 3D point cloud fusion and rendering method provided in an embodiment of the present application;

[0020] FIG5 is a flow chart of another method for real-time 3D point cloud fusion and rendering provided by an embodiment of the present application;

[0021] FIG6 is a schematic diagram of a real-time 3D point cloud fusion and rendering device provided in an embodiment of the present application;

[0022] FIG7 is a schematic diagram of the hardware structure of a real-time three-dimensional point cloud fusion and rendering device provided by the present application. DETAILED DESCRIPTION

[0023] The technical solutions in the embodiments of the present application will be described below with reference to the accompanying drawings.

[0024] To facilitate understanding of the embodiments of the present application, the following points are explained:

[0025] First, in this application, unless otherwise specified or there is a logical conflict, the terms and / or descriptions between different embodiments are consistent and can be referenced by each other. The technical features in different embodiments can be combined to form new embodiments based on their internal logical relationships.

[0026] Second, in this application, "at least one" means one or more, and "more" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. In the text description of this application, the character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b and c can mean: a, or b, or c, or a and b, or a and c, or b and c, or a, b and c. Wherein a, b and c can be single or multiple, respectively.

[0027] Third, throughout this application, the terms "first," "second," and various numerical references (e.g., #1, #2, etc.) are used to distinguish between different 3D point clouds, rather than to describe a specific order or precedence. It should be understood that such references are interchangeable, where appropriate, to allow for the description of solutions beyond the embodiments of this application.

[0028] Fourth, in this application, expressions such as "when," "under the circumstances of," and "if" all imply that a corresponding action will be taken under certain objective circumstances. They do not limit the timeframe, do not require a judgment action to be taken when the action is taken, and do not imply any other limitations. Furthermore, the judgment action following these conditional conjunctions does not imply that the judgment action following the conditional conjunctions is the only condition for achieving the result; additional conditions may also be included to achieve the result.

[0029] Fifth, in this application, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or are inherent to these processes, methods, products or apparatuses.

[0030] The real-time three-dimensional point cloud fusion and rendering method, device and system proposed in the embodiments of the present application can be applied to scenarios such as augmented reality, virtual reality, mixed reality and 3D printing.

[0031] In order to better understand the solution of the embodiment of the present application, the possible system structure of the embodiment of the present application is briefly introduced below with reference to Figure 1.

[0032] FIG1 is a schematic diagram of a real-time three-dimensional point cloud fusion and rendering system provided in an embodiment of the present application.

[0033] As shown in Figure 1, the real-time 3D point cloud fusion and rendering system includes a scanning device 110, a computing and processing device 120, and a display device 130. The scanning device 110 and the computing and processing device 120 can achieve data transmission via a wired or wireless connection, and the computing and processing device 120 and the display device 130 can achieve data transmission via a wired or wireless connection. For example, the scanning device 110 may include a camera 111, a lidar 112, a mobile phone 113, a 3D scanner 114, and a handheld 3D scanner 115. It should be understood that the scanning device 110 is consumer-grade, and the camera 111 may be a combination of a depth camera and an RGB camera. The depth camera may be based on structured light technology, binocular vision technology, direct time of flight (DTOF) technology, or indirect time of flight (ITOF) technology. The display device 130 may include a display 131, a projector, or other display device for displaying the reconstructed 3D model.

[0034] It should be understood that the scanning device 110 and the computing and processing device 120 may be independent devices, or the computing and processing device 120 may also be the processor of the scanning device 110 , and this embodiment of the present application does not limit this.

[0035] FIG2 is a schematic diagram of another real-time three-dimensional point cloud fusion and rendering system provided in an embodiment of the present application.

[0036] As shown in Figure 2, the real-time 3D point cloud fusion and rendering system may include a scanning device 110, a cloud processing device 140, and a display device 130. The scanning device 110 uploads the scanned image to the cloud processing device 140, which then sends the reconstructed 3D model to the display device 130.

[0037] In order to facilitate understanding of the solution, a brief description of the technical terms involved in the embodiments of this application is first given.

[0038] 1. Three-dimensional grid structure

[0039] The three-dimensional grid structure can also be called a fusion field, which can be a global data cube. Figure 3 is a schematic diagram of a three-dimensional grid structure provided in an embodiment of the present application. As shown in Figure 3, the global data cube is evenly divided into n×n×n voxels according to a certain accuracy. Voxel is the basic unit of three-dimensional space. The three-dimensional grid structure can also be implemented through a spatial "hash" index. The three-dimensional grid structure can also be implemented through a multi-level spatial structure.

[0040] Consumer-grade real-time 3D scanners typically include both depth and color cameras. These systems present several challenges. Firstly, due to the relatively poor depth quality of consumer-grade depth cameras, real-time scanning can easily lead to delamination, flying spots, and noise near the model surface. Secondly, consumer-grade color cameras are typically rolling shutter cameras, which can easily cause color blur and artifacts during real-time scanning.

[0041] The image information collected by consumer-grade scanning devices has the above problems. Therefore, there are currently some solutions in the image fusion process and model rendering process to solve the above problems.

[0042] For example, the full-fusion patch rendering method traverses all voxels in the fusion field, updates the voxels in the fusion field based on the collected point cloud of the current frame, then traverses all voxels in the updated fusion field, extracts the fusion model from the fusion field, and finally, by traversing the fusion model, renders the fusion model using polygonal patches. In this method, the above problems can be solved well through the full-fusion method, but the fusion process consumes a lot of computing power and memory, and sometimes even requires acceleration through a graphics processing unit (GPU). In addition, polygonal patch rendering can easily introduce polygonal angular issues, causing model distortion.

[0043] For example, the fully fused point cloud rendering method reconstructs 3D models similarly to the fully fused patch rendering method. Each step requires traversing all voxels in the fusion field, but the difference is that dense point cloud rendering is used instead of polygonal patch rendering, resulting in high-fidelity models. However, the fully fused point cloud rendering method also consumes a lot of computing power and memory during implementation, and sometimes even requires GPU acceleration.

[0044] To address the above issues, the present invention proposes a real-time 3D point cloud fusion and rendering method, device, and system that can ensure the quality of the reconstructed 3D model while reducing computing power and memory requirements. This will be described in detail below with reference to the accompanying drawings.

[0045] FIG4 is a flow chart of a real-time three-dimensional point cloud fusion and rendering method provided in an embodiment of the present application.

[0046] S410 , obtaining the first 3D point cloud of the Mth frame, where the first 3D point cloud of the mth frame is point cloud data corresponding to the scan data of the mth frame, where M is a positive integer, 1≤m≤M.

[0047] As a possible implementation method, the first 3D point cloud of M frames is obtained through a consumer-grade lidar.

[0048] The M frames of the first 3D point cloud can be transmitted from the laser radar to the computing and processing device via a wired or wireless network.

[0049] As a possible implementation manner, M frames of first 2D images are acquired, where the first 2D images include depth information; and the M frames of first 2D images are converted into M frames of first 3D point clouds according to the depth information.

[0050] For example, M frames of first 2D images may be acquired through a depth camera.

[0051] It should be understood that the types of depth cameras have been described in detail in FIG1 and will not be repeated here.

[0052] Specifically, images may be collected based on a user's triggering operation on the scanner, or a scanning image sequence may be acquired according to a preset scanning frequency.

[0053] For another example, a depth camera and an RGB camera can be used to obtain M frames of first 2D images. In this case, the mth frame of the first 2D image includes not only depth information but also color information. The following detailed description will be given in FIG5 using the case where the first 2D image includes both depth information and color information as an example.

[0054] S420 : Determine voxels in the three-dimensional grid structure that are related to the first 3D point cloud of the m-th frame as incremental voxels of the m-th frame.

[0055] As a possible implementation manner, the incremental voxel of the m-th frame is a voxel corresponding to the surface of the first 3D point cloud of the m-th frame in the three-dimensional grid structure.

[0056] It should be understood that the area formed by the mth frame incremental voxels is not the entire area of ​​the 3D grid structure, but the area within the voxel range corresponding to the first 3D point cloud corresponding to the mth scan data frame. The specific implementation method will be described in detail in conjunction with FIG5.

[0057] S430 , traverse the m-th frame incremental voxels, and fuse the first parameter information included in the m-th frame first 3D point cloud into the m-th frame incremental voxels to obtain the m-th frame incremental fusion result corresponding to the m-th frame incremental voxels.

[0058] As a possible implementation manner, the first parameter information may include at least one of the following information: the point cloud position, color, normal vector, confidence of the first 3D point cloud of the mth frame, and the update time of updating the first area.

[0059] Here, traversing the incremental voxels of the m-th frame may also be referred to as traversing the voxels corresponding to the area formed by the incremental voxels of the m-th frame.

[0060] It should be understood that when the first parameter information includes the color of the first 3D point cloud of the mth frame, the first 2D image of the mth frame also includes color information obtained by the RGB camera. The fusion method will be described in detail below with reference to FIG5 .

[0061] In the above technical solution, the fusion process of the real-time three-dimensional point cloud fusion and rendering method does not traverse all voxels in the three-dimensional grid structure, but traverses the incremental voxels corresponding to the scanning data, which can reduce computing resources and memory resources, thereby helping to obtain a real-time fused point cloud model more efficiently.

[0062] S440 , extracting M-frame incremental fusion results to obtain an incremental point cloud model, where the M-frame incremental fusion results include an m-th frame incremental fusion result.

[0063] Exemplarily, the incremental voxels of the m-th frame are traversed, and the incremental fusion results of the m-th frame are interpolated and extracted to extract the incremental fusion results of the M frames, thereby obtaining the incremental point cloud model.

[0064] It should be understood that the incremental fusion result of the mth frame may include one or more attributes of the point cloud position, normal vector, color, and confidence corresponding to the incremental voxel. It should be understood that the type and number of attributes included in the fused voxel depends on the first parameter information used in the fusion process.

[0065] It should also be understood that during the subsequent rendering process of real-time 3D point cloud fusion and rendering, only the surface portion is rendered, that is, the portion with opposite TSDF values ​​or a TSDF value of 0. Therefore, although the resolution of the incremental point cloud model extracted above is relatively coarse, it can still ensure the basic effect of the reconstructed 3D model.

[0066] In this technical solution, rather than traversing all voxels in the 3D grid each time a fusion result is extracted, only the incremental voxels corresponding to the scanned data of that frame need to be traversed. This reduces computational resources and memory usage during point cloud model extraction, thereby saving the overall time required for real-time 3D point cloud fusion and rendering. Although the incremental point cloud model obtained through this extraction has a coarser resolution, the incremental point cloud model is extracted more quickly, thereby improving the speed of real-time 3D model reconstruction.

[0067] Voxel size determines the output resolution. Due to computing power limitations, voxel size can be large. If the point cloud model rendered directly is sparse, that is, the resolution of the extracted point cloud model is coarse, polygonal corners may be introduced, resulting in model distortion. Therefore, to further ensure the rendering effect, the following improvements can be made to step S440.

[0068] As a possible implementation method, the extracted M-frame incremental fusion results are upsampled to obtain an incremental point cloud model.

[0069] It should be understood that the incremental point cloud model obtained after upsampling is a high-fidelity point cloud model with finer and denser resolution.

[0070] In the above technical solution, by upsampling the extracted M-frame incremental fusion results, a higher resolution and high-fidelity incremental point cloud model can be obtained, thereby reducing the model distortion in the subsequent rendering process.

[0071] Similar to the step in S430, real-time scanning of the target object generally requires multiple frames of scanning data. That is, a point cloud model can be obtained based on the multiple frames of scanning data of the real-time scanning of the target object.

[0072] As a possible implementation method, when the number of scan data frames reaches a first threshold, the voxels corresponding to the 3D point clouds of these scan data frames are traversed, and the fusion results of the fused voxels are extracted to obtain an incremental point cloud model.

[0073] In this way, the number of times the incremental point cloud model is extracted can be reduced, but there is no need to traverse all voxels in the three-dimensional grid structure every time. While reducing the computing power and memory in the traversal process to a certain extent, the frequency of point cloud model extraction is reduced, thereby reducing computing power and memory in the point cloud model extraction process.

[0074] S450, obtaining configuration information, and rendering the incremental point cloud model of S440 according to the configuration information, wherein the configuration information is determined according to user needs.

[0075] It should be understood that the rendering object is the incremental point cloud model of S440, that is, the incremental point cloud model corresponding to the incremental voxels of one frame is rendered and colored. Alternatively, if the incremental point cloud model is formed by the fusion of multiple frames of incremental data, the incremental point cloud model corresponding to the incremental voxels of multiple frames can also be rendered and colored.

[0076] It should also be understood that the incremental point cloud model of S440, which is the rendering object, can be a coarse-resolution incremental point cloud model, or it can be an up-sampled, high-fidelity and high-resolution incremental point cloud model, and the embodiment of the present application does not limit this.

[0077] As a possible implementation manner, the configuration information may be received from the display device.

[0078] It should be understood that the configuration information may be obtained through interaction between the user and the display device, and then sent to the computing and processing device via the display device.

[0079] In this way, the display device can obtain configuration information related to user needs in real time, which helps to obtain the real-time three-dimensional point cloud fusion and rendering effects required by the user and improve the user experience.

[0080] As a possible implementation manner, the configuration information may also be preset.

[0081] For example, when a user needs to observe the fusion quality of an incremental point cloud model, the rendering object included in the configuration information is the confidence attribute of each voxel of the incremental model.

[0082] Specific configuration information may be configuring voxels with confidence less than a second threshold to be red, configuring voxels with confidence greater than a third threshold to be green, and configuring voxels with confidence between the second threshold and the third threshold to be uniformly transitioned.

[0083] For another example, when the user needs to observe the details of the point cloud model, the rendering object included in the configuration information is the normal vector and point cloud position of each voxel of the incremental point cloud model.

[0084] The specific configuration information may be to configure voxels whose normal vectors are consistent with the direction of ambient light to be bright, and to configure voxels whose normal vectors have a large angle with the direction of ambient light or are blocked by the point cloud position to be dark.

[0085] In this way, when the user needs to observe the details of the point cloud model, if the uneven surface is uniformly rendered and colored bright white, the uneven geometric surface will not be visible. Through the above method, the interference of color on the geometric surface can be avoided, and the user can better observe the details of the incremental point cloud model, thereby improving the user experience.

[0086] For another example, when the user needs to observe the color of the point cloud model, the rendering object included in the configuration information is the color attribute of each voxel in the incremental point cloud model.

[0087] Specifically, when the target object is a centrally symmetrical object with a geometric surface, the scanning quality cannot be distinguished only by geometric information. For example, a sphere can be rendered and colored by the color attributes of each voxel of the incremental point cloud model, making it easier for users to observe the quality of the incremental point cloud model.

[0088] In the above technical solution, the incremental point cloud model, that is, each voxel in the incremental point cloud model, is rendered and colored. In other words, only the incremental voxels corresponding to the 3D point cloud of each frame of scan data are rendered and colored, and there is no need to render and color the areas corresponding to non-incremental voxels. This can reduce computing power and memory during the real-time 3D point cloud fusion and rendering process, while effectively suppressing the high noise and color model problems of the real-time rendering model. In addition, due to the high efficiency of incremental updates and incremental upsampling, and the intuitive rendering and coloring characteristics of multiple fusion attributes, a high-fidelity, dense, intuitive and easy-to-identify point cloud model can be obtained in real time.

[0089] The real-time 3D point cloud fusion and rendering method of an embodiment of the present application will be described in detail below with reference to Figure 5. Figure 5 takes the case where the first 2D image of M frames includes depth information and color information as an example for detailed description.

[0090] FIG5 is a flow chart of another real-time three-dimensional point cloud fusion and rendering method provided in an embodiment of the present application.

[0091] S501 : Acquire M frames of first 2D images, where the M frames of first 2D images include depth information and color information, and M is a positive integer.

[0092] Exemplarily, a depth camera captures M frames of first 2D images of a target object, wherein the mth frame of the first 2D image corresponds to the mth scan data frame. It should be understood that the mth scan data frame represents any frame of scan data captured by the depth camera.

[0093] As a possible implementation manner, M frames of first 2D images are received from a scanning device, for example, a depth camera.

[0094] It should be understood that the M-frame first 2D image with depth information differs from the brightness values ​​stored by pixels in a grayscale image. Instead, the depth information of the M-frame first 2D image is the distance from the target point to the depth camera, i.e., the depth value, stored by the pixels. A depth image with depth information is also a two-dimensional image and, therefore, is also referred to herein as the M-frame first 2D image.

[0095] S502 , estimating first parameter information of each pixel point based on the M frames of first 2D images to obtain the M frames of first 3D point cloud, where the first parameter information includes three-dimensional coordinates, a normal vector, and a confidence level.

[0096] Specifically, according to the first 2D image of the m-th frame, parameter information of each pixel in the first 2D image of the m-th frame is estimated to obtain a point cloud of the first 3D image of the m-th frame.

[0097] In other words, the depth information of the first 2D image of the mth frame is converted into a three-dimensional point cloud with attributes such as three-dimensional coordinates, normal vectors, and confidence levels.

[0098] Specifically, the first 2D image of the mth frame is transformed according to the posture projection to obtain the first 3D point cloud of the mth frame.

[0099] It should be understood that since the first 2D image of the mth frame includes not only depth information but also color information, the first parameter information of the first 3D point cloud of the mth frame used in the subsequent fusion process includes at least one of the following information: the point cloud position, color, normal vector, confidence level of the first 3D point cloud, and the fusion time corresponding to the fusion of the incremental voxels of the mth frame. The specific fusion of the incremental voxels of the mth frame will be described in detail in the subsequent process.

[0100] For example, during the fusion process, only the point cloud position, that is, the three-dimensional coordinates, of the first 3D point cloud of the mth frame can be used for fusion. The embodiment of the present application does not limit the amount of first parameter information of the 3D point cloud used in the fusion process, and can be a single one, or multiple or all of the above information.

[0101] S503 : Determine the voxels in the three-dimensional grid structure that are related to the first 3D point cloud of the m-th frame as incremental voxels of the m-th frame.

[0102] Specifically, the first 3D point cloud of the mth frame is transformed into a coordinate system of a three-dimensional grid structure. In different coordinate systems of three-dimensional grid structures, there may be different ways of determining the incremental voxels of the mth frame.

[0103] Among them, the three-dimensional grid structure can be a global data cube, a spatial "hash" index or a multi-level spatial structure.

[0104] The following specifically describes a possible implementation method for determining the incremental voxels of the mth frame using a three-dimensional structure as a global data cube and a spatial “hash” index.

[0105] S503a: Determine voxels in the three-dimensional grid structure that meet a preset cutoff distance as incremental voxels of the m-th frame. The preset cutoff distance is related to the surface of the first 3D point cloud of the m-th frame.

[0106] The incremental voxel of the m-th frame may also be understood as a corresponding voxel in the three-dimensional grid structure near the surface of the first 3D point cloud of the m-th frame.

[0107] Exemplarily, the three-dimensional grid structure may be a predefined global data cube, which is evenly divided into n×n×n voxels according to a certain accuracy, as shown in FIG3 .

[0108] Any point in the first 3D point cloud of the mth frame can be mapped to a corresponding voxel in the global data cube. For each voxel, a truncated signed distance function (TSDF) value can be obtained relative to the mth scan data frame. The TSDF value usually ranges from [-1, 1] and is defined as the truncated signed distance between voxels at the corresponding depth value.

[0109] Since the scanning device obtains the surface information of the target object, each point in each frame of 3D point cloud data is a point on the surface of the reconstructed object. Therefore, the TSDF value can also be understood as the minimum directed distance value from the voxel to the real-time 3D point cloud fusion and rendering surface. When the TSDF value is less than 0, it indicates that the voxel is outside the real-time 3D point cloud fusion and rendering object, that is, in front of the real-time 3D point cloud fusion and rendering surface; when the TSDF value is equal to 0, it indicates that the voxel coincides with the point on the 3D model surface, that is, the voxel is a point on the surface of the reconstructed object; when the TSDF value is greater than 0, it indicates that the voxel is inside the reconstructed object, that is, behind the real-time 3D point cloud fusion and rendering surface. In other words, the closer the voxel is to the real-time 3D point cloud fusion and rendering surface, the closer the TSDF value of the voxel is to 0.

[0110] Specifically, the TSDF value of the voxel corresponding to each point in the first 3D point cloud of the mth frame relative to the mth scan data frame is calculated; the voxel whose absolute value of the TSDF value is less than the preset truncation distance is determined as the incremental voxel of the mth frame.

[0111] For example, the preset cutoff distance may be 1. That is, the voxels other than those with TSDF values ​​of 1 and -1 are determined as the incremental voxels of the m-th frame.

[0112] S503b: In the three-dimensional grid structure, voxels corresponding to the first 3D point cloud of the m-th frame are determined as the incremental voxels of the m-th frame, which have a circle center that satisfies a preset radius.

[0113] Specifically, with the voxel corresponding to the first 3D point cloud of the mth frame as the center of the circle, the voxels that meet the preset radius can be implemented through spatial "hash" indexing.

[0114] S504 , traverse the m-th frame incremental voxels, and fuse the first parameter information included in the m-th frame first 3D point cloud into the m-th frame incremental voxels to obtain the m-th frame incremental fusion result corresponding to the m-th frame incremental voxels.

[0115] It should be understood that the first parameter information of the 3D point cloud corresponding to the m-th frame scanning data is fused to obtain the m-th frame incremental fusion result, thereby updating the three-dimensional grid structure, and only incrementally updating the m-th frame incremental voxels.

[0116] In other words, the incremental voxels of the mth frame are traversed and the data information of the voxels corresponding to the incremental voxels of the mth frame are updated according to the first parameter information included in the first 3D point cloud of the mth frame. The first parameter information has been described in detail in FIG4 and will not be repeated here.

[0117] The above process is only for real-time 3D point cloud fusion and rendering. It can be based on one frame of scanning data to obtain the fusion result, or it can be based on real-time scanning of multiple frames of scanning data of the target object to obtain the scanning results corresponding to the multiple frames of scanning data.

[0118] In the above technical solution, each time a frame of scanning data is obtained, only the voxels corresponding to the scanning data are traversed to realize the fusion process. Compared with traversing all voxels in the three-dimensional grid structure each time a frame of scanning data is obtained, unnecessary calculation and memory consumption can be avoided, and the real-time fusion scanning model can be obtained more efficiently.

[0119] S505 , extracting M-frame incremental fusion results to obtain an incremental point cloud model, where the M-frame incremental fusion results include an m-th frame incremental fusion result.

[0120] S506 , obtaining configuration information, and rendering the incremental point cloud model of S505 according to the configuration information, wherein the configuration information is determined according to user needs.

[0121] It should be understood that S505 and S506 may refer to S440 and S450 respectively, and are not described in detail here.

[0122] S507: Send the rendered incremental point cloud model to a display device.

[0123] As a possible implementation method, after the user sees the rendered incremental point cloud model through the display device, the display device can obtain the user's new configuration information and send the new configuration information to the computing processing device to re-render the incremental point cloud model.

[0124] Specifically, the display device interacts with the user through buttons on a user interface (UI) of the display device. During the scanning data acquisition process, the display device can acquire the user's configuration information based on the user's button operations on the UI.

[0125] In the above technical solution, the real-time rendered point cloud model is presented in real time on the display device, which helps the user determine the area that needs to be scanned to obtain more data frames based on the surface quality of the incremental point cloud model presented by the display device, thereby subsequently improving the scanning efficiency and scanning quality, and providing intuitive and visual feedback of the scanning data.

[0126] The above describes a detailed description of the real-time 3D point cloud fusion and rendering method. Below, a brief description of the real-time 3D point cloud fusion and rendering apparatus will be provided in conjunction with Figures 6 and 7. It should be understood that the description of the apparatus embodiment corresponds to the description of the method embodiment. Therefore, any details not described in detail can be referred to above in the method embodiment, and for the sake of brevity, they will not be repeated here.

[0127] 6 is a schematic diagram of a real-time 3D point cloud fusion and rendering device provided in an embodiment of the present application. The real-time 3D point cloud fusion and rendering device includes: an acquisition unit 601 and a processing unit 602.

[0128] The acquisition unit 601 is configured to acquire M frames of first 3D point cloud, where the m-th frame of the first 3D point cloud is point cloud data corresponding to the m-th frame of scanning data, and M is a positive integer, 1≤m≤M.

[0129] The processing unit 602 is configured to determine voxels in the three-dimensional grid structure that are related to the first 3D point cloud of the m-th frame as incremental voxels of the m-th frame.

[0130] The processing unit 602 is further configured to traverse the m-th frame incremental voxels and fuse the first parameter information included in the m-th frame first 3D point cloud into the m-th frame incremental voxels to obtain the m-th frame incremental fusion result corresponding to the m-th frame incremental voxels.

[0131] The processing unit 602 is specifically configured to extract M-frame incremental fusion results to obtain an incremental point cloud model, where the M-frame incremental fusion results include an m-th frame incremental fusion result.

[0132] The acquisition unit 601 is further configured to acquire configuration information, which is determined based on user requirements.

[0133] The processing unit 602 is further configured to render the incremental point cloud model according to the configuration information.

[0134] Optionally, the incremental voxel of the m-th frame is a voxel corresponding to the first 3D point cloud of the m-th frame in the three-dimensional grid structure.

[0135] Optionally, the processing unit 602 is specifically configured to

[0136] Voxels within a preset cutoff distance in the 3D grid structure are determined as incremental voxels for the mth frame, where the preset cutoff distance is related to the surface of the first 3D point cloud in the mth frame. Alternatively, voxels within a preset radius, centered around the voxel corresponding to the first 3D point cloud in the mth frame, are determined as incremental voxels for the mth frame.

[0137] Optionally, the acquisition unit 601 is specifically configured to acquire M frames of first 2D images, the M frames of first 2D images including depth information. The processing unit 602 is specifically configured to convert the M frames of first 2D images into M frames of first 3D point clouds according to the depth information.

[0138] Optionally, the processing unit 602 is specifically configured to upsample the extracted M-frame incremental fusion results to obtain an incremental point cloud model.

[0139] Optionally, the real-time 3D point cloud fusion and rendering apparatus further includes a receiving unit 603. The receiving unit 603 is configured to send the rendered incremental point cloud model to a display device.

[0140] Optionally, the receiving unit 603 is configured to receive configuration information from a display device.

[0141] Optionally, the first parameter information is at least one of the following information: point cloud position, color, normal vector, confidence level of the first 3D point cloud, and fusion time corresponding to the fusion of incremental voxels of the m-th frame.

[0142] It should be understood that the above description is merely an exemplary description. The real-time 3D point cloud fusion and rendering device is used to perform the methods or steps mentioned in the aforementioned method embodiments. Therefore, the device corresponds to the aforementioned method embodiments. For details, please refer to the description of the aforementioned method embodiments and will not be repeated here.

[0143] Figure 7 is a schematic diagram of the hardware structure of a real-time 3D point cloud fusion and rendering device provided by an embodiment of the present application. The agricultural trajectory point recognition device 700 shown in Figure 7 may include: a memory 710, a processor 720, and a communication interface 730. The memory 710, processor 720, and communication interface 730 are interconnected via an internal connection path. The memory 710 is used to store instructions, and the processor 720 is used to execute the instructions stored in the memory 720 to control the input / output interface 730 to receive / send at least some parameters of configuration information. Optionally, the memory 710 may be coupled to the processor 720 via an interface or integrated with the processor 720.

[0144] It should be noted that the communication interface 730 uses a transceiver device such as, but not limited to, a transceiver to implement communication between the communication device 700 and other devices or a communication network. The communication interface 730 may also include an input / output interface.

[0145] During implementation, each step of the above method can be completed by an integrated logic circuit of the hardware in the processor 720 or by instructions in the form of software. The method disclosed in conjunction with the embodiments of the present application can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor. The software module can be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 710, and the processor 720 reads the information in the memory 710 and completes the steps of the above method in combination with its hardware. To avoid repetition, it will not be described in detail here.

[0146] It should be understood that in the embodiments of the present application, the processor may be a central processing unit (CPU), a GPU, or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0147] As a possible implementation method, the above-mentioned real-time 3D point cloud fusion and rendering method is calculated by a CPU single thread.

[0148] As a possible implementation method, the above-mentioned real-time three-dimensional point cloud fusion and rendering method is performed by multi-threaded computing on computing devices such as CPU or GPU.

[0149] It should also be understood that in the embodiments of the present application, the memory in any of the above embodiments may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM). By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0150] As a possible implementation method, the memory area corresponding to the measured point cloud model and the memory area corresponding to the rendered incremental point cloud model are extracted and separated.

[0151] It should be understood that a memory copy is required during the process of rendering the incremental point cloud model.

[0152] In the above technical solution, only the voxels of the corresponding area that have undergone the fusion process need to be copied, thereby reducing resource consumption.

[0153] It should be understood that the term "and / or" in this document simply describes a relationship between related objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this document generally indicates that the related objects are in an "or" relationship.

[0154] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0155] An embodiment of the present application also provides a real-time three-dimensional point cloud fusion and rendering system including a scanning device 110 and a computing and processing device 120, wherein the computing and processing device 120 can be the above-mentioned real-time three-dimensional point cloud fusion and rendering device 600 or the real-time three-dimensional point cloud fusion and rendering device 700.

[0156] Optionally, the computing and processing device 120 may be located in a cloud processing device 140 .

[0157] Optionally, if the computing and processing device 120 is located in the cloud processing device 140 , the system further includes a device for receiving instructions from the computing and processing device 120 .

[0158] Optionally, the real-time 3D point cloud fusion and rendering system may further include a display device 130 .

[0159] An embodiment of the present application also provides a device, which includes a processing unit and a storage unit, wherein the storage unit is used to store instructions, and the processing unit executes the instructions stored in the storage unit so that the device performs the above-mentioned real-time three-dimensional point cloud fusion and rendering method.

[0160] An embodiment of the present application further provides a computer-readable medium storing a program code. When the computer program code is executed on a computer, the computer executes the method in FIG. 4 or FIG. 5 .

[0161] An embodiment of the present application also provides a chip, comprising: at least one processor and a memory, wherein the at least one processor is coupled to the memory and is configured to read and execute instructions in the memory to execute the method in FIG. 4 or FIG. 5 above.

[0162] An embodiment of the present application also provides an agricultural vehicle, comprising: at least one processor and a memory, wherein the at least one processor is coupled to the memory and is configured to read and execute instructions in the memory to execute the method in FIG. 4 or FIG. 5 .

[0163] As used in this specification, the terms "component," "unit," "system," and the like are used to represent computer-related entities, hardware, firmware, a combination of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, and / or a computer. By way of illustration, both an application running on a computing device and a computing device can be a component. One or more components can reside in a process and / or an execution thread, and a component can be located on a computer and / or distributed between two or more computers. In addition, these components can be executed from various computer-readable media having various data structures stored thereon. Components can communicate, for example, via local and / or remote processes based on signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system, and / or a network, such as the Internet interacting with other systems via signals).

[0164] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0165] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0166] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0167] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0168] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0169] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory ROM, a random access memory RAM, a magnetic disk, or an optical disk.

[0170] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A real-time 3D point cloud fusion and rendering method, characterized in that: include: Acquire M frames of first 3D point cloud in real time, where the first 3D point cloud of the mth frame is point cloud data corresponding to the mth frame of scanning data, where M is a positive integer, 1≤m≤M; Determine the voxels in the three-dimensional grid structure that are related to the first 3D point cloud in the m-th frame as the m-th frame incremental voxels; Traversing the m-th frame incremental voxels, fusing the first parameter information included in the first 3D point cloud of the m-th frame into the m-th frame incremental voxels to obtain the m-th frame incremental fusion result corresponding to the m-th frame incremental voxels; Extracting M-frame incremental fusion results to obtain an incremental point cloud model, wherein the M-frame incremental fusion results include the m-th frame incremental fusion result; Acquire configuration information, where the configuration information is determined according to user needs; The incremental point cloud model is rendered according to the configuration information.

2. The method according to claim 1, characterized in that The m-th frame incremental voxel is a voxel corresponding to the first 3D point cloud in the m-th frame in the three-dimensional grid structure.

3. The method according to claim 2, characterized in that The step of determining the voxels in the three-dimensional grid structure related to the first 3D point cloud in the m-th frame as the m-th frame incremental voxels comprises: Determine voxels within a preset cutoff distance in the three-dimensional grid structure as the m-th frame incremental voxels, wherein the preset cutoff distance is related to the surface of the first 3D point cloud in the m-th frame; or In the three-dimensional grid structure, a voxel corresponding to the first 3D point cloud in the m-th frame is taken as a circle center and a voxel that satisfies a preset radius is determined as the m-th frame incremental voxel.

4. The method according to any one of claims 1 to 3, characterized in that: The obtaining of the first 3D point cloud of the M frame comprises: Acquire M frames of first 2D images, where the M frames of first 2D images include depth information; The M frames of first 2D images are converted into the M frames of first 3D point clouds according to the depth information.

5. The method according to any one of claims 1 to 3, characterized in that: The extracting of the M-frame incremental fusion results to obtain an incremental point cloud model includes: The extracted M-frame incremental fusion results are upsampled to obtain the incremental point cloud model.

6. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: Send the rendered incremental point cloud model to the display device.

7. The method according to claim 6, characterized in that The obtaining of configuration information includes: receiving the configuration information from the display device; The configuration information includes: one or more of a confidence attribute, a normal vector and a point cloud position or a color attribute of each voxel.

8. The method according to claims 1 to 3, characterized in that The first parameter information is at least one of the following information: The point cloud position, color, normal vector, confidence of the first 3D point cloud in the mth frame and the fusion time corresponding to the fusion of the incremental voxels of the mth frame.

9. A real-time 3D point cloud fusion and rendering device, characterized in that: include: A processor and a memory, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute: the real-time three-dimensional point cloud fusion and rendering method as described in any one of claims 1 to 8.

10. A real-time 3D point cloud fusion and rendering system, characterized in that: The system includes a scanning device and an apparatus for implementing the real-time three-dimensional point cloud fusion and rendering method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: Used to store a computer program, wherein the computer program enables a computer to execute: the real-time three-dimensional point cloud fusion and rendering method as described in any one of claims 1 to 8.

Images