WebGL-based point cloud rendering real-time hole repairing method and system

Abstract

The invention discloses a WebGL-based point cloud rendering real-time hole repairing system and method, and the method comprises the steps: rendering a first depth point cloud texture, and coding a depth value into an RGBA channel; rendering a second depth point cloud texture with the amplified point size by using average point spacing information generated by an octree structure, and encoding a depth value into an RGBA channel; comparing whether the depth values obtained by decoding the same pixel of the first depth point cloud texture and the second depth point cloud texture are consistent or not, and if not, judging that the region is a hole of the point cloud; and when a hole exists, edge detection and intelligent interpolation filling are carried out. According to the method, a multi-layer depth texture comparison mechanism and an octree space segmentation strategy are introduced, efficient detection and filling of point cloud holes are realized, and meanwhile, the real-time rendering fluency and data integrity of a Web end are ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of point cloud rendering and hole filling, and in particular to a method, system, device and medium for repairing real-time holes in point cloud rendering based on WebGL. Background Art

[0002] Point cloud data usually contains xyz coordinates and the RGB color values ​​of the corresponding coordinates. The distance between each point is called the point spacing.

[0003] Point cloud data is collected by equipment, which can collect three-dimensional spatial data at a specified time. However, due to limitations such as the equipment collection distance, the collected point cloud will have holes (i.e., there are large point spacings between some points).

[0004] The point cloud is rasterized and rendered based on WebGL (that is, colored using the RGB color values ​​of the points) and rendered as a color texture. At this time, the color texture may have holes due to the large point spacing. Therefore, it is important to automatically identify and fill the holes rendered in the color texture.

[0005] Application areas of point cloud rendering and hole filling technology:

[0006] 1. Public safety 3D scene reconstruction and visualization: In the field of public safety, the point cloud scheduling and rendering technology of the present invention can be used to reconstruct and visualize the 3D scene of an accident scene, disaster area or crime scene in real time. Relevant departments can quickly browse the 3D points of the reconstructed scene to provide support for emergency response, investigation and evidence collection, and safety assessment.

[0007] 2. 3D modeling and visualization: In architectural design, urban planning and virtual reality, real-time point cloud rendering technology is used to visualize three-dimensional models in real time, allowing designers to intuitively view and modify designs.

[0008] 3. Geographic Information System (GIS): Through real-time rendering and hole filling technology, point cloud data (such as LiDAR data) is integrated into GIS applications to improve the display effect and analysis accuracy of geographic data such as terrain and buildings.

[0009] 4. Smart City: In the construction of smart cities, real-time point cloud data is used to monitor urban infrastructure, update urban models in real time, and provide data support for urban management and decision-making.

[0010] 5. Online education and training: In online education platforms, 3D point cloud rendering technology is used to create virtual laboratories or interactive learning environments, allowing students to explore and interact in real time, thereby enhancing learning effects.

[0011] Potential application areas:

[0012] 1. Environmental monitoring and management: In the field of environmental protection and resource management, point cloud rendering technology is used to monitor natural resources in real time, evaluate environmental changes, and support sustainable development decisions.

[0013] 2. Cultural heritage protection: 3D scanning and real-time rendering of historical sites or cultural relics for display and protection on digital platforms, providing support for cultural heritage research and education.

[0014] Technical features of the prior art:

[0015] 1. Poisson reconstruction algorithm: mainly generates a surface model by sampling point cloud data. The algorithm uses the normal information and position of the point to derive a smooth surface by solving a Poisson equation. This method requires the calculation of the normal vector of each point in advance and assumes that the point cloud density is uniform.

[0016] 2. Depth texture generation: Existing technologies usually generate depth textures to represent the depth information of each pixel in the scene. The depth texture is generally encoded in 8-bit or 16-bit grayscale format, which limits the accuracy of the depth value and may cause quantization errors when processing large-scale scenes.

[0017] 3. Grayscale encoding and depth information: In existing technologies, depth information is usually presented in the form of grayscale images, and the depth value directly corresponds to the grayscale value. This method is susceptible to errors when comparing different depths, especially in scenes where objects are larger near and smaller far away, making it difficult to accurately distinguish the depth information of close and distant objects.

[0018] 4. Computational complexity and performance issues: The Poisson reconstruction process involves complex numerical calculations and solution processes, which takes up a lot of time and memory, especially when processing large-scale point cloud data, which may lead to insufficient real-time processing capabilities. In addition, the complexity of the algorithm requires a large amount of memory and computing resources during the implementation process, limiting its application in resource-constrained environments (such as web browsers).

[0019] 5. Hole filling method: Hole filling in existing technologies usually relies on a simple neighborhood interpolation algorithm, which interpolates the color or depth information of surrounding known points. This method does not work well when dealing with complex contours or high-frequency details, and may lead to visually incoherent filling results.

[0020] 6. Dependence on CPU calculation: Most existing technologies rely on the CPU for processing and calculation of point cloud data, especially when performing complex reconstruction and interpolation algorithms. This design results in slow processing speed and high dependence on computing resources, making it difficult to achieve efficient performance in real-time rendering scenarios.

[0021] In point cloud rendering and hole filling technology, most existing methods rely on point cloud interpolation technology based on Poisson sampling. This method mainly fills the holes in the point cloud data by inserting new points, making the overall data present a more complete structure. However, in practical applications, this type of technology has the following major defects on the browser side:

[0022] 1. Low computational efficiency

[0023] Poisson sampling is a complex interpolation algorithm that requires a lot of calculations to determine the appropriate position of the insertion point. Due to the limited computing power of the Web end, it cannot support high-complexity real-time calculations, resulting in increased rendering delays, making the filled point cloud rendering unable to meet real-time requirements. This is an obvious bottleneck for applications that require interactive operations.

[0024] 2. Large memory usage

[0025] Poisson sampling requires intensive calculation of the point cloud hole area and keeps the relevant information of the interpolation point (such as color and position information), which will significantly increase the memory load of the browser. Especially in large-scale point cloud scenes, the memory limitation of the browser makes this method unsuitable for real-time applications on the web. High memory consumption not only affects the rendering smoothness, but may also cause the browser to crash or data loading failure.

[0026] 3. Complex color value processing

[0027] The Poisson sampling method needs to refer to the color information of surrounding points when generating new points to maintain visual consistency. Since point cloud data usually has a large resolution, the sampling algorithm frequently reads and calculates the color values ​​of adjacent points, which incurs additional computing and memory overhead. This complexity of color processing further prolongs the computing cycle and affects the real-time rendering effect of the Web end.

[0028] 4. Long calculation cycle

[0029] Poisson sampling frequently performs spatial interpolation operations during the calculation process, which is cumbersome and leads to a long calculation cycle. In a browser rendering environment that requires high-frequency updates, this bottleneck in calculation time often prevents the interpolated data from being displayed in a timely manner, seriously affecting the user's interactive experience. Summary of the invention

[0030] The purpose of the present invention is to address the inefficiency and memory usage problems of existing solutions on the browser side, and propose a real-time point cloud rendering and hole filling method based on WebGL to achieve more efficient point cloud data processing and rendering.

[0031] The objective of the present invention is achieved through the following technical solutions:

[0032] A real-time hole repair method for point cloud rendering based on WebGL, comprising:

[0033] Render the first depth point cloud texture and encode the depth value into the RGBA channel;

[0034] Using the average point spacing information generated by the octree structure, the second depth point cloud texture is rendered with the enlarged point size, and the depth value is encoded into the RGBA channel;

[0035] Compare the first depth point cloud texture and the second depth point cloud texture to see if the decoded depth values ​​of the same pixel are consistent. If they are inconsistent, determine that this area is a hole in the point cloud;

[0036] When holes exist, edge detection and intelligent interpolation filling are performed.

[0037] Preferably, before comparing whether the depth values ​​obtained by decoding the same pixel of the first depth point cloud texture and the second depth point cloud texture are consistent, the RGB color value of the point is used for coloring.

[0038] The present invention realizes efficient detection of holes by rendering deep textures with two different point sizes. First, the first depth point cloud texture is rendered according to the normal point size, and the depth value is encoded into the RGBA channel. Then, the second depth point cloud texture is rendered with the enlarged point size using the average point spacing information generated by the octree structure. The enlarged depth texture covers more areas, so that potential hole areas can be detected by subsequent texture comparison.

[0039] The comparison principle of the first depth point cloud texture and the second depth point cloud texture is based on pixel coverage. When a pixel in the enlarged texture has a depth value, but the corresponding pixel in the normal texture has no depth value, this area can be determined as a hole in the point cloud. This detection method avoids complex spatial interpolation and sampling calculations, and realizes rapid identification of hole areas through simple texture comparison, effectively improving the efficiency of hole detection.

[0040] Preferably, before rendering the first depth point cloud texture, the point cloud data is hierarchically organized using an octree structure on the server side; when constructing the octree structure, the average point spacing information of each node is recorded.

[0041] Preferably, the edge detection and intelligent interpolation filling are specifically as follows:

[0042] The process of edge detection and intelligent interpolation filling is completed in the fragment shader;

[0043] Compare the depth value of each fragment with the depth values ​​of the surrounding adjacent fragments to determine whether the hole area is outside the contour;

[0044] If the hole area is outside the contour, the area is filled with intelligent interpolation.

[0045] Preferably, the intelligent interpolation filling of the area is: using the color and depth values ​​of the surrounding adjacent points, and generating new filling points through an interpolation algorithm, so that the filling points are consistent with the surrounding adjacent points in visual effect.

[0046] Preferably, the intelligent interpolation filling algorithm performs color interpolation on adjacent fragments.

[0047] The octree structure is used on the server to organize the point cloud data in layers to improve transmission and rendering efficiency.

[0048] When constructing the octree, the average point spacing information of each node is recorded to provide a reference for the enlarged rendering of the depth texture.

[0049] Through intelligent scheduling algorithms, node data within the user's perspective is transmitted first, reducing the computing and memory burden on the Web side.

[0050] The present invention uses an octree structure on the server side to organize point cloud data in layers to improve transmission and rendering efficiency. When constructing the octree, the average point spacing information of each node is recorded to provide a reference for the enlarged rendering of the depth texture. In addition, through the intelligent scheduling algorithm, the node data within the user's perspective is transmitted first, reducing the computing and memory burden of the Web side.

[0051] The octree structure provides hierarchical management and on-demand loading capabilities, making the transmission and rendering of point cloud data more efficient. Due to memory limitations and network bandwidth constraints on the Web side, the use of octree data scheduling can significantly reduce the amount of data and ensure the smoothness of the real-time rendering process. At the same time, by recording the average point spacing to generate an enlarged depth texture, it provides an accurate reference for hole detection and filling.

[0052] After the hole detection is completed, the fragment shader further performs edge judgment based on the surrounding depth values ​​to determine whether the hole area is located at the edge of the contour. If the hole area is outside the contour, the area is intelligently interpolated and filled. When filling, the present invention uses the color and depth values ​​of the surrounding adjacent points to generate new filling points through a simple interpolation algorithm, so that the supplementary point cloud is consistent with the original data in visual effect.

[0053] A point cloud rendering real-time hole repair system based on WebGL, comprising a first depth point cloud texture rendering module, a second depth point cloud texture rendering module, a color point cloud rendering module, a hole judgment module, and an edge detection and intelligent interpolation filling module;

[0054] The first depth point cloud texture rendering module renders a first depth point cloud texture and encodes the depth value into an RGBA channel;

[0055] The second depth point cloud texture rendering module uses the average point spacing information generated by the octree structure to render the second depth point cloud texture with the enlarged point size, and encodes the depth value into the RGBA channel;

[0056] The color point cloud rendering module uses the RGB color value of the point to color the point cloud;

[0057] The hole judgment module compares the first depth point cloud texture and the second depth point cloud texture to determine whether the depth values ​​obtained by decoding the same pixel are consistent. If they are inconsistent, the region is determined to be a hole in the point cloud.

[0058] The hole judgment module is connected to the edge detection and intelligent interpolation filling modules.

[0059] A computer device includes a processor, wherein the processor is used to execute a computer program stored in a memory to implement the above-mentioned WebGL-based point cloud rendering real-time hole repair method.

[0060] A computer-readable storage medium stores a computer program, and a processor is used to execute the computer program stored in the storage medium to implement the above-mentioned WebGL-based point cloud rendering real-time hole repair method.

[0061] Beneficial effects:

[0062] Enhanced real-time performance: Fast detection and filling of hole areas are achieved based on multi-layer depth texture comparison, avoiding high-computation sampling methods, so that hole filling can be completed in real time in the WebGL environment.

[0063] Reduced memory usage: Using lightweight interpolation methods instead of complex color sampling significantly reduces memory requirements and is suitable for use in browser environments with limited memory.

[0064] Improved rendering fluency: Traditional Poisson sampling is eliminated to reduce the pressure on rendering and the occupation of computing resources during the sampling process. At the same time, the combination of the octree structure and the scheduling algorithm realizes on-demand loading and hierarchical transmission, avoiding unnecessary redundant data transmission, thus ensuring the real-time rendering effect of the point cloud.

[0065] Visual consistency: Intelligent interpolation fill maintains the continuity of the depth and color of the supplemented points with the surrounding points, making the rendered point cloud more natural and complete in visual effect.

[0066] Through this innovative WebGL-based real-time rendering and hole filling technology, the present invention overcomes the limitations of existing solutions in Web-side applications and provides a more efficient and reliable solution for real-time display and interaction of point cloud data. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 It is a rendering flow chart of the real-time hole repair method of point cloud rendering based on WebGL of the present invention.

[0068] Figure 2 It is a rendering timing diagram of the real-time hole repair method of point cloud rendering based on WebGL of the present invention.

[0069] Figure 3 It is a flowchart of intelligent interpolation filling.

[0070] Figure 4 It is a structural schematic diagram of the real-time hole repair system of point cloud rendering based on WebGL of the present invention. DETAILED DESCRIPTION

[0071] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present invention.

[0072] The terms "first", "second", etc. in the claims and specification of the present application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable under appropriate circumstances. This is merely a way of distinguishing objects with the same properties in the embodiments of the present application. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, so that a process, method, system, product or apparatus that includes a series of units is not necessarily limited to those units, but may include other units not expressly listed or inherent to these processes, methods, products or apparatus.

[0073] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by technicians in the technical field to which this application belongs. The terms used in this article and in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application.

[0074] like Figure 1 As shown, the rendering process of the real-time hole repair method of point cloud rendering based on WebGL of the present invention is as follows:

[0075] Render depth point cloud textures with normal point size;

[0076] Render the magnified depth point cloud texture based on the average point gap generated by the octree;

[0077] Normal rendering of color point cloud;

[0078] Compare two depth textures;

[0079] Whether there are cavities;

[0080] in the case of,

[0081] Perform edge detection;

[0082] Intelligent interpolation to fill the hole area;

[0083] Update the rendering results.

[0084] If no,

[0085] Finish.

[0086] Figure 2 It is a rendering timing diagram of the real-time hole repair method of point cloud rendering based on WebGL of the present invention.

[0087] In a disclosed embodiment of the present invention, a real-time hole repair method for point cloud rendering based on WebGL includes:

[0088] Render the first depth point cloud texture and encode the depth value into the RGBA channel;

[0089] Using the average point spacing information generated by the octree structure, the second depth point cloud texture is rendered with the enlarged point size, and the depth value is encoded into the RGBA channel;

[0090] Color using the rgb color value of the point;

[0091] Compare the first depth point cloud texture and the second depth point cloud texture to see if the decoded depth values ​​of the same pixel are consistent. If they are inconsistent, determine that this area is a hole in the point cloud;

[0092] When holes exist, edge detection and intelligent interpolation filling are performed.

[0093] The present invention achieves efficient detection of holes by rendering deep textures with two different point sizes. First, a deep point cloud texture is rendered according to the normal point size, and the depth value is encoded into the RGBA channel. Then, the average point spacing information generated by the octree structure is used to render a second deep point cloud texture with an enlarged point size. The enlarged depth texture covers more area, so that potential hole areas can be detected through subsequent texture comparison.

[0094] The comparison principle of two depth textures is based on pixel coverage. There will be holes in the depth texture before the point cloud is enlarged. When the depth information obtained by decoding the same pixel (RGBA) in the two textures before and after enlargement is inconsistent, the area is a hole; for example, when a pixel in the enlarged texture has a depth value, but the corresponding pixel in the normal texture has no depth value, it can be determined that this area is a hole in the point cloud. This detection method avoids complex spatial interpolation and sampling calculations, and realizes rapid identification of hole areas through simple texture comparison, effectively improving the efficiency of hole detection.

[0095] Depth information uses float32, which is 4 bytes long = 32 bits. Each RGBA channel is 8 bits, and a total of 4 channels can encode 32 bits of content. Choosing to use RGBA channels to encode high-precision depth textures instead of traditional single-channel grayscale depth textures brings significant technical advantages:

[0096] Higher depth accuracy

[0097] Using RGBA channels for depth encoding can significantly improve the accuracy of depth data. Usually, grayscale depth textures use a single 8-bit channel to represent depth information, which is not accurate enough in some scenarios, especially for point cloud data with large ranges or high precision requirements. RGBA channels provide 4 8-bit channels, decomposing and storing depth values ​​in each channel, so as to achieve 32-bit accuracy. This method can more accurately restore depth relationships in large scenes or multi-level structures and reduce errors caused by insufficient depth quantization.

[0098] Reducing depth approximation error

[0099] Grayscale depth textures are prone to depth approximation problems when representing near and far objects, especially in close-range point cloud rendering, where high depth changes may cause layer mismatch due to insufficient precision. The RGBA encoding method can split the depth value into higher-bit representations, thereby avoiding approximation errors between near and far levels and making the depth value of the edge of the object more continuous. In hole filling, this high precision can ensure more accurate depth judgment of the edge area and avoid misjudgment.

[0100] Improving the dynamic range of depth data

[0101] The method of separating depth values ​​by RGBA channels can expand the dynamic range of depth data, allowing depth values ​​to be distributed in a wider numerical space. For scenes with large height differences or wide viewing angles in point cloud data, RGBA depth encoding can record subtle depth changes without sacrificing accuracy, effectively maintaining the integrity and accuracy of the rendered scene. This enhanced dynamic range can significantly improve the overall visual effect and maintain the layered sense of scene depth when filling holes on the Web.

[0102] Facilitates simultaneous processing of color and depth

[0103] It is also feasible to use RGBA channels to encode depth information and other data (such as high-precision normals or contour information) in a single texture. This design can achieve synchronous data processing. For example, during the rendering process, the high-precision depth value of RGBA can not only be used to detect hole areas, but also some color or normal information can be directly integrated into the same texture. This structure reduces texture switching and computational overhead during the rendering process, and is particularly suitable for real-time rendering tasks on the browser side.

[0104] Compatibility and performance optimization

[0105] In modern WebGL environments, the encoding of RGBA channels has good compatibility. Most WebGL implementations provide hardware acceleration support for the RGBA format, which can efficiently render and process high-precision texture data without sacrificing performance. In addition, the RGBA encoding method is easy to align with the data format of other rendering pipelines in the depth filling algorithm, achieving unified format management, which helps to further optimize the utilization of memory and computing resources.

[0106] In a disclosed embodiment of the present invention, before rendering the first depth point cloud texture, the point cloud data is hierarchically organized using an octree structure at the server;

[0107] When constructing the octree structure, the average point spacing information of each node is recorded.

[0108] Through intelligent scheduling algorithms, node data within the user's perspective is transmitted first, reducing the computing and memory burden on the Web side.

[0109] The present invention uses an octree structure on the server side to organize point cloud data in layers to improve transmission and rendering efficiency. When constructing the octree, the average point spacing information of each node is recorded to provide a reference for the enlarged rendering of the depth texture. In addition, through the intelligent scheduling algorithm, the node data within the user's perspective is transmitted first, reducing the computing and memory burden of the Web side.

[0110] The octree structure provides hierarchical management and on-demand loading capabilities, making the transmission and rendering of point cloud data more efficient. Due to memory limitations and network bandwidth constraints on the Web side, the use of octree data scheduling can significantly reduce the amount of data and ensure the smoothness of the real-time rendering process. At the same time, by recording the average point spacing to generate an enlarged depth texture, it provides an accurate reference for hole detection and filling.

[0111] In a disclosed embodiment of the present invention, the process of edge detection and intelligent interpolation filling is completed in a fragment shader.

[0112] This design makes full use of the parallel computing power of the GPU to improve rendering efficiency and filling quality.

[0113] GPU parallel computing has the following advantages:

[0114] Efficient parallel processing capabilities

[0115] GPU has a large-scale parallel processing architecture and can handle the calculation tasks of a large number of fragments at the same time. During the edge detection and intelligent interpolation filling process, each fragment can be calculated independently without relying on the results of other fragments. This parallel computing capability makes the edge detection process fast and efficient, unaffected by the size of the data, and suitable for real-time rendering tasks of large-scale point cloud data.

[0116] Real-time edge detection

[0117] In the fragment shader, the edge detection algorithm can directly compare the depth value of each fragment with the depth values ​​of the surrounding adjacent fragments to determine whether the fragment is at the edge of the contour. Through parallel calculation, edge detection can be completed quickly in each rendering frame, ensuring that users get instant feedback during interactive operations. This method is much faster than the traditional CPU serial calculation method, greatly improving the user experience.

[0118] Efficient implementation of intelligent interpolation algorithm

[0119] After detecting the hole area, the intelligent interpolation filling algorithm interpolates the colors of adjacent fragments in the fragment shader. This process can use simple linear interpolation or weighted average algorithms to quickly calculate the color values ​​around each hole fragment. Due to the low latency of the GPU when processing parallel data, the interpolation process is not only fast, but also able to update the filling results in real time to maintain the smoothness of rendering.

[0120] Reduce CPU Load

[0121] Transferring edge detection and interpolation filling calculations to the GPU can significantly reduce the burden on the CPU, allowing it to focus on other logical operations and interactive responses. This design achieves more efficient resource utilization and optimizes the overall system performance.

[0122] like Figure 3 As shown, in a disclosed embodiment of the present invention, the depth value of each fragment is compared with the depth values ​​of the surrounding adjacent fragments to determine whether the hole area is outside the contour;

[0123] If the hole area is outside the contour, the area is filled with intelligent interpolation.

[0124] The intelligent interpolation filling of the area is: using the color and depth values ​​of the surrounding adjacent points, and generating new filling points through an interpolation algorithm, so that the filling points are consistent with the surrounding adjacent points in visual effect.

[0125] The intelligent interpolation filling algorithm performs color interpolation on adjacent fragments.

[0126] After the hole detection is completed, the fragment shader further performs edge judgment based on the surrounding depth values ​​to determine whether the hole area is located at the edge of the contour. If the hole area is outside the contour, the area is intelligently interpolated and filled. When filling, the present invention uses the color and depth values ​​of the surrounding adjacent points to generate new filling points through a simple interpolation algorithm, so that the supplementary point cloud is consistent with the original data in visual effect.

[0127] The core of intelligent interpolation filling is to use the color values ​​and depth data of adjacent points to generate reasonable supplementary points for the hole area through linear interpolation or weighted average. This method can not only effectively reduce the amount of calculation, but also ensure the consistency of color and depth, and avoid the complex color calculation in methods such as Poisson sampling. This lightweight interpolation method has extremely low memory usage and is particularly suitable for real-time rendering on the browser side.

[0128] like Figure 4 As shown, in a disclosed embodiment of the present invention, a point cloud rendering real-time hole repair system based on WebGL includes a first depth point cloud texture rendering module, a second depth point cloud texture rendering module, a color point cloud rendering module, a hole judgment module, and an edge detection and intelligent interpolation filling module;

[0129] The first depth point cloud texture rendering module renders a first depth point cloud texture and encodes the depth value into an RGBA channel;

[0130] The second depth point cloud texture rendering module uses the average point spacing information generated by the octree structure to render the second depth point cloud texture with the enlarged point size, and encodes the depth value into the RGBA channel;

[0131] The color point cloud rendering module uses the RGB color value of the point to color the point cloud;

[0132] The hole judgment module compares the first depth point cloud texture and the second depth point cloud texture to determine whether the depth values ​​obtained by decoding the same pixel are consistent. If they are inconsistent, the region is determined to be a hole in the point cloud.

[0133] The hole judgment module is connected to the edge detection and intelligent interpolation filling modules.

[0134] In a disclosed embodiment of the present invention, a computer device includes a processor, wherein the processor is used to execute a computer program stored in a memory to implement the real-time hole repair method for point cloud rendering based on WebGL as described in any one of claims 1-6.

[0135] In a disclosed embodiment of the present invention, a computer-readable storage medium stores a computer program, and a processor is used to execute the computer program stored in the storage medium to implement the real-time hole repair method for point cloud rendering based on WebGL as described in any one of claims 1-6.

[0136] This paper proposes a real-time point cloud rendering and hole filling solution based on WebGL, aiming to solve the problems of low computational efficiency, high memory usage, complex color processing and long computational cycle of the existing Poisson sampling method on the browser side. By introducing a multi-layer depth texture comparison mechanism and an octree space segmentation strategy, this paper realizes efficient detection and filling of point cloud holes, while ensuring the smoothness and data integrity of real-time rendering on the Web side. The specific technical solution and principle are described as follows:

[0137] 1: Hole detection method based on multi-layer depth texture

[0138] The present invention achieves efficient detection of holes by rendering deep textures with two different point sizes. First, a deep point cloud texture is rendered according to the normal point size, and the depth value is encoded into the RGBA channel. Then, the average point spacing information generated by the octree structure is used to render a second deep point cloud texture with an enlarged point size. The enlarged depth texture covers more area, so that potential hole areas can be detected through subsequent texture comparison.

[0139] The comparison principle of two depth textures is based on pixel coverage. There will be holes in the depth texture before the point cloud is enlarged. When the depth information obtained by decoding the same pixel (RGBA) in the two textures before and after enlargement is inconsistent, the area is a hole; for example, when a pixel in the enlarged texture has a depth value, but the corresponding pixel in the normal texture has no depth value, it can be determined that this area is a hole in the point cloud. This detection method avoids complex spatial interpolation and sampling calculations, and realizes rapid identification of hole areas through simple texture comparison, effectively improving the efficiency of hole detection.

[0140] Depth information uses float32, which is 4 bytes long = 32 bits. Each RGBA channel is 8 bits, and a total of 4 channels can encode 32 bits of content. Choosing to use RGBA channels to encode high-precision depth textures instead of traditional single-channel grayscale depth textures brings significant technical advantages:

[0141] Higher depth accuracy

[0142] Using RGBA channels for depth encoding can significantly improve the accuracy of depth data. Usually, grayscale depth textures use a single 8-bit channel to represent depth information, which is not accurate enough in some scenarios, especially for point cloud data with large ranges or high precision requirements. RGBA channels provide 4 8-bit channels, decomposing and storing depth values ​​in each channel, so as to achieve 32-bit accuracy. This method can more accurately restore depth relationships in large scenes or multi-level structures and reduce errors caused by insufficient depth quantization.

[0143] Reducing depth approximation error

[0144] Grayscale depth textures are prone to depth approximation problems when representing near and far objects, especially in close-range point cloud rendering, where high depth changes may cause layer mismatch due to insufficient precision. The RGBA encoding method can split the depth value into higher-bit representations, thereby avoiding approximation errors between near and far levels and making the depth value of the edge of the object more continuous. In hole filling, this high precision can ensure more accurate depth judgment of the edge area and avoid misjudgment.

[0145] Improving the dynamic range of depth data

[0146] The method of separating depth values ​​by RGBA channels can expand the dynamic range of depth data, allowing depth values ​​to be distributed in a wider numerical space. For scenes with large height differences or wide viewing angles in point cloud data, RGBA depth encoding can record subtle depth changes without sacrificing accuracy, effectively maintaining the integrity and accuracy of the rendered scene. This enhanced dynamic range can significantly improve the overall visual effect and maintain the layered sense of scene depth when filling holes on the Web.

[0147] Facilitates simultaneous processing of color and depth

[0148] It is also feasible to use RGBA channels to encode depth information and other data (such as high-precision normals or contour information) in a single texture. This design can achieve synchronous data processing. For example, during the rendering process, the high-precision depth value of RGBA can not only be used to detect hole areas, but also some color or normal information can be directly integrated into the same texture. This structure reduces texture switching and computational overhead during the rendering process, and is particularly suitable for real-time rendering tasks on the browser side.

[0149] Compatibility and performance optimization

[0150] In modern WebGL environments, the encoding of RGBA channels has good compatibility. Most WebGL implementations provide hardware acceleration support for the RGBA format, which can efficiently render and process high-precision texture data without sacrificing performance. In addition, the RGBA encoding method is easy to align with the data format of other rendering pipelines in the depth filling algorithm, achieving unified format management, which helps to further optimize the utilization of memory and computing resources.

[0151] 2: Edge detection and intelligent interpolation filling

[0152] After the hole detection is completed, the fragment shader further performs edge judgment based on the surrounding depth values ​​to determine whether the hole area is located at the edge of the contour. If the hole area is outside the contour, the area is intelligently interpolated and filled. When filling, the present invention uses the color and depth values ​​of the surrounding adjacent points to generate new filling points through a simple interpolation algorithm, so that the supplementary point cloud is consistent with the original data in visual effect.

[0153] The core of intelligent interpolation filling is to use the color values ​​and depth data of adjacent points to generate reasonable supplementary points for the hole area through linear interpolation or weighted average. This method can not only effectively reduce the amount of calculation, but also ensure the consistency of color and depth, and avoid the complex color calculation in methods such as Poisson sampling. This lightweight interpolation method has extremely low memory usage and is particularly suitable for real-time rendering on the browser side.

[0154] 3: Octree structure optimization and data scheduling

[0155] The present invention uses an octree structure on the server side to organize point cloud data in layers to improve transmission and rendering efficiency. When constructing the octree, the average point spacing information of each node is recorded to provide a reference for the enlarged rendering of the depth texture. In addition, through the intelligent scheduling algorithm, the node data within the user's perspective is transmitted first, reducing the computing and memory burden of the Web side.

[0156] The octree structure provides hierarchical management and on-demand loading capabilities, making the transmission and rendering of point cloud data more efficient. Due to memory limitations and network bandwidth constraints on the Web side, the use of octree data scheduling can significantly reduce the amount of data and ensure the smoothness of the real-time rendering process. At the same time, by recording the average point spacing to generate an enlarged depth texture, it provides an accurate reference for hole detection and filling.

[0157] The edge detection and intelligent interpolation filling process is completed in the fragment shader. This design fully utilizes the parallel computing capabilities of the GPU to improve rendering efficiency and filling quality. GPU parallel computing has the following advantages

[0158] Efficient parallel processing capabilities

[0159] GPU has a large-scale parallel processing architecture and can handle the calculation tasks of a large number of fragments at the same time. During the edge detection and intelligent interpolation filling process, each fragment can be calculated independently without relying on the results of other fragments. This parallel computing capability makes the edge detection process fast and efficient, unaffected by the size of the data, and suitable for real-time rendering tasks of large-scale point cloud data.

[0160] Real-time edge detection

[0161] In the fragment shader, the edge detection algorithm can directly compare the depth value of each fragment with the depth values ​​of the surrounding adjacent fragments to determine whether the fragment is at the edge of the contour. Through parallel calculation, edge detection can be completed quickly in each rendering frame, ensuring that users get instant feedback during interactive operations. This method is much faster than the traditional CPU serial calculation method, greatly improving the user experience.

[0162] Efficient implementation of intelligent interpolation algorithm

[0163] After detecting the hole area, the intelligent interpolation filling algorithm interpolates the colors of adjacent fragments in the fragment shader. This process can use simple linear interpolation or weighted average algorithms to quickly calculate the color values ​​around each hole fragment. Due to the low latency of the GPU when processing parallel data, the interpolation process is not only fast, but also able to update the filling results in real time to maintain the smoothness of rendering.

[0164] Reduce CPU Load

[0165] Transferring edge detection and interpolation filling calculations to the GPU can significantly reduce the burden on the CPU, allowing it to focus on other logical operations and interactive responses. This design achieves more efficient resource utilization and optimizes the overall system performance.

[0166] Compared with the existing technologies, the real-time point cloud rendering and hole filling method created by the present invention has the following significant beneficial effects and advantages:

[0167] 1. High-precision depth information processing: This invention uses RGBA channels to encode depth textures. Compared with traditional grayscale depth textures, this method provides higher depth accuracy and larger dynamic range. This enables the depth value to more accurately represent the distance and shape of the object, reduces the impact of quantization errors, and effectively improves the depth discrimination ability in complex scenes.

[0168] 2. Real-time edge detection and intelligent interpolation: By implementing edge detection and intelligent interpolation filling in the fragment shader, the present invention fully utilizes the parallel computing capability of the GPU. Unlike the traditional method that relies on the CPU for complex calculations, this method can quickly process the edge information and hole filling of each fragment in each rendering frame, thereby realizing the possibility of real-time rendering and significantly improving the user experience.

[0169] 3. Optimization of resource utilization: Since complex computing tasks are transferred to the GPU for execution, the burden on the CPU is significantly reduced, allowing the system to use resources more efficiently. In terms of memory usage and computing cycles, the design of the method of the present invention enables efficient point cloud rendering and processing in resource-constrained environments (such as Web browsers), breaking through the limitations of traditional methods.

[0170] 4. Improve filling quality and consistency: The intelligent interpolation filling algorithm in the present invention can better consider the contour information of the surrounding environment during the filling process by combining the surrounding depth values ​​to interpolate the color value. Compared with the simple neighborhood interpolation in the existing technology, this method can fill the hole area more accurately, reduce the visual incoherence of the filling result, and improve the consistency and aesthetics of the overall rendering effect.

[0171] 5. Simplify the development and integration process: By implementing point cloud rendering and hole filling functions on the existing WebGL platform, the present invention reduces the complexity of developers migrating between different platforms. Web-based technologies can be more easily integrated into existing applications and online services.

[0172] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A real-time hole repair method for point cloud rendering based on WebGL, characterized in that: include: Render the first depth point cloud texture and encode the depth value into the RGBA channel; Using the average point spacing information generated by the octree structure, the second depth point cloud texture is rendered with the enlarged point size, and the depth value is encoded into the RGBA channel; Compare the first depth point cloud texture and the second depth point cloud texture to see if the decoded depth values ​​of the same pixel are consistent. If they are inconsistent, determine that this area is a hole in the point cloud; When holes exist, edge detection and intelligent interpolation filling are performed.

2. The real-time hole repair method for point cloud rendering based on WebGL according to claim 1, characterized in that: Before comparing the first depth point cloud texture and the second depth point cloud texture to determine whether the same pixel is decoded and the depth value is obtained consistently, the RGB color value of the point is used for coloring.

3. The real-time hole repair method for point cloud rendering based on WebGL according to claim 1, characterized in that: Before rendering the first depth point cloud texture, the point cloud data is hierarchically organized using an octree structure on the server side; When constructing the octree structure, the average point spacing information of each node is recorded.

4. The real-time hole repair method for point cloud rendering based on WebGL according to claim 3, characterized in that: Through intelligent scheduling algorithms, node data within the user's perspective is transmitted first.

5. The real-time hole repair method for point cloud rendering based on WebGL according to claim 1, characterized in that: The edge detection and intelligent interpolation filling are specifically as follows: The process of edge detection and intelligent interpolation filling is completed in the fragment shader; Compare the depth value of each fragment with the depth values ​​of the surrounding adjacent fragments to determine whether the hole area is outside the contour; If the hole area is outside the contour, the area is filled with intelligent interpolation.

6. The real-time hole repair method for point cloud rendering based on WebGL according to claim 5, characterized in that: The intelligent interpolation filling of the area is: using the color and depth values ​​of the surrounding adjacent points, and generating new filling points through an interpolation algorithm, so that the filling points are consistent with the surrounding adjacent points in visual effect.

7. The real-time hole repair method for point cloud rendering based on WebGL according to claim 5, characterized in that: The intelligent interpolation filling algorithm performs color interpolation on adjacent fragments.

8. A real-time hole repair system for point cloud rendering based on WebGL, characterized in that: It includes a first depth point cloud texture rendering module, a second depth point cloud texture rendering module, a color point cloud rendering module, a hole judgment module, and an edge detection and intelligent interpolation filling module; The first depth point cloud texture rendering module renders a first depth point cloud texture and encodes the depth value into an RGBA channel; The second depth point cloud texture rendering module uses the average point spacing information generated by the octree structure to render the second depth point cloud texture with the enlarged point size, and encodes the depth value into the RGBA channel; The color point cloud rendering module uses the RGB color value of the point to color the point cloud; The hole judgment module compares the first depth point cloud texture and the second depth point cloud texture to determine whether the depth values ​​obtained by decoding the same pixel are consistent. If they are inconsistent, the region is determined to be a hole in the point cloud. The hole judgment module is connected to the edge detection and intelligent interpolation filling modules.

9. A computer device, characterized in that: It includes a processor, which is used to execute a computer program stored in a memory to implement the real-time hole repair method for point cloud rendering based on WebGL as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The processor is used to execute a computer program stored in a storage medium to implement the real-time hole repair method for point cloud rendering based on WebGL as described in any one of claims 1 to 7.

Images