3D Gaussian sputtering block rendering method and system

Abstract

The invention relates to a 3D Gaussian sputtering block rendering method and system, electronic equipment and a storage medium, and the method comprises the steps: segmenting an overall point cloud into a plurality of different point cloud blocks, calculating a to-be-rendered point based on the point cloud blocks, and adaptively adjusting the hierarchy of the to-be-rendered point cloud blocks according to the hardware operation performance. Proper to-be-rendered points can be flexibly and conveniently selected, and repeated rendering is avoided; furthermore, blocks with proper sizes and proper grades are selected according to depths, so that the effects of clear near scene and blurred distant scene can be achieved, and the viewing quality of a rendered picture is maximized; moreover, due to the training of each point cloud block, the size, precision and quality of the point cloud of each block can be guaranteed, and the intersection of each point cloud block is smooth.

Description

Technical Field

[0001] The present application relates to the field of three-dimensional graphics rendering technology, and in particular to a 3D Gaussian sputtering block rendering method, system, computer device and computer-readable storage medium. Background Art

[0002] 3D Gaussian Splatting (3DGS) is a graphics technique used to represent and render three-dimensional scenes. The basic idea of ​​this method is to use Gaussian functions to represent points in three-dimensional space and their properties, such as color, transparency, etc. Gaussian splatting can be used in various applications, including computer graphics, computer vision, data visualization, etc.

[0003] Traditional 3D Gaussian point cloud rendering technology, as a point-based graphics rendering method, constructs a 3D scene by rendering all 3D points in the field of view. However, this method has at least the following problems when processing large-scale point cloud data:

[0004] 1. When the amount of point cloud data is huge, rendering a large number of points will lead to high performance requirements for rendering hardware, which is particularly prominent on devices with limited resources;

[0005] 2. For large areas that require high-precision rendering, existing technologies often have difficulty guaranteeing the quality of point cloud rendering, which affects the authenticity and detail of the final visual effect;

[0006] 3. The lack of flexibility in rendering strategies cannot adapt to the needs of devices with different performance, nor can it provide differentiated rendering clarity for different areas of the scene;

[0007] 4. When multiple Gaussian splatter point clouds are rendered together, there are usually noise points around each point cloud, which affects the rendering effect at the junction.

[0008] Currently, no effective solution has been proposed for the problem in related technologies that when the amount of 3D Gaussian point cloud data is huge, rendering a large number of points will require high performance of the rendering hardware. Summary of the invention

[0009] The embodiments of the present application provide a 3D Gaussian sputtering block rendering method, apparatus, system, computer equipment and computer-readable storage medium to at least solve the problem of low flexibility caused by the rendering method in the related art being restricted by the computing performance of the device end.

[0010] In a first aspect, an embodiment of the present application provides a 3D Gaussian sputtering block rendering method, the method comprising:

[0011] The initial 3D Gaussian point cloud is divided into a plurality of blocks of different levels, and block training is performed based on the blocks to obtain a plurality of point cloud blocks of different levels;

[0012] Adding tags to the 3D points in each point cloud block and storing them independently as a preset data structure, wherein the tags are used to distinguish the relationship between any 3D point and blocks at different levels;

[0013] Based on the 3D points of the preset data structure and the pixel depth of each 3D point under the experience perspective, point cloud blocks to be rendered are selected and sorted to obtain a set of blocks to be rendered;

[0014] Based on the computing performance of the hardware device, non-overlapping 3D points to be rendered are obtained from the set of blocks to be rendered to form a set of points to be rendered, and a three-dimensional rendering result is obtained according to the set of points to be rendered.

[0015] In some embodiments, based on the computing performance of the hardware device, in the set of blocks to be rendered, non-overlapping 3D points to be rendered are obtained to form a set of points to be rendered, and obtaining a three-dimensional Gaussian rendering result according to the set of points to be rendered includes:

[0016] In all point cloud blocks at the highest level, obtain the first 3D point in the current field of view, and add the first 3D point to the set of points to be rendered;

[0017] In all point cloud blocks of other levels except the highest level, select a second 3D point that does not overlap with the first 3D point and is located in the current field of view, and add the second 3D point to the set of points to be rendered;

[0018] Determine whether the number of 3D points in the point set to be rendered is less than or equal to a preset point threshold, and if so, perform rendering based on the point set to be rendered to obtain the three-dimensional rendering result,

[0019] If not, the point cloud block level in the to-be-rendered point set is adjusted to obtain an updated to-be-rendered point set, and rendering is performed based on the updated to-be-rendered point set to obtain the three-dimensional rendering result.

[0020] In some embodiments, in the set of blocks to be rendered, point cloud blocks with higher levels are sorted before point cloud blocks with lower levels;

[0021] The preset point threshold is determined based on the computing performance of the hardware device and empirical knowledge in the field, and is used to indicate the maximum value of the computing performance of the hardware device that can support efficient rendering of 3D points.

[0022] In some embodiments, adjusting the point cloud block level in the to-be-rendered point set to obtain an updated to-be-rendered point set includes:

[0023] In the set of blocks to be rendered, under the premise of keeping the point cloud block with the largest pixel ratio in the highest level unchanged,

[0024] According to the order from high to low levels, each high-level point cloud block is replaced with an adjacent low-level point cloud block with an overlapping area, and the repeated areas are merged to obtain the updated point set to be rendered.

[0025] In some embodiments, multiple sets of preset resolutions are determined based on the constraint that each level of point cloud blocks covers the entire point cloud data and the number of point cloud blocks at each level is minimized, and the initial 3D Gaussian point cloud is divided into multiple levels of blocks according to the preset resolutions.

[0026] In some of the embodiments, block training is performed based on the blocks to obtain a plurality of trained point cloud blocks;

[0027] The images corresponding to each of the blocks are obtained respectively, and the images of the extended range centered on the block are used to form a captured image;

[0028] Perform training based on the acquired image to obtain an initialized point cloud block corresponding to the block;

[0029] According to the point cloud segmentation strategy, in the trained point cloud block, the point cloud area corresponding to the outward expansion range is deleted to obtain the trained point cloud block.

[0030] In some embodiments, based on the pixel depth of each 3D point under the experience perspective, a point cloud block to be rendered is selected, and the set of blocks to be rendered includes:

[0031] Based on the scene depth map from the experience perspective, obtain the pixel depth of each 3D point;

[0032] Based on the pixel depth and the preset distant view distance threshold, point cloud blocks to be rendered that match each depth interval are selected and sorted, and point cloud blocks at the same level are sorted according to the number of pixels to obtain the set of blocks to be rendered.

[0033] In a second aspect, an embodiment of the present application provides a 3D Gaussian sputtering block rendering system, the system comprising: a segmentation module, a preprocessing module and a rendering module, wherein:

[0034] The segmentation module is used to segment the initial 3D Gaussian point cloud into multiple blocks of different levels, and perform block training based on the blocks to obtain multiple point cloud blocks of different levels;

[0035] The preprocessing module is used to add tags to the 3D points in each point cloud block and store them independently as a preset data structure, wherein the tags are used to distinguish the relationship between any 3D point and blocks at different levels, and based on the 3D points in the preset data structure and the pixel depth of each 3D point under the experience perspective, select and sort the point cloud blocks to be rendered to obtain a set of blocks to be rendered;

[0036] The rendering module is used to obtain non-overlapping 3D points to be rendered in the set of blocks to be rendered to form a set of points to be rendered based on the computing performance of the hardware device, and obtain a three-dimensional rendering result according to the set of points to be rendered.

[0037] In a third aspect, an embodiment of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the first aspect above when executing the computer program.

[0038] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect above.

[0039] Compared with the related art, the 3D Gaussian sputtering block rendering method provided in the embodiment of the present application divides the overall point cloud into multiple different point cloud blocks and trains them independently. Based on the point cloud blocks, by calculating the points to be rendered, and adaptively adjusting the level of the point cloud blocks to be rendered according to the hardware computing performance, it is possible to flexibly and conveniently select appropriate points to be rendered to avoid repeated rendering; further, by selecting blocks of appropriate size and level according to the depth, it is also possible to achieve the effect of clear near and blurred far views, thereby maximizing the viewing quality of the rendered image; and since each point cloud block is trained, the point cloud size, accuracy and quality of each block can also be guaranteed, and the junctions of each point cloud block are smooth. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0041] Figure 1 is a flow chart of a 3D Gaussian sputtering block rendering method according to an embodiment of the present application;

[0042] Figure 2 is a schematic diagram of a 3D Gaussian point projection result according to an embodiment of the present application;

[0043] Figure 3 is a schematic diagram of an actual projection result of a 3D Gaussian point cloud according to an embodiment of the present application;

[0044] Figure 4 Schematic diagram of the 3D Gaussian point cloud segmentation result according to an embodiment of the present application

[0045] Figure 5 is a structural block diagram of a 3D Gaussian sputtering block rendering system according to an embodiment of the present application;

[0046] Figure 6 It is a schematic diagram of the internal structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0048] Obviously, the drawings described below are only some examples or embodiments of the present application. For ordinary technicians in this field, the present application can also be applied to other similar scenarios based on these drawings without creative work. In addition, it can also be understood that although the efforts made in this development process may be complicated and lengthy, for ordinary technicians in this field related to the content disclosed in this application, some changes in design, manufacturing or production based on the technical content disclosed in this application are just conventional technical means, and should not be understood as insufficient content disclosed in this application.

[0049] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0050] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantitative limitation, and may represent the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0051] In the related technology, the traditional 3D Gaussian point cloud rendering technology, as a point-based graphics rendering method, constructs a 3D scene by rendering all points in the field of view. However, this method faces significant performance bottlenecks when processing large-scale point cloud data.

[0052] 1. When the amount of point cloud data is huge, rendering a large number of points requires the rendering hardware to consume a lot of computing performance, which is particularly prominent on devices with limited resources;

[0053] 2. For large areas that require high-precision rendering, existing technologies often have difficulty guaranteeing the quality of point cloud rendering, which affects the authenticity and detail of the final visual effect;

[0054] 3. The lack of flexibility in rendering strategies cannot adapt to the needs of devices with different performance, nor can it provide differentiated rendering clarity for different areas of the scene;

[0055] 4. When multiple Gaussian splatter point clouds are rendered together, there are usually noise points around each point cloud, which affects the rendering effect at the junction.

[0056] In view of the above problems, the present application provides a 3D Gaussian sputtering block rendering method. Figure 1 is a flowchart of a 3D Gaussian sputtering block rendering method according to an embodiment of the present application, such as Figure 1As shown, the process includes the following steps:

[0057] S101, dividing the initial 3D Gaussian point cloud into multiple blocks of different levels, and performing block training based on the blocks to obtain multiple point cloud blocks of different levels;

[0058] Among them, 3D Gaussian point cloud is different from ordinary point cloud. Each point is composed of an ellipsoid. It does not rely on traditional sensor acquisition equipment. It analyzes the description and model of the scene and uses the Gaussian function as the basis to "sputter" the density and distribution of points at the position that needs to be expressed, thereby generating point cloud data.

[0059] For large areas that require high-precision rendering, considering that the common rendering method in this field is to render all 3D one by one, this not only requires high performance of the hardware equipment, but also often makes it difficult to ensure the quality of 3D Gaussian point cloud rendering, thus affecting the authenticity and detail performance of the final visual effect;

[0060] In this embodiment, before rendering, the initial 3D Gaussian point cloud data is divided into a plurality of small blocks at different levels, and training is performed independently on each small block to ensure that the size and quality of each point cloud are constant, thereby improving the overall rendering effect.

[0061] The 3D Gaussian point cloud is segmented according to a preset resolution. The preset resolution has multiple parameters for dividing the initial 3D Gaussian point cloud into multiple levels, and the size and number of blocks in different levels are different, while the size of blocks in the same level is the same. It should be noted that the preset resolution can be flexibly specified according to the overall point cloud size and scene requirements, so as to achieve as few and effective block divisions as possible.

[0062] (1) Before point cloud segmentation, the following preprocessing steps need to be performed to achieve data preparation and rule definition, including:

[0063] Step 1, set the Z axis of the 3D Gaussian point cloud to the gravity direction upward, and the subsequent point cloud blocks are all performed in the plane direction (X / Y axis); point clouds in other directions can be transferred to this coordinate system, or blocked in their respective coordinate systems by analogy.

[0064] Step 2, define the calculation method of the 3D Gaussian point cloud bounding box:

[0065] Since 3dgs point cloud is different from ordinary point cloud, each point of which is composed of an ellipsoid, the boundingbox of the point cloud defined in this embodiment is the outer contour of the ellipse projected on the X / Y axis plane by the ellipsoid equation of all 3d Gaussian points;

[0066] Figure 2 is a schematic diagram of a 3D Gaussian point projection result according to an embodiment of the present application, such as Figure 2 As shown, the projection of ellipsoid A on plane XY is ellipse Axy. Further, Figure 3 is a schematic diagram of the actual projection result of the 3D Gaussian point cloud according to an embodiment of the present application. The projection of the 3D Gaussian point cloud on the xy plane is as follows Figure 3 As shown, the red box represents its bounding box;

[0067] Through the above method, the spatial distribution characteristics of the 3DGS point cloud can be effectively described, and an accurate bounding box is provided, which is suitable for various application scenarios that require accurate boundary description.

[0068] (2) Further, dividing the 3D Gaussian point cloud into multiple blocks of different levels includes the following steps:

[0069] Step 1, first, determine the preset resolution for segmentation;

[0070] The preset resolution in step S101 includes multiple data, which are used to divide the 3D Gaussian point cloud into blocks of different levels and sizes; specifically, the preset resolution can be determined by using an arithmetic progression, a geometric progression, a factorial, etc.;

[0071] In an exemplary embodiment, Figure 4 is a schematic diagram of the 3D Gaussian point cloud segmentation result according to an embodiment of the present application, such as Figure 4 As shown, if a geometric progression is used to set the preset resolution for segmentation, each block at the smallest level (level 0) can be set to 2X2 meters; the block at the next higher level (level 1) is 4X4 meters, and the block at the next higher level is 8X8 meters, and finally, the highest level is the range that can completely include the point cloud; for example, a 3D Gaussian point cloud may have a bounding box size of 40X30 meters, and its highest level is 64X64 meters.

[0072] It should also be noted that the process of setting the preset resolution requires that the blocks at each level need to satisfy that the edge of the outermost block is greater than or equal to the edge of the point cloud bounding box to ensure that the point cloud is completely included; further, it is necessary to find a suitable block center position to keep the number of blocks at each level to a minimum while satisfying that the edge of the outermost block is greater than or equal to the edge of the point cloud bounding box. In this embodiment, the block center position can be calculated by dynamic programming.

[0073] It should be understood that, under the constraints of the above rules, technicians in this field can flexibly specify the preset resolution and block center position. Through the above segmentation method, setting the block edge of each level must be greater than or equal to the bounding box edge of the point cloud can ensure that each block can completely contain the point cloud data to avoid data loss or incompleteness; in addition, inside the bounding box, selecting a center position that can minimize the number of blocks can also ensure that the size of each layer of blocks is appropriate, while minimizing the number of blocks, ensuring that the edge of each block is greater than the edge of the bounding box.

[0074] (3) Independently train each block at different levels obtained by segmentation;

[0075] The training process can include feature extraction of point clouds, Gaussian distribution fitting of points, noise removal, etc. The purpose of point cloud training is to improve the quality and consistency of point cloud data;

[0076] In this embodiment, it is considered that there are usually noise points around each point cloud, which affects the rendering effect at the junction; in this embodiment, when training each block, the image of the point cloud area itself and the image in its extended area are trained, aiming to improve the quality and consistency of the point cloud data and reduce the block boundary effect, thereby achieving higher realism and detail expression.

[0077] In an exemplary embodiment, for a point cloud block of n×n meters, training can be performed on the collected images of the (n+2m)*(n+2m) meter area near the center point of the block. After the training is completed, the point cloud block strategy is used to crop the redundant part (n+m)*m*4, leaving a point cloud of nXn;

[0078] It can be understood that such processing can not only ensure the size, accuracy and quality of each block point cloud, but also, because the training process uses the images in the point cloud area itself and its extended area for training, it can obtain more contextual information, thereby improving the shape consistency of each point cloud data, smoothing the junctions of blocks, and reducing block boundary effects.

[0079] In addition, after the segmentation and training are completed, the number of 3D Gaussian points in each point cloud block is counted. If the number of 3D points in a point cloud block exceeds a preset point threshold, the 3D number of the point cloud block is compressed so that the number of 3D points in the point cloud block does not exceed the preset point threshold; wherein the preset point threshold can be determined according to the terminal type, for example, on the web side, the preset point threshold can be set to 1000000, and on the server, the preset point threshold can be set to 50000000; in addition, the specific method of compressing the 3D number can be but is not limited to being implemented based on pruning algorithms such as random sampling, farthest point sampling, and voxel grid downsampling.

[0080] Through the above step S101, the original 3D Gaussian point cloud is divided into multiple blocks belonging to different levels, and each block and its outer expansion range image are used to independently train each block, which can ensure the accuracy and quality of each point cloud and achieve smoothness at the junction of the point clouds.

[0081] S102, adding a mark to the 3D points in each point cloud block, and independently storing it as a preset data structure, wherein the mark is used to distinguish the relationship between any 3D point and blocks at different levels;

[0082] In this embodiment, the function of this step is to prepare the data structure, obtain the relationship between each 3D point and each level of the block, and the overlap relationship between the point cloud blocks at different levels. Specifically, in an exemplary embodiment, the data definition stage of the 3D Gaussian point includes the following steps:

[0083] Step 1, set the representation of each block as D(1, i), where 1 is the level to which it belongs, i is the sequence number of the block in all blocks, set the boundingbox(D(l, i)) of the block to represent the range of the block, and further define any 3D point P in the point cloud. If P∈boundingbox(D(l, i)), it means that P is in the area of ​​the point cloud block D(1, i);

[0084] Step 2, add a mark to each 3D point and store the 3D point as a preset data structure.

[0085] Specifically, traverse each block D(m, j) in level0, level1, ...., level n-1. If That is, if blocks D(n, i) and D(m, j) have overlapping areas, then traverse the point P in block D(n, i). If the coordinates of P are within the boundingbox(D(l, i)), then P is considered to fall within block D(1, i);

[0086] The above steps are equivalent to recording the coordinates of each point P in a point cloud block D(n, i) at level n, the block number where it falls in level0, level1, ..., level n-1, and adding the block number of the point at different levels to the set S_render(P, Q);

[0087] After the above steps, any 3D point in the original 3D Gaussian point cloud corresponds to one of the above-mentioned sets S_render; in an exemplary embodiment: S_render(Pi, D(3,1)) = {D(0,i), D(1,j), D(2,k)}, indicating that the point Pi in the point cloud block D(3,1) falls within the point cloud blocks D(0,i), D(1,j), D(2,k) at the same time; it should be noted that since the two point cloud blocks at the same level do not overlap, each level in S_render must contain only one D.

[0088] Through the above step S102, the relationship between each 3D Gaussian point and each block at different levels, as well as the overlap relationship between point cloud blocks at different levels are clearly established, thereby achieving data preparation before rendering.

[0089] S103, based on the 3D points of the preset data structure and the pixel depth of each 3D point under the experience perspective, selecting and sorting point cloud blocks to be rendered to obtain a set of blocks to be rendered;

[0090] In this embodiment, blocks of appropriate levels are selected according to the depth under the current experience perspective. Point cloud blocks with higher division levels are used in areas with smaller depths, and point cloud blocks with lower division levels are used in areas with larger depths. Based on this strategy, a set of blocks to be rendered is formed. Since point cloud blocks in the near distance are smaller and have higher 3D point density, a relatively clearer effect can be obtained. Point cloud blocks in the far distance are rendered based on larger blocks, which can obtain a relatively blurred effect. Therefore, under the premise of ensuring user experience, rendering resources are more applied to a relatively close range where users are more sensitive, thereby maximizing the viewing quality of subsequent rendered images as a whole.

[0091] Specifically, the step includes:

[0092] Step 1, obtain the depth map under the current user experience perspective. Optionally, this step can be implemented using a pre-made scene model; or, the depth map can be obtained based on the 3D Gaussian point cloud through projection, interpolation, etc.; since the specific implementation method is a conventional and general means in this field, it will not be repeated in this embodiment.

[0093] Step 2: Based on the pixel depth and the preset distant view distance threshold, select and sort the point cloud blocks to be rendered that match each depth interval, and sort the point cloud blocks at the same level according to the number of pixels to obtain a set of blocks to be rendered;

[0094] In an exemplary embodiment, three levels of distant view distance thresholds (d1, d2, d3) are set from near to far;

[0095] a. For 3D points d1 that are smaller than the distant view distance threshold, query the level 0 patch numbers corresponding to the 3D points one by one to determine how many level 0 patches there are in total, and sort the level 0 patches according to the number of pixels contained in the level 0 patches that appear in the current field of view, where the level 0 patches with more pixels are ranked first;

[0096] b. For pixel depths greater than d1 but less than d2, query the level 1 patch number corresponding to the 3D point one by one. Check how many level 1 patches there are in total, and sort the level 1 patches according to the number of pixels in the patch that appear in the current field of view, with the patches with more pixels being ranked first;

[0097] c. For pixel depths greater than d2 but less than d3, query the level 2 patch number corresponding to the 3D point one by one. Check how many level 2 patches there are in total, and sort the level 3 patches according to the number of pixels in the patch that appear in the current field of view, with the patches with more pixels at the front.

[0098] d, for pixels with depth greater than d3, render the area with the highest level;

[0099] The final set of blocks to be rendered S1 = {D(0, i1), D(0, i2), .... D(1, j1), ..., D(3, k1)....};

[0100] It should be noted that the above-mentioned method of setting three thresholds and selecting the point cloud blocks to be rendered based on the thresholds is only a specific example; it should be understood that in actual applications, under the technical framework provided in the application proposal for selecting smaller point clouds in the near distance and larger point clouds in the far distance according to the scene depth, the specific classification of the thresholds and the selection of specific parameters can also be more flexible and diverse.

[0101] Through the above step S103, blocks of appropriate corresponding sizes are selected at different depths according to the depth values ​​of the pixels. Since the density of feature points near is higher and the density of feature points far is lower, a rendering effect of clear near and blurred far can be achieved in the subsequent rendering link, thereby maximizing the rendering effect as a whole.

[0102] S104, based on the computing performance of the hardware device, in the set of blocks to be rendered, non-overlapping 3D points to be rendered are obtained to form a set of points to be rendered, and a three-dimensional rendering result is obtained according to the set of points to be rendered.

[0103] In this embodiment, the role of this step is to select rendering points that are non-repetitive and match the scene depth, and to determine whether the number of 3D points to be rendered matches the computing performance of the current device. If not, the established block level adjustment strategy is used to select subsequent low-level point cloud blocks to form a set of points to be rendered, and based on the updated set of points to be rendered, continue to render point cloud blocks with a smaller 3D point density, thereby achieving a balance between rendering quality and rendering efficiency, while controlling the scale of the overall point cloud, reducing dependence on hardware device performance, and achieving the technical effect of flexibly selecting rendering strategies based on the computing power of the device.

[0104] Specifically, step S104 specifically includes:

[0105] (1) Construction of the point set to be rendered

[0106] Step 1. Initialize the point set Q to be rendered to be empty, that is, initialize an empty set Q to store the points to be rendered;

[0107] Step 2, traverse step S103 to obtain a rendering block set S1 = {D(0, i1), D(0, i2), .... D(1, j1), ..., D(3, k1)....}, where D(0, i1) represents the i1th block at level 0, D(1, j1) represents the j1th block at level 1, and so on, until the last element;

[0108] Step 3. If the block to be rendered is a point cloud block D(0, i) of level 0 (i.e., the highest level), put all the points in the block into the point set Q. That is, for a block of level 0, directly add all its points to the point set Q.

[0109] Step 4. If the block to be rendered is level 1 or above, that is, D(1, i), 1 ≥ 1, for blocks with a level greater than 0 (i.e., blocks of a higher level), follow the steps below to process:

[0110] S4.1, define S2 as the set of all blocks in S1 with level>0, that is, the block set consisting of all point cloud blocks in other levels except the first level;

[0111] S4.2, traverse all 3D points Pj in S2, if Pj is in the current field of view, then the point is not rendered and skipped;

[0112] S4.3, if S_render(P)∩S2, it means that the point P exists in multiple point cloud blocks at different levels, and the point P must be added to the set Q in the above Step 3. Therefore, to avoid repeated rendering, in step S4.3, the point P is also configured to be skipped. Otherwise, it means that the point P is not added to the set Q, and the index xj of the point is put into the set Q;

[0113] Step 5: The points in the point set Q are the points to be rendered.

[0114] (2) Adaptive adjustment and update stage of the point set to be rendered

[0115] Considering that the computing performance of different hardware devices varies greatly, in this embodiment, in the rendering stage, the block-level adjustment strategy provided in the following steps Step 1 to Step 3 is also used to achieve the effect of flexibly determining the number of rendering points according to the hardware computing power. Specifically:

[0116] After step 3, the total number of 3D points of the point set Q to be rendered is determined, and it is judged whether the total number is less than or equal to the preset point threshold. If so, it means that the computing power of the current hardware device can support efficient rendering while ensuring the rendering quality. Otherwise, it means that the computing power of the current hardware device cannot support it, and the point set Q to be rendered needs to be adjusted.

[0117] It should be noted that the preset point threshold is determined by a person skilled in the art based on the hardware performance of the device and certain empirical knowledge. Since the core point of this technical means is to adjust the appropriate total number of rendering points according to the hardware computing power, how to determine the specific value of the preset point threshold has no effect on the core invention of this application. Therefore, it will not be repeated in this embodiment.

[0118] Specifically, adjusting the point set Q to be rendered includes the following steps:

[0119] Step 1, based on the order of the set of blocks to be rendered obtained in step S103, D(1, i) is replaced with D(1+1, j) having the overlapping area from high to low level;

[0120] Step 2: If a region is included multiple times, merge it to avoid repeated calculations.

[0121] Step 3. Based on the updated point set Q to be rendered, recalculate the points to be rendered through the above steps. If one round of adjustment and update still cannot meet the requirements, it is necessary to iterate and update multiple times until the number of 3D points in the set Q is less than or equal to the above preset point threshold; at this time, the point set Q is matched with the hardware computing power and can actually be used for rendering.

[0122] In addition, it should be noted that in the point set Q, for the point cloud blocks of Level 0, for the area with the largest pixel ratio, because this area is the most important and has the highest pixel ratio, the alignment is not replaced in this implementation step to ensure the rendering quality.

[0123] Through the above steps S101 to S104, compared with the method of directly rendering all 3D based on the overall Gaussian point cloud in the related art; the method provided in this embodiment, by dividing the overall point cloud into multiple different point cloud blocks and performing independent training, further, based on each point cloud block, by calculating the points to be rendered and adaptively adjusting the level of the point cloud blocks to be rendered according to the hardware computing performance, it is possible to flexibly and conveniently select the most appropriate points to be rendered according to the computing performance of the device side to avoid repeated rendering; further, blocks of appropriate size and level can be selected according to the depth of each object in the experience scene to achieve a clear effect in the near distance and blurred in the distant view, thereby maximizing the viewing quality of the rendered picture; and since each point cloud is trained in blocks, the point cloud size, accuracy and quality of each block can also be guaranteed, and the intersections of each point cloud block are smooth.

[0124] This embodiment also provides a 3D Gaussian sputtering block rendering system, which is used to implement the above embodiments and preferred implementations, and will not be repeated here. As used below, the terms "module", "unit", "subunit", etc. can implement a combination of software and / or hardware for a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0125] Figure 5 is a structural block diagram of a 3D Gaussian sputtering block rendering system according to an embodiment of the present application, such as Figure 5 As shown, the system includes: a segmentation module 50, a pre-processing module 51 and a rendering module 52, wherein:

[0126] The segmentation module 50 is used to segment the initial 3D Gaussian point cloud into a plurality of blocks of different levels, and perform block training based on the blocks to obtain a plurality of point cloud blocks of different levels;

[0127] The preprocessing module 51 is used to add tags to the 3D points in each point cloud block and store them independently as a preset data structure, wherein the tags are used to distinguish the relationship between any 3D point and blocks at different levels; and, based on the 3D points in the preset data structure and the pixel depth of each 3D point under the experience perspective, select and sort the point cloud blocks to be rendered to obtain a set of blocks to be rendered;

[0128] The rendering module 52 is used to obtain non-overlapping 3D points to be rendered in the set of blocks to be rendered to form a set of points to be rendered based on the computing performance of the hardware device, and obtain a three-dimensional rendering result according to the set of points to be rendered.

[0129] Through the above system, compared with the method of rendering all 3D directly based on the overall Gaussian point cloud in the related art, the system provided by this embodiment, by dividing the overall point cloud into multiple different point cloud blocks and performing independent training, further, based on each point cloud block, by calculating the points to be rendered and adaptively adjusting the level of the point cloud blocks to be rendered according to the hardware computing performance, it is possible to flexibly and conveniently select the most appropriate points to be rendered according to the computing performance of the device side to avoid repeated rendering; further, blocks of appropriate size and level can be selected according to the depth of each object in the experience scene to achieve a clear effect in the near distance and blurred effect in the distant view, thereby maximizing the viewing quality of the rendered picture.

[0130] In one embodiment, Figure 6 is a schematic diagram of the internal structure of an electronic device according to an embodiment of the present application, such as Figure 6 As shown, an electronic device is provided, which may be a server, and its internal structure diagram may be as shown in Figure 6 As shown. The electronic device includes a processor, a network interface, an internal memory and a non-volatile memory connected through an internal bus, wherein the non-volatile memory stores an operating system, a computer program and a database. The processor is used to provide computing and control capabilities, the network interface is used to communicate with an external terminal through a network connection, the internal memory is used to provide an environment for the operation of the operating system and the computer program, the computer program is executed by the processor to implement a 3D Gaussian sputtering block method, and the database is used to store data.

[0131] Those skilled in the art will understand that Figure 6 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0132] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0133] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A 3D Gaussian sputtering block rendering method, characterized in that: The method comprises: The initial 3D Gaussian point cloud is divided into a plurality of blocks of different levels, and block training is performed based on the blocks to obtain a plurality of point cloud blocks of different levels; Adding tags to the 3D points in each point cloud block and storing them independently as a preset data structure, wherein the tags are used to distinguish the relationship between any 3D point and blocks at different levels; Based on the 3D points of the preset data structure and the pixel depth of each 3D point under the experience perspective, point cloud blocks to be rendered are selected and sorted to obtain a set of blocks to be rendered; Based on the computing performance of the hardware device, non-overlapping 3D points to be rendered are obtained from the set of blocks to be rendered to form a set of points to be rendered, and a three-dimensional rendering result is obtained according to the set of points to be rendered.

2. The method according to claim 1, characterized in that Based on the computing performance of the hardware device, in the set of blocks to be rendered, obtaining non-overlapping 3D points to be rendered to form a set of points to be rendered includes: In all point cloud blocks at the highest level, obtain the first 3D point in the current field of view, and add the first 3D point to the set of points to be rendered; In all point cloud blocks of other levels except the highest level, select a second 3D point that does not overlap with the first 3D point and is located in the current field of view, and add the second 3D point to the set of points to be rendered; Determine whether the number of 3D points in the point set to be rendered is less than or equal to a preset point threshold, and if so, perform rendering based on the point set to be rendered to obtain the three-dimensional rendering result, If not, adjust the point cloud block level in the to-be-rendered point set to obtain an updated to-be-rendered point set.

3. The method according to claim 2, characterized in that In the set of blocks to be rendered, point cloud blocks with higher levels are sorted before point cloud blocks with lower levels; The preset point threshold is determined based on the computing performance of the hardware device and empirical knowledge in the field, and is used to indicate the maximum value of the computing performance of the hardware device that can support efficient rendering of 3D points.

4. The method according to claim 3, characterized in that The point cloud block level in the to-be-rendered point set is adjusted, and the updated to-be-rendered point set is obtained, including: In the set of blocks to be rendered, under the premise of keeping the point cloud block with the largest pixel ratio in the highest level unchanged, According to the order from high to low levels, each high-level point cloud block is replaced with an adjacent low-level point cloud block with an overlapping area, and the repeated areas are merged to obtain the updated point set to be rendered.

5. The method according to claim 1, characterized in that Multiple sets of preset resolutions are determined based on the constraint that the point cloud blocks at each level cover the entire point cloud data and the number of point cloud blocks at each level is minimized, and the initial 3D Gaussian point cloud is divided into blocks of multiple levels according to the preset resolutions.

6. The method according to claim 5, characterized in that Perform block training based on the blocks to obtain multiple trained point cloud blocks; The images corresponding to each of the blocks are obtained respectively, and the images of the extended range centered on the block are used to form a captured image; Perform training based on the acquired image to obtain an initialized point cloud block corresponding to the block; According to the point cloud segmentation strategy, in the trained point cloud block, the point cloud area corresponding to the outer expansion range is deleted to obtain the trained point cloud block.

7. The method according to claim 1, characterized in that According to the pixel depth of each 3D point in the experience perspective, select the point cloud block to be rendered, and the set of blocks to be rendered includes: Based on the scene depth map from the experience perspective, obtain the pixel depth of each 3D point; Based on the pixel depth and the preset distant view distance threshold, point cloud blocks to be rendered that match each depth interval are selected and sorted, and point cloud blocks at the same level are sorted according to the number of pixels to obtain the set of blocks to be rendered.

8. A 3D Gaussian sputtering block rendering system, characterized in that: The system comprises: a segmentation module, a preprocessing module and a rendering module, wherein: The segmentation module is used to segment the initial 3D Gaussian point cloud into a plurality of blocks of different levels, and perform block training based on the blocks to obtain a plurality of point cloud blocks of different levels; The preprocessing module is used to add tags to the 3D points in each point cloud block and store them independently as a preset data structure, wherein the tags are used to distinguish the relationship between any 3D point and blocks at different levels; Based on the 3D points of the preset data structure and the pixel depth of each 3D point under the experience perspective, point cloud blocks to be rendered are selected and sorted to obtain a set of blocks to be rendered; The rendering module is used to obtain non-overlapping 3D points to be rendered in the set of blocks to be rendered to form a set of points to be rendered based on the computing performance of the hardware device, and obtain a three-dimensional rendering result according to the set of points to be rendered.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images