Image rendering method, device and equipment based on three-dimensional Gaussian model, medium and product

Abstract

The invention discloses an image rendering method and device based on a three-dimensional Gaussian model, equipment, a medium and a product, and relates to the technical field of image processing, and the method comprises the steps: receiving a to-be-rendered image; and performing image rendering on the to-be-rendered image through the optimized three-dimensional Gaussian model to obtain a rendered image, the three-dimensional Gaussian model being constructed through a three-dimensional Gaussian backbone network, a resolution scheduler and a clone scheduler. Therefore, model optimization is carried out on the three-dimensional Gaussian model through the three-dimensional Gaussian backbone network, the resolution scheduler and the clone trainer, image rendering is carried out through the optimized three-dimensional Gaussian model after the to-be-rendered image is received, and the rendered image is obtained. The problem that in the prior art, a three-dimensional Gaussian acceleration technology cannot give consideration to the training speed, the rendering quality and the universality is solved, and the efficiency of image rendering based on the three-dimensional Gaussian model is improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image rendering method, apparatus, device, and product based on a three-dimensional Gaussian model. Background Technology

[0002] Existing 3D Gaussian (3DGS) acceleration techniques are mainly divided into two categories. One category is Gaussian primitive pruning and compaction optimization, represented by methods such as Mini-Splatting and Taming-3DGS. This reduces computational cost by selecting key Gaussian primitives and limiting the total number of primitives. For example, Mini-Splatting can compress Gaussian primitives to 0.492M. The other category is training process scheduling optimization, represented by methods such as DashGaussian and Flash3DGS. This improves efficiency by optimizing resolution adjustment and resource allocation. DashGaussian, for instance, shortens training time to 5.62 minutes. In recent years, wavelet-based methods such as Wavelet-GS have attempted to optimize 3DGS by utilizing the frequency-space joint capture capability of discrete wavelet transform. Some methods, such as DashGaussian, have also attempted to guide scheduling through decomposition.

[0003] However, existing technologies have obvious shortcomings. First, Gaussian pruning and densification optimization suffer from fuzzy pruning criteria and blind densification targets, which can easily lead to a decrease in rendering quality (e.g., Mini-Splatting's LPIPS is 0.2271) or computational redundancy. The scheduling optimization during the training process only focuses on frequency information and ignores spatial locality, resulting in insufficient scheduling targeting and significant Gaussian redundancy (e.g., DashGaussian is 2.053M). Wavelet-based methods only stay at the component partitioning level and are not deeply integrated with training scheduling and Gaussian management, making it impossible to solve the core decision-making problems of "when to adjust the resolution" and "where to enhance Gaussian". Overall, it is difficult to simultaneously balance training speed, rendering quality, and versatility.

[0004] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] The main purpose of this application is to provide an image rendering method, apparatus, device, medium and product based on a three-dimensional Gaussian model, which aims to solve the technical problem that the existing three-dimensional Gaussian acceleration technology cannot balance training speed, rendering quality and versatility.

[0006] To achieve the above objectives, this application proposes an image rendering method based on a three-dimensional Gaussian model, the image rendering method based on a three-dimensional Gaussian model comprising: Receive the image to be rendered; The image to be rendered is rendered using the optimized 3D Gaussian model to obtain the rendered image. The 3D Gaussian model is constructed using a 3D Gaussian backbone network, a resolution scheduler, and a clone scheduler.

[0007] In one embodiment, before the step of rendering the image to be rendered using an optimized three-dimensional Gaussian model to obtain a rendered image, the method further includes: Receive the training image set; Discrete wavelet transform is performed on several images in the training image set to obtain low-frequency approximation coefficients and high-frequency detail coefficients; Based on the low-frequency approximation coefficients and high-frequency detail coefficients, the dynamic resolution parameters are calculated by the resolution scheduler. A high-frequency importance map is obtained by calculating the high-frequency detail coefficients on the several images; Based on the high-frequency importance graph, the Gaussian element set in the three-dimensional Gaussian backbone network is optimized by the clone scheduler to obtain the optimized Gaussian element set. The optimized 3D Gaussian model is obtained based on the dynamic resolution parameters and the optimized Gaussian metaset.

[0008] In one embodiment, the step of calculating the dynamic resolution parameters by the resolution scheduler based on the low-frequency approximation coefficients and the high-frequency detail coefficients includes: The wavelet saliency of the several images is calculated using the low-frequency approximation coefficients and the high-frequency detail coefficients to obtain a single-image wavelet saliency map; The saliency average of the training image set is calculated using the low-frequency approximation coefficients and the high-frequency detail coefficients to obtain a scene saliency map; The standard deviation and mean of the scene saliency map are calculated to obtain the scene complexity coefficient. The dynamic resolution parameters are calculated using the scene saliency map and the scene complexity coefficient.

[0009] In one embodiment, the step of calculating the dynamic resolution parameters using the scene saliency map and scene complexity coefficients includes: The resolution update interval is determined by the scene complexity coefficient, and the resolution level is calculated by the densification period and the resolution update interval. The maximum resolution scale is determined using the scene saliency map. The dynamic resolution parameters are calculated based on the resolution update interval, resolution level, and maximum resolution scale.

[0010] In one embodiment, the step of optimizing the Gaussian element set in the three-dimensional Gaussian backbone network using the clone scheduler based on the high-frequency importance graph to obtain the optimized Gaussian element set includes: When the number of iterations of the dynamic resolution parameter reaches the cloning threshold, the scores of several Gaussian elements in the Gaussian element set are calculated using the high-frequency importance graph to obtain the cloning score. Based on the cloning score, a Gaussian unit to be cloned is selected from the plurality of Gaussian units by means of a sorting threshold; After cloning the Gaussian unit to be cloned, it is added to the Gaussian unit set, and the Gaussian unit set is pruned to obtain an optimized Gaussian unit set.

[0011] In one embodiment, the step of calculating the clonal scores by performing score calculations on a number of Gaussian elements in the Gaussian element set using the high-frequency importance graph includes: The Gaussian primitives are projected onto the training image set to obtain the two-dimensional coordinates of the Gaussian primitives; Based on the high-frequency importance map, bilinear interpolation sampling is performed on the two-dimensional coordinates to obtain the global view scores of the several Gaussian elements; The high-frequency importance of the several high-level elements is determined based on the global view score; The clone score of the Gaussian primitives is calculated by using the rendering weights and high-frequency importance of the Gaussian primitives.

[0012] Furthermore, to achieve the above objectives, this application also proposes an image rendering apparatus based on a three-dimensional Gaussian model, the image rendering apparatus based on a three-dimensional Gaussian model comprising: The receiving module is used to receive the image to be rendered; The rendering module is used to render the image to be rendered using an optimized 3D Gaussian model to obtain a rendered image. The 3D Gaussian model is constructed using a 3D Gaussian backbone network, a resolution scheduler, and a clone scheduler.

[0013] Furthermore, to achieve the above objectives, this application also proposes an image rendering device based on a three-dimensional Gaussian model, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the image rendering method based on a three-dimensional Gaussian model as described above.

[0014] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the image rendering method based on the three-dimensional Gaussian model as described above.

[0015] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the image rendering method based on a three-dimensional Gaussian model as described above.

[0016] One or more technical solutions proposed in this application have at least the following technical effects: This application proposes an image rendering method, apparatus, device, medium, and product based on a 3D Gaussian model. The method involves receiving an image to be rendered; rendering the image using an optimized 3D Gaussian model to obtain a rendered image. The 3D Gaussian model is constructed using a 3D Gaussian backbone network, a resolution scheduler, and a clone scheduler. Therefore, by optimizing the 3D Gaussian model using the 3D Gaussian backbone network, resolution scheduler, and clone trainer, and then rendering the received image using the optimized 3D Gaussian model, the method solves the problem in existing 3D Gaussian acceleration techniques that cannot simultaneously balance training speed, rendering quality, and versatility, thus improving the efficiency of image rendering based on a 3D Gaussian model. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating an embodiment of the image rendering method based on a 3D Gaussian model provided in this application. Figure 2 This is a flowchart illustrating Embodiment 2 of the image rendering method based on a 3D Gaussian model provided in this application; Figure 3 A simplified flowchart illustrating the image rendering method based on a three-dimensional Gaussian model provided in Embodiment 2 of this application; Figure 4This is a schematic diagram of the module structure of the image rendering device based on a three-dimensional Gaussian model according to an embodiment of this application; Figure 5 This is a schematic diagram of the hardware operating environment involved in the image rendering method based on the three-dimensional Gaussian model in the embodiments of this application.

[0020] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0021] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0022] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0023] The main solution of this application embodiment is as follows: receiving a training image set; performing discrete wavelet transform on several images in the training image set to obtain low-frequency approximation coefficients and high-frequency detail coefficients; calculating dynamic resolution parameters using the resolution scheduler based on the low-frequency approximation coefficients and high-frequency detail coefficients; calculating a high-frequency importance map for the several images using the high-frequency detail coefficients; optimizing the Gaussian primitive set in the 3D Gaussian backbone network using the cloning scheduler based on the high-frequency importance map to obtain an optimized Gaussian primitive set; obtaining the optimized 3D Gaussian model based on the dynamic resolution parameters and the optimized Gaussian primitive set. The image wavelet saliency of the several images is calculated using the low-frequency approximation coefficients and high-frequency detail coefficients to obtain a single-image wavelet saliency map; the average saliency of the training image set is calculated using the low-frequency approximation coefficients and high-frequency detail coefficients to obtain a scene saliency map; the standard deviation and mean of the scene saliency map are calculated to obtain a scene complexity coefficient; and the dynamic resolution parameters are calculated using the scene saliency map and the scene complexity coefficient. The resolution update interval is determined by the scene complexity coefficient, and the resolution level is calculated using the densification period and the resolution update interval. The maximum resolution scale is determined by the scene saliency map. Dynamic resolution parameters are calculated based on the resolution update interval, resolution level, and maximum resolution scale. When the number of iterations of the dynamic resolution parameters reaches the cloning threshold, the scores of several Gaussian elements in the Gaussian element set are calculated using the high-frequency importance map to obtain cloning scores. Based on the cloning scores, Gaussian elements to be cloned are selected from the several Gaussian elements using a sorting threshold. The Gaussian elements to be cloned are cloned and added to the Gaussian element set, and the Gaussian element set is pruned to obtain an optimized Gaussian element set. The Gaussian primitives are projected onto the training image set to obtain their two-dimensional coordinates. Based on the high-frequency importance map, bilinear interpolation is performed on these two-dimensional coordinates to obtain the global view score of each Gaussian primitive. The high-frequency importance of each Gaussian primitive is determined based on its global view score. The clone score of each Gaussian primitive is calculated using its rendering weights and high-frequency importance. This solves the problem that existing 3D Gaussian acceleration techniques cannot simultaneously balance training speed, rendering quality, and versatility, enabling image rendering and improving the efficiency of image rendering based on the 3D Gaussian model.Based on the present invention, addressing the issue that the training efficiency of 3D Gaussian models remains a core bottleneck in practical applications due to the large amount of redundant computation and unnecessary Gaussian units during training, resulting in excessively long training times and difficulty in meeting the needs of real-time applications and edge deployments, thus leading to low efficiency, a 3D Gaussian model-based image rendering method is designed. The effectiveness of the 3D Gaussian model-based image rendering method of the present invention is verified when rendering images. Finally, the efficiency of image rendering based on the 3D Gaussian model using the method of the present invention is significantly improved.

[0024] In this embodiment, for ease of description, the following description will focus on an image rendering device based on a three-dimensional Gaussian model.

[0025] Due to the limitations of existing 3D Gaussian acceleration techniques, image rendering based on 3D Gaussian models struggles to simultaneously achieve training speed, rendering quality, and versatility, exhibiting several technical shortcomings. One is the Gaussian grammar pruning and densification optimization problem: vague pruning criteria and blind densification targets easily lead to decreased rendering quality or computational redundancy, affecting model application performance. Another is the training process scheduling optimization problem: focusing only on frequency information while ignoring spatial locality results in insufficient scheduling targeting and significant Gaussian grammar redundancy, failing to efficiently improve training efficiency. Furthermore, there is the issue of wavelet-based methods: they remain at the component partitioning level without deep integration with training scheduling and Gaussian grammar management, failing to address core decision-making problems and further restricting the technology's versatility.

[0026] This application provides a solution that optimizes a 3D Gaussian model using a 3D Gaussian backbone network, a resolution scheduler, and a clone trainer. Upon receiving an image to be rendered, the optimized 3D Gaussian model is used for image rendering to obtain the rendered image. This solves the problem that existing 3D Gaussian acceleration techniques cannot balance training speed, rendering quality, and versatility, and improves the efficiency of image rendering based on the 3D Gaussian model.

[0027] Based on this, embodiments of this application provide an image rendering method based on a three-dimensional Gaussian model, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the image rendering method based on a three-dimensional Gaussian model according to this application.

[0028] In this embodiment, the image rendering method based on the three-dimensional Gaussian model includes steps S01~S02: Step S01: Receive the image to be rendered; Before the implementation of this embodiment, it should be clear that existing 3DGS acceleration technologies are mainly divided into two categories: Gaussian pruning and densification (such as Mini-Splatting) and training process scheduling optimization (such as DashGaussian). In recent years, wavelet optimization methods have also emerged. However, existing technologies suffer from problems such as blind pruning and densification, insufficient scheduling targeting, and shallow integration of multiple methods. These issues can easily lead to a decrease in rendering quality or computational redundancy, and cannot simultaneously balance training speed, rendering quality, and versatility.

[0029] Therefore, in order to solve the above problems, this embodiment first receives the image to be rendered. The image to be rendered can be any image data input by the user that needs to be reconstructed in three dimensions or synthesized in view. For example, it can contain multiple two-dimensional images of the target scene from different perspectives. These images will serve as the basic data source for subsequent three-dimensional Gaussian model optimization and image rendering, providing the original visual information input for subsequent steps.

[0030] Step S02: The image to be rendered is rendered using the optimized 3D Gaussian model to obtain the rendered image. The 3D Gaussian model is constructed using a 3D Gaussian backbone network, a resolution scheduler, and a clone scheduler.

[0031] Subsequently, the optimized 3D Gaussian model is used to render the image to be rendered, resulting in the rendered image. The construction of the 3D Gaussian model integrates a 3D Gaussian backbone network, a resolution scheduler, and a clone scheduler. The 3D Gaussian backbone network, as the core part of the model, is responsible for feature extraction and preliminary 3D Gaussian distribution parameterization of the input image data to be rendered, providing a basic geometric and appearance representation for subsequent rendering. The resolution scheduler is used to dynamically adjust the processing resolution according to the needs of different stages or the characteristics of the image data during model training or rendering, in order to balance the consumption of computing resources and the accuracy of rendering details. The clone scheduler performs cloning operations on the primitives in the 3D Gaussian model under specific conditions, and captures detailed information in the scene more efficiently by copying and adjusting existing Gaussian primitives, avoiding the computational redundancy caused by blind densification, while enhancing the model's ability to express complex structures. The three work together to form an optimized 3D Gaussian model that can balance training speed, rendering quality, and versatility.

[0032] Specifically, before step S02 above, which involves rendering the image to be rendered using the optimized 3D Gaussian model to obtain the rendered image, the method further includes: Step S021: Receive the training image set; Step S022: Perform discrete wavelet transform on several images in the training image set to obtain low-frequency approximation coefficients and high-frequency detail coefficients; Step S023: Based on the low-frequency approximation coefficients and high-frequency detail coefficients, the dynamic resolution parameters are calculated by the resolution scheduler. Step S024: Calculate the high-frequency importance map by using the high-frequency detail coefficients on the plurality of images; Step S025: Based on the high-frequency importance graph, the Gaussian element set in the three-dimensional Gaussian backbone network is optimized by the clone scheduler to obtain the optimized Gaussian element set. Step S026: Based on the dynamic resolution parameters and the optimized Gaussian set, the optimized three-dimensional Gaussian model is obtained.

[0033] This embodiment first receives the training image set to provide basic data support for subsequent wavelet decomposition and model optimization. In this step, it is necessary to ensure the integrity and validity of the input data, which is a prerequisite for the entire training process.

[0034] Subsequently, discrete wavelet transform was performed on several images in the training image set to achieve joint frequency-spatial feature extraction. Specifically, for each training image... Performing a discrete wavelet transform yields the decomposition result:

[0035] in These are low-frequency approximation coefficients (capturing the overall structure of the scene while preserving the global contour information of the image). J represents the high-frequency detail coefficients of the k-th level decomposition (including three directional subbands: horizontal LH, vertical HL, and diagonal HH, accurately capturing local key features such as edges and textures), and J represents the wavelet decomposition level (which can be flexibly set to 2-4 levels according to actual scene requirements). Through the above steps, wavelet transform is introduced into 3D Gaussian training, providing multi-scale feature basis for subsequent dynamic adjustment and precise optimization.

[0036] Based on the low-frequency approximation coefficients (reflecting global structural complexity) and high-frequency detail coefficients (reflecting local feature richness) obtained from the wavelet decomposition above, dynamic resolution parameters are calculated through a resolution scheduler. Unlike the traditional fixed resolution training mode, the resolution scheduler can dynamically match the training resolution according to the feature complexity of different regions of the image. Lower resolution is used in global structural regions to improve efficiency, while higher resolution is used in dense detail regions to ensure quality, thus achieving an adaptive balance between efficiency and quality.

[0037] Subsequently, high-frequency importance maps are obtained by calculating high-frequency detail coefficients on several images. Since high-frequency coefficients mainly reflect visually sensitive areas such as image edges and textures, high-frequency importance maps can accurately quantify the feature importance of different image regions, providing spatial location guidance for subsequent targeted optimization of Gaussian elements and solving the problem of blind allocation of Gaussian elements in traditional methods.

[0038] Furthermore, this embodiment optimizes the Gaussian primitive set in the 3D Gaussian backbone network based on the high-frequency importance graph using a cloning scheduler. The scheduler increases the Gaussian primitive density in regions with high high-frequency importance (such as edges and textures) and reduces redundant primitives in smooth regions with low importance, thus achieving on-demand allocation of Gaussian primitives. This avoids the quality degradation caused by blind pruning in methods such as Mini-Splatting and reduces primitive redundancy problems in methods such as DashGaussian.

[0039] Finally, based on the dynamic resolution parameters (to optimize training efficiency) and the optimized Gaussian meta-set (to ensure rendering quality), an optimized 3D Gaussian model is obtained by fusion, thereby achieving deep collaboration between resolution scheduling and Gaussian meta-management, breaking through the bottleneck of existing technologies that make it difficult to simultaneously balance training speed, rendering quality, and versatility.

[0040] More specifically, step S023 above, which involves calculating the dynamic resolution parameters using the resolution scheduler based on the low-frequency approximation coefficients and high-frequency detail coefficients, includes: Step S0231: Calculate the wavelet saliency of the several images using the low-frequency approximation coefficients and high-frequency detail coefficients to obtain a single-image wavelet saliency map; Step S0232: Calculate the saliency average of the training image set using the low-frequency approximation coefficients and high-frequency detail coefficients to obtain a scene saliency map; Step S0233: Calculate the standard deviation and mean of the scene saliency map to obtain the scene complexity coefficient; Step S0234: Calculate the dynamic resolution parameters using the scene saliency map and scene complexity coefficient.

[0041] Through low-frequency approximation coefficients ( (reflecting the global structure of the image) and high-frequency detail coefficients ( The wavelet saliency of several images (including local features in the horizontal, vertical and diagonal directions) is calculated to obtain a single image wavelet saliency map. This breaks through the limitation of relying only on high-frequency or low-frequency single features in the traditional method. That is, the low-frequency coefficients ensure the global consistency of saliency assessment, while the high-frequency coefficients strengthen the saliency characterization of local sensitive areas such as edges and textures. This allows the single image saliency map to reflect the importance of the overall structure and accurately locate key local details.

[0042] Subsequently, based on the obtained single-image wavelet saliency maps, the average saliency of the training image set is calculated to obtain the scene saliency map. This achieves a leap in analytical dimensions from local single-image analysis to global scene analysis. It can intuitively reflect the average saliency distribution of different regions in the entire training scene, avoid the local bias that may exist in single image analysis, provide global feature basis for subsequent overall scene complexity assessment, and is a key foundation for achieving unified scheduling across images.

[0043] Subsequently, the scene saliency map was analyzed. Perform standard deviation calculation ( ) and mean calculation ( The scene complexity coefficient is obtained by combining the scene-level complexity evaluation formula. The specific formula is as follows:

[0044] In the formula Used to avoid the denominator being zero The value ranges from 0.8 to 1.3. The above steps combine statistical characteristics with quantitative formulas. It reflects the degree of difference in significance between different areas within the scene (the greater the difference, the more diverse the details). Representing the overall saliency level of the scene, ξ, calculated using a weighted formula, can accurately quantify the richness of scene details. A higher value indicates richer changes in scene details, providing a quantifiable complexity measurement standard for subsequent resolution adjustments and overcoming the subjectivity problem of traditional complexity assessment.

[0045] Finally, the scene saliency map was used. (Locating "where high resolution is needed") and scene complexity coefficient (Determine "how high a resolution is needed") Calculate the dynamic resolution parameters, when the scene complexity coefficient At a higher level (rich in detail), combined with In regions of medium to high saliency, the training resolution of the corresponding region is automatically increased. When the scene is relatively flat, the overall resolution is appropriately reduced to improve efficiency. This decision-making mechanism, which considers both location and degree, enables the on-demand allocation of resolution resources. Compared with traditional scheduling methods that rely on a single feature, this mechanism is more intelligent and precise, effectively balancing training speed and rendering quality.

[0046] Further, step S0234 above, the step of calculating the dynamic resolution parameters using the scene saliency map and scene complexity coefficient, includes: Step S02341: Determine the resolution update interval through the scene complexity coefficient, and calculate the resolution level through the densification period and the resolution update interval; Step S02342: Determine the maximum resolution scale using the scene saliency map; Step S02343: Calculate the dynamic resolution parameters based on the resolution update interval, resolution level, and maximum resolution scale.

[0047] First, by considering the scene complexity coefficient Determine the resolution update interval and adopt the same method as... The calculation logic is inversely proportional, and the formula is as follows:

[0048] In the formula, Int is the optimal fixed interval for pre-verification (default is set to 250, which can be flexibly adjusted to 200-300 according to hardware resources) to achieve dynamic adaptation of the update frequency in complex scenarios. (High) corresponds to smaller This ensures that the resolution responds more frequently to changes in detail, even in simple scenes. (Low) increase To reduce ineffective updates and improve efficiency, this is then combined with densification cycles. and The formula for calculating the number of resolution levels L is:

[0049] This approach deeply binds resolution scheduling with the primitive densification process, keeping the resolution level and primitive optimization rhythm synchronized, thus avoiding resource waste or quality loss caused by the disconnect between the two in traditional scheduling.

[0050] Subsequently, scene saliency maps were used. Determine the maximum resolution scale Specifically, through solving Received, among which For size wavelet saliency of the center window To achieve overall scene saliency, thus breaking through the limitations of traditional fixed maximum resolution, the design focuses on capturing core frequency components. The central window saliency accounts for 1 / 4 of the total saliency, which not only ensures that the core structure and key frequency information of the scene are captured first in the initial low-resolution training stage, but also reserves space for detail supplementation in subsequent high-resolution layers. This realizes the resolution progression logic of first capturing the core and then the details, and improves the utilization efficiency of training resources.

[0051] Finally, update the interval based on resolution. (Scheduling frequency), resolution level L (number of scheduling stages), and maximum resolution scale (Scale upper limit), combined with the hierarchical target saliency and the initiation iteration rule, the dynamic resolution parameters are calculated, where the target saliency of each level is calculated by smoothing step size interpolation, the formula is:

[0052] In the formula For minimum significance, The smooth step size function avoids training instability caused by abrupt changes in resolution scale between different layers, and the timing of each layer's initial iteration follows...

[0053] in This ensures that the highest resolution layer has sufficient training iteration time to optimize details, and realizes multi-dimensional collaborative decision-making of frequency, stage and scale. The generated dynamic resolution parameters not only conform to the changing laws of scene complexity, but also take into account training stability and efficiency. Compared with the single parameter driven scheduling method, it is more scientific and practical.

[0054] This embodiment, through the above-described scheme, specifically involves receiving an image to be rendered; rendering the image using an optimized 3D Gaussian model to obtain a rendered image. The 3D Gaussian model is constructed using a 3D Gaussian backbone network, a resolution scheduler, and a clone scheduler. Thus, by optimizing the 3D Gaussian model using the 3D Gaussian backbone network, resolution scheduler, and clone trainer, and then rendering the received image using the optimized 3D Gaussian model, the embodiment solves the problem in existing 3D Gaussian acceleration techniques that cannot simultaneously balance training speed, rendering quality, and versatility, thereby improving the efficiency of image rendering based on the 3D Gaussian model.

[0055] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 2 In step S025 of the first embodiment above, which optimizes the Gaussian primitive set in the three-dimensional Gaussian backbone network by using the clone scheduler based on the high-frequency importance graph to obtain the optimized Gaussian primitive set, the image rendering method based on the three-dimensional Gaussian model further includes steps S0251 to S0253: Step S0251: When the number of iterations of the dynamic resolution parameter reaches the cloning threshold, the scores of several Gaussian elements in the Gaussian element set are calculated using the high-frequency importance graph to obtain the cloning score. Step S0252: Based on the cloning score, select the Gaussian units to be cloned from the plurality of Gaussian units by means of a sorting threshold; Step S0253: After cloning the Gaussian unit to be cloned, add it to the Gaussian unit set, and prune the Gaussian unit set to obtain an optimized Gaussian unit set.

[0056] First, determine if the iteration count of the dynamic resolution parameter has reached the cloning threshold. The cloning trigger condition follows dynamic rules; when iteration t satisfies... Cloning is triggered at time, where Let be the resolution scale for the t-th iteration. For the previous cloning iteration, and (represented by the minimum and target intervals respectively), coupling resolution changes with the iteration interval avoids frequent cloning or prolonged periods without primitive updates during resolution abrupt changes. After cloning is triggered, the cloning score is obtained by calculating the scores of several Gaussian primitives in the Gaussian primitive set using a high-frequency importance graph. The high-frequency importance graph is obtained using the formula:

[0057] Generate, where This indicates bilinear upsampling to the original image resolution. (As a high-frequency hierarchical weighting factor), this formula strengthens the importance of high-resolution high-frequency subbands through hierarchical weighting, enabling the clone score to accurately reflect the contribution of primitives to detailed features, and providing a quantitative basis for subsequent screening.

[0058] Based on cloning scores, a ranking threshold is used to select Gaussian units to be cloned from a number of Gaussian units. Unlike traditional fixed-ratio screening, the ranking threshold can be dynamically adjusted according to the complexity of the scene, which is particularly useful in scenes with rich details. (High) Appropriately lower the threshold to increase the number of clones, and raise the threshold to simplify the screening in smooth scenarios. By quantifying and sorting the clone scores, ensure that the selected Gaussian primitives to be cloned are all key primitives that contribute the most to high-frequency details, avoid computational redundancy caused by blind cloning, and achieve targeted optimization of Gaussian primitive enhancement.

[0059] Finally, the selected Gaussian primitives to be cloned are cloned and added to the Gaussian primitive set. At the same time, pruning operations are performed to obtain an optimized Gaussian primitive set. The specific execution rules are as follows: the top 1% of Gaussian primitives in terms of cloning score are selected for cloning to ensure the efficiency of primitive enhancement. Gaussian simplification is used for pruning at 3500 iterations. By merging similar Gaussian primitives and simplifying primitives with small variance, the pressure of aggressive cloning on the computational load is alleviated. The simultaneous execution of cloning enhancement and pruning simplification is achieved, forming a closed loop of improving the best and eliminating the worst. It not only supplements the primitives needed for details through cloning, but also eliminates redundancy through pruning. This solves the problem of quality and efficiency imbalance caused by traditional methods that only increase or only decrease without increasing.

[0060] Specifically, step S0251 above, which involves calculating the scores of several Gaussian elements in the Gaussian element set using the high-frequency importance graph to obtain the cloning scores, includes: Step S02511: Project the plurality of Gaussian primitives onto the training image set to obtain the two-dimensional coordinates of the plurality of Gaussian primitives; Step S02512: Based on the high-frequency importance map, perform bilinear interpolation sampling on the two-dimensional coordinates to obtain the global view score of the plurality of Gaussian elements; Step S02513: Determine the high-frequency importance of the plurality of Gaussian elements based on the global view score; Step S02514: The clone score of the several Gaussian units is calculated by using the rendering weights and high-frequency importance of the several Gaussian units.

[0061] First, several Gaussian primitives are projected onto the training image set to obtain the two-dimensional coordinates of each primitive. This establishes a direct spatial relationship between the three-dimensional Gaussian primitives and the two-dimensional image view, overcoming the limitation of traditional methods that rely solely on three-dimensional coordinates to determine the importance of primitives. It can accurately locate the corresponding position of Gaussian elements in a specific image view, laying a spatial matching foundation for subsequent calculation of importance by combining high-frequency image features.

[0062] Subsequently, based on the high-frequency importance map, bilinear interpolation sampling is performed on the two-dimensional coordinates to obtain the single-view components in the global view score of several Gaussian elements. Before sampling, the coordinates are normalized to... , where the x component is The y-component is This normalization process ensures the consistency of coordinate sampling in images of different resolutions, while bilinear interpolation sampling effectively utilizes the feature information of adjacent pixels in the high-frequency importance map, improving the smoothness and accuracy of the sampling results, and ultimately obtaining Gaussian elements. High-frequency importance of single views in view n This achieves precise binding between high-frequency features and the spatial location of primitives.

[0063] Based on high-frequency importance of single view The global high-frequency importance of several Gaussian elements is determined by summation calculation. The summation range is the Gaussian element. All visible views enable the aggregation and upgrade from local features in a single view to global features across multiple views. It can comprehensively reflect the high-frequency contribution of Gaussian elements under all visible views in the entire training scene, avoids the view bias that may exist in single-view sampling, makes the importance assessment more comprehensive and objective, and provides global feature support for cloning priority judgment.

[0064] Through several high-level primitive rendering weights (Cumulative mixed weights) and normalized high-frequency importance The cloning fraction Si is calculated using the following formula: In the formula (Adjustable to 1.5-2.5) is the high-frequency influence weighting factor. This calculation method integrates two key factors: rendering contribution and high-frequency importance. This reflects the fundamental contribution of primitives to the overall rendering effect. This reflects its value in enhancing detailed features. The factors can be flexibly adjusted according to the needs of scene details, and the influence weight of high frequency importance can be adjusted. Compared with the traditional score calculation that only relies on a single indicator, this embodiment can more accurately quantify the cloning value of primitives, and ensure that the cloning operation prioritizes key primitives that make a large contribution to rendering and have high demand for detail enhancement, thereby improving the efficiency and targeting of high primitive optimization.

[0065] This embodiment, through the above-described scheme, specifically involves calculating the cloning score by using the high-frequency importance graph to score several Gaussian elements in the Gaussian element set when the number of iterations of the dynamic resolution parameter reaches the cloning threshold. Based on the cloning score, Gaussian elements to be cloned are selected from these elements using a sorting threshold. After cloning, these Gaussian elements are added to the Gaussian element set, and the Gaussian element set is pruned to obtain an optimized Gaussian element set. Thus, by optimizing the 3D Gaussian model through a 3D Gaussian backbone network, a resolution scheduler, and a cloning trainer, and then rendering the image using the optimized 3D Gaussian model after receiving the image to be rendered, the embodiment solves the problem in existing 3D Gaussian acceleration techniques that cannot simultaneously balance training speed, rendering quality, and versatility, thereby improving the efficiency of image rendering based on the 3D Gaussian model.

[0066] For example, to help understand the implementation flow of the image rendering method based on the three-dimensional Gaussian model obtained by combining this embodiment with the above embodiment one, please refer to... Figure 3 , Figure 3 A simplified flowchart of an image rendering method based on a 3D Gaussian model is provided, specifically: Using Gaussian elements and N sets of input images as initial inputs, the discrete wavelet transform (DWT) is first performed on the input images to decompose them into low-frequency components that capture the overall structure of the scene and high-frequency components that carry detailed features. Based on these two components, the saliency map and the high-frequency importance map are further calculated.

[0067] The resolution scheduler and clone scheduler are integrated into the 3DGS backbone network. The saliency map and high-frequency importance map generated after the input image is processed by DWT guide the two schedulers to dynamically adjust the training parameters. During training, the resolution scheduler decides when to update the resolution, and the clone scheduler decides where to enhance Gaussian units. The two work together to achieve redundancy-free and targeted 3DGS optimization.

[0068] In the specific training process, Gaussian primitives first enter the resolution scheduler, where they dynamically adjust the resolution parameters under the guidance of the saliency map to accelerate training. Then they enter the cloning scheduler, where they are guided by the high-frequency importance map to perform targeted cloning of Gaussian primitives (supplementing detail region primitives). After collaborative optimization by the two schedulers, the corresponding point cloud is generated. Finally, the new perspective image is output through the rendering stage, completing the entire 3DGS optimization and rendering process.

[0069] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the image rendering method based on the three-dimensional Gaussian model of this application. Any simple transformations based on this technical concept are within the protection scope of this application.

[0070] This application also provides an image rendering device based on a three-dimensional Gaussian model; please refer to [reference needed]. Figure 4 The image rendering device based on the three-dimensional Gaussian model includes: Receiver module 10 is used to receive the image to be rendered; The rendering module 20 is used to render the image to be rendered using the optimized three-dimensional Gaussian model to obtain the rendered image. The three-dimensional Gaussian model is constructed using a three-dimensional Gaussian backbone network, a resolution scheduler, and a clone scheduler.

[0071] The image rendering apparatus based on a 3D Gaussian model provided in this application, employing the image rendering method based on a 3D Gaussian model in the above embodiments, can solve the technical problem that existing 3D Gaussian acceleration techniques cannot simultaneously achieve training speed, rendering quality, and versatility. Compared with the prior art, the beneficial effects of the image rendering apparatus based on a 3D Gaussian model provided in this application are the same as those of the image rendering method based on a 3D Gaussian model provided in the above embodiments, and other technical features in the image rendering apparatus based on a 3D Gaussian model are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0072] This application provides an image rendering device based on a three-dimensional Gaussian model. The image rendering device based on a three-dimensional Gaussian model includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the image rendering method based on a three-dimensional Gaussian model in the above embodiment 1.

[0073] The following is for reference. Figure 5 This document illustrates a structural schematic diagram of an image rendering device based on a three-dimensional Gaussian model suitable for implementing embodiments of this application. The image rendering device based on a three-dimensional Gaussian model in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital radio receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 5 The image rendering device based on the three-dimensional Gaussian model shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0074] like Figure 5As shown, the image rendering device based on a 3D Gaussian model may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory 1002 or a program loaded from a storage device 1003 into a random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the image rendering device based on the 3D Gaussian model. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, a touch screen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the image rendering device based on a 3D Gaussian model to communicate wirelessly or wiredly with other devices to exchange data. Although the figure shows image rendering devices based on a 3D Gaussian model with various systems, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0075] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0076] The image rendering device based on a 3D Gaussian model provided in this application, employing the image rendering method based on a 3D Gaussian model in the above embodiments, can solve the technical problem that existing 3D Gaussian acceleration techniques cannot simultaneously achieve training speed, rendering quality, and versatility. Compared with the prior art, the beneficial effects of the image rendering device based on a 3D Gaussian model provided in this application are the same as those of the image rendering method based on a 3D Gaussian model provided in the above embodiments, and other technical features in this image rendering device based on a 3D Gaussian model are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0077] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0078] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0079] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the image rendering method based on a three-dimensional Gaussian model in the above embodiments.

[0080] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0081] The aforementioned computer-readable storage medium may be included in a three-dimensional Gaussian model-based image rendering device; or it may exist independently and not be assembled into a three-dimensional Gaussian model-based image rendering device.

[0082] The aforementioned computer-readable storage medium carries one or more programs that, when executed by an image rendering device based on a three-dimensional Gaussian model, cause the image rendering device based on a three-dimensional Gaussian model to: receive an image to be rendered; and render the image to be rendered using an optimized three-dimensional Gaussian model to obtain a rendered image, wherein the three-dimensional Gaussian model is constructed through a three-dimensional Gaussian backbone network, a resolution scheduler, and a clone scheduler.

[0083] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0084] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0085] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0086] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described image rendering method based on a three-dimensional Gaussian model. This solves the technical problem in the prior art where three-dimensional Gaussian acceleration technology cannot simultaneously achieve training speed, rendering quality, and versatility. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the image rendering method based on a three-dimensional Gaussian model provided in the above embodiments, and will not be repeated here.

[0087] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the image rendering method based on a three-dimensional Gaussian model as described above.

[0088] The computer program product provided in this application can solve the technical problem that existing 3D Gaussian acceleration technology cannot simultaneously achieve training speed, rendering quality, and versatility. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the image rendering method based on the 3D Gaussian model provided in the above embodiments, and will not be repeated here.

[0089] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. An image rendering method based on a three-dimensional Gaussian model, characterized in that, The image rendering method based on the three-dimensional Gaussian model includes: Receive the image to be rendered; The image to be rendered is rendered using the optimized 3D Gaussian model to obtain the rendered image. The 3D Gaussian model is constructed using a 3D Gaussian backbone network, a resolution scheduler, and a clone scheduler.

2. The image rendering method based on a three-dimensional Gaussian model as described in claim 1, characterized in that, Before the step of rendering the image to be rendered using the optimized three-dimensional Gaussian model to obtain the rendered image, the method further includes: Receive the training image set; Discrete wavelet transform is performed on several images in the training image set to obtain low-frequency approximation coefficients and high-frequency detail coefficients; Based on the low-frequency approximation coefficients and high-frequency detail coefficients, the dynamic resolution parameters are calculated by the resolution scheduler. A high-frequency importance map is obtained by calculating the high-frequency detail coefficients on the several images; Based on the high-frequency importance graph, the Gaussian element set in the three-dimensional Gaussian backbone network is optimized by the clone scheduler to obtain the optimized Gaussian element set. The optimized 3D Gaussian model is obtained based on the dynamic resolution parameters and the optimized Gaussian metaset.

3. The image rendering method based on a three-dimensional Gaussian model as described in claim 2, characterized in that, The step of calculating the dynamic resolution parameters using the resolution scheduler based on the low-frequency approximation coefficients and high-frequency detail coefficients includes: The wavelet saliency of the several images is calculated using the low-frequency approximation coefficients and the high-frequency detail coefficients to obtain a single-image wavelet saliency map; The saliency average of the training image set is calculated using the low-frequency approximation coefficients and the high-frequency detail coefficients to obtain a scene saliency map; The standard deviation and mean of the scene saliency map are calculated to obtain the scene complexity coefficient. The dynamic resolution parameters are calculated using the scene saliency map and the scene complexity coefficient.

4. The image rendering method based on a three-dimensional Gaussian model as described in claim 3, characterized in that, The step of calculating the dynamic resolution parameters using the scene saliency map and scene complexity coefficient includes: The resolution update interval is determined by the scene complexity coefficient, and the resolution level is calculated by the densification period and the resolution update interval. The maximum resolution scale is determined using the scene saliency map. The dynamic resolution parameters are calculated based on the resolution update interval, resolution level, and maximum resolution scale.

5. The image rendering method based on a three-dimensional Gaussian model as described in claim 2, characterized in that, The step of optimizing the Gaussian element set in the three-dimensional Gaussian backbone network using the clone scheduler based on the high-frequency importance graph to obtain the optimized Gaussian element set includes: When the number of iterations of the dynamic resolution parameter reaches the cloning threshold, the scores of several Gaussian elements in the Gaussian element set are calculated using the high-frequency importance graph to obtain the cloning score. Based on the cloning score, a Gaussian unit to be cloned is selected from the plurality of Gaussian units by means of a sorting threshold; After cloning the Gaussian unit to be cloned, it is added to the Gaussian unit set, and the Gaussian unit set is pruned to obtain an optimized Gaussian unit set.

6. The image rendering method based on a three-dimensional Gaussian model as described in claim 5, characterized in that, The step of calculating the clonal scores by using the high-frequency importance graph to score several Gaussian elements in the Gaussian element set includes: The Gaussian primitives are projected onto the training image set to obtain the two-dimensional coordinates of the Gaussian primitives; Based on the high-frequency importance map, bilinear interpolation sampling is performed on the two-dimensional coordinates to obtain the global view scores of the several Gaussian elements; The high-frequency importance of the several high-level elements is determined based on the global view score; The clone score of the Gaussian primitives is calculated by using the rendering weights and high-frequency importance of the Gaussian primitives.

7. An image rendering device based on a three-dimensional Gaussian model, characterized in that, The image rendering device based on the three-dimensional Gaussian model includes: The receiving module is used to receive the image to be rendered; The rendering module is used to render the image to be rendered using an optimized 3D Gaussian model to obtain a rendered image. The 3D Gaussian model is constructed using a 3D Gaussian backbone network, a resolution scheduler, and a clone scheduler.

8. An image rendering device based on a three-dimensional Gaussian model, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the image rendering method based on a three-dimensional Gaussian model as described in any one of claims 1 to 6.

9. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the image rendering method based on a three-dimensional Gaussian model as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the image rendering method based on a three-dimensional Gaussian model as described in any one of claims 1 to 6.

Images