Image rendering method and apparatus, computing device

Abstract

Embodiments of the present specification provide image rendering methods and devices, and computing devices. The image rendering method comprises: obtaining a rendering sequence of a three-dimensional Gaussian model and a target visual element, wherein the three-dimensional Gaussian model comprises a plurality of three-dimensional Gaussian points, and any three-dimensional Gaussian point comprises depth information; determining at least one depth gap in the three-dimensional Gaussian model based on the depth information of the plurality of three-dimensional Gaussian points; adding the target visual element to the rendering sequence based on the depth gap; and rendering the rendering sequence to obtain a target rendering image. The depth consistency fusion of the target visual element and the three-dimensional scene in the rendering process is realized, the accurate position and hierarchical relationship of the target visual element in the three-dimensional space are ensured, so that the target visual element can dynamically adjust the display position with the change of the scene depth, the display quality of the target visual element in the target rendering image of the three-dimensional Gaussian model is improved, and the rendering efficiency of the target rendering image is improved.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of computer technology, and in particular to an image rendering method, an image rendering apparatus, and a computing device. Background Technology

[0002] With the popularization of 3D interactive technology, embedding visual elements such as user interface (UI) into 3D scenes has become a common requirement in applications such as augmented reality, virtual reality, and 3D games.

[0003] Currently, methods for overlaying 3D scenes and visual elements primarily involve image overlay. This involves rendering a 3D model to obtain a 2D model image, rendering the visual elements to obtain 2D visual element images, and then overlaying the two images for display. Alternatively, image compositing techniques can be used to merge the 2D visual element images with the 2D model images.

[0004] However, in the aforementioned technical solutions, the rendering of various visual elements and 3D models is performed independently. This makes it impossible to adjust the display effect of visual elements according to the actual spatial information of the 3D scene, resulting in visual elements failing to truly integrate into the 3D scene. They often remain only at the top or bottom layer, unable to form correct spatial relationships with the entities in the 3D scene, thus leading to poor final rendered image quality. Furthermore, subsequent image overlay operations prolong the overall rendering process, resulting in low rendering efficiency. Therefore, there is an urgent need for an image rendering method with better rendering quality and higher rendering efficiency. Summary of the Invention

[0005] In view of this, embodiments of this specification provide an image rendering method. One or more embodiments of this specification also relate to an image rendering apparatus, a computing device, a computer-readable storage medium, and a computer program product, to address the technical deficiencies existing in the prior art.

[0006] According to a first aspect of the embodiments of this specification, an image rendering method is provided, comprising: Obtain the rendering sequence of the 3D Gaussian model and the target visual elements, wherein the 3D Gaussian model includes multiple 3D Gaussian points, and each 3D Gaussian point includes depth information; Based on the depth information of multiple 3D Gaussian points, at least one depth gap in the 3D Gaussian model is determined. Based on depth gaps, target visual elements are added to the rendering sequence; Render the rendering sequence to obtain the target rendered image.

[0007] According to a second aspect of the embodiments of this specification, an image rendering apparatus is provided, comprising: The acquisition module is configured to acquire the rendering sequence of a 3D Gaussian model and the target visual elements, wherein the 3D Gaussian model includes multiple 3D Gaussian points, and any 3D Gaussian point includes depth information. The determination module is configured to determine at least one depth gap in a 3D Gaussian model based on depth information from multiple 3D Gaussian points. The module is configured to add target visual elements to the rendering sequence based on depth gaps; The rendering module is configured to render the rendering sequence to obtain the target rendered image.

[0008] According to a third aspect of the embodiments of this specification, a computing device is provided, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer program / instructions, which, when executed by the processor, implement the steps of the above-described image rendering method.

[0009] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores a computer program / instructions that, when executed by a processor, implement the steps of the above-described image rendering method.

[0010] According to a fifth aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described image rendering method.

[0011] One embodiment of this specification implements an image rendering method, including: obtaining a rendering sequence of a three-dimensional Gaussian model and a target visual element, wherein the three-dimensional Gaussian model includes multiple three-dimensional Gaussian points, and any three-dimensional Gaussian point includes depth information; determining at least one depth gap in the three-dimensional Gaussian model based on the depth information of the multiple three-dimensional Gaussian points; adding the target visual element to the rendering sequence based on the depth gap; and rendering the rendering sequence to obtain a target rendered image.

[0012] By acquiring the rendering sequence of the 3D Gaussian model and the target visual elements, the depth gap is determined based on the depth information of the 3D Gaussian points, and the target visual elements are dynamically added to the corresponding depth positions in the rendering sequence. This achieves consistent depth fusion between the target visual elements and the 3D scene during the rendering process. By making the target visual elements naturally present according to the depth hierarchy of the 3D scene, the depth misalignment and visual fragmentation problems caused by image overlay are avoided, ensuring the accurate position and hierarchical relationship of the target visual elements in 3D space. By directly adding the visual elements to the rendering sequence for unified rendering processing, the execution efficiency of the rendering process is improved, ensuring accurate matching between the visual elements and the 3D Gaussian model in the depth dimension. This results in a more natural and coherent 3D interactive effect, allowing the target visual elements to dynamically adjust their display position according to changes in scene depth. This improves the display quality of the target visual elements in the target rendered image of the 3D Gaussian model and the rendering efficiency of the target rendered image. Attached Figure Description

[0013] Figure 1 This is a flowchart illustrating an image rendering method provided in one embodiment of this specification; Figure 2 This is a schematic diagram of the structure of an image rendering apparatus provided in one embodiment of this specification; Figure 3 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0014] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0015] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0016] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0017] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in one or more embodiments of this specification are obtained through open-source datasets or public datasets that comply with their license agreements, or are obtained with full authorization from the relevant parties. Moreover, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0018] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0019] 3D Gaussian Splatting (3DGS) is a representation method that uses Gaussian ellipsoids to model 3D scenes. It achieves efficient rendering by rasterizing the Gaussian ellipsoid into an image. Compared with implicit representation methods such as neural radiation fields, its explicit representation facilitates downstream tasks such as dynamic reconstruction, geometric editing, and physical simulation.

[0020] Transparency (Alpha value) is a parameter that represents the degree of transparency of an image or object. It is usually represented by an Alpha value, ranging from 0 (completely transparent) to 1 (completely opaque). In computer graphics, it is used to control the transparency and blending effects of an image. It is a key parameter for achieving semi-transparent effects and affects the calculation of visibility probability.

[0021] Alpha blending is a computer graphics image compositing technique that uses alpha channels (transparency channels) to control the transparency of images or layers, achieving smooth transitions and blending effects between multiple images or layers. This technique is widely used in 3D rendering, video editing, game development, and other fields, enabling the correct display of semi-transparent objects and natural layering, enhancing the realism and depth of visual effects. It is one of the key technologies in modern graphics rendering pipelines.

[0022] A user interface (UI) is the medium and bridge for interaction between humans and computer systems. It can include all elements and methods of information exchange between users and the system, such as graphical interfaces, command-line interfaces, and touch interfaces. The main function of a user interface is to provide users with intuitive and convenient operating methods, enabling them to effectively control and use the computer system, while presenting the system's status and information in a user-understandable form. It is an indispensable and important component of modern software systems.

[0023] Heap sort is a comparison-based sorting algorithm based on a heap data structure (a special type of complete binary tree). Its core idea is to construct a max-heap or min-heap from the sequence to be sorted, and then repeatedly swap the top element (maximum or minimum value) with the tail element, adjusting the heap structure to obtain an ordered sequence. Heap sort can be used for efficient sorting of large amounts of data, ensuring good time complexity even in the worst case.

[0024] A graphics processing unit (GPU) is a microprocessor specifically designed for processing graphics and image computing. It has a large number of parallel processing cores that can execute thousands of computing tasks simultaneously. It adopts a single instruction multiple data architecture to achieve efficient parallel computing and plays a key role in fields such as 3D graphics rendering, video processing, and deep learning. Through its parallel computing architecture, it provides higher computing performance than traditional processors.

[0025] The Central Processing Unit (CPU) is the core processing unit of a computer, responsible for executing program instructions and controlling computer operations. It adopts a sequential processing architecture to manage data flow and computing tasks, plays a key role in general computing, and has complex control logic and cache systems. It can efficiently handle branch prediction and complex logical operations, forming a complementary computing architecture with specialized processing units such as graphics processing units.

[0026] Convolutional Neural Networks (CNNs) are a deep learning neural network architecture that extracts hierarchical features from input data through a combination of convolutional layers, pooling layers, and fully connected layers. Convolutional layers use learnable filters to slide across the input data to detect local patterns, while pooling layers downsample the feature maps to reduce computation and enhance translation invariance. CNNs have shown excellent performance in computer vision tasks such as image recognition, object detection, and semantic segmentation.

[0027] The Transformer network is a deep learning architecture based on self-attention mechanism. It captures the dependencies between different positions in the input sequence through multi-head attention mechanism. It can process sequence data without using recurrent neural networks or convolutional neural networks, and has made breakthrough progress in natural language processing tasks. Its encoder and decoder structure can process the entire input sequence in parallel, improving training efficiency and model performance.

[0028] With the popularization of 3D interactive technology, embedding visual elements such as user interfaces (UI) into 3D scenes has become a common requirement in applications such as augmented reality, virtual reality, and 3D games.

[0029] Currently, methods for overlaying 3D scenes and visual elements primarily involve image overlay. This involves first rendering a 3D model to obtain a 2D model image, then rendering the visual elements to obtain 2D visual element images, and finally overlaying the two images for display. Alternatively, image compositing techniques can be used to merge the 2D visual element images with the 2D model images.

[0030] However, in the above-mentioned technical solutions, the rendering of various visual elements and 3D models is carried out independently. It is impossible to adjust the display effect of visual elements based on the actual spatial information of the 3D scene, which results in visual elements not being able to truly integrate into the 3D scene. They are often only located at the top or bottom layer and cannot form the correct spatial relationship with the entities in the 3D scene, resulting in poor display quality of the final rendered image. At the same time, the subsequent image overlay operation will prolong the overall rendering and display process, resulting in low rendering efficiency.

[0031] In view of this, an image rendering method is provided in this specification. This specification also relates to an image rendering apparatus, a computing device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.

[0032] See Figure 1 , Figure 1 A flowchart of an image rendering method according to an embodiment of this specification is shown, which specifically includes the following steps.

[0033] Step 102: Obtain the rendering sequence of the 3D Gaussian model and the target visual elements, wherein the 3D Gaussian model includes multiple 3D Gaussian points, and any 3D Gaussian point includes depth information.

[0034] The image rendering method provided in one or more embodiments of this specification can be specifically applied to the production and display of various three-dimensional interactive content, including applications such as augmented reality, virtual reality, and 3D games. This image rendering method can be deployed on computing platforms that support 3D graphics rendering, including hardware platforms such as personal computers, workstations, cloud rendering servers, and mobile terminal devices, as well as software platforms such as various game engines, 3D modeling software, and virtual reality development frameworks. In specific applications, this image rendering method is particularly suitable for scenarios that require embedding various visual elements, such as user interface elements, into 3D scenes, thereby enhancing the realism and naturalness of the 3D interactive experience.

[0035] A 3D Gaussian model is a 3D scene representation model built using 3D Gaussian Splatting (3DGS) technology. It can fit and express the geometric structure and appearance features of a 3D scene using a large number of 3D Gaussian points with specific properties. 3D Gaussian models can be used to achieve high-precision 3D scene reconstruction and rendering, representing each point in the scene as a distribution of 3D Gaussian points with attributes such as position, covariance, opacity, and spherical harmonics. Specifically, each 3D Gaussian point in the model can be independently controlled in terms of shape, size, color, and transparency. The final image can be formed by superimposing and projecting a large number of 3D Gaussian points, thus achieving efficient 3D scene rendering while ensuring rendering quality.

[0036] A rendering sequence is a list of 3D Gaussian points arranged in a specific order. It can include rendering order information, rendering parameter information, and spatial position information for each Gaussian point. The rendering sequence guides the rendering process of a 3D Gaussian model, ensuring that each Gaussian point is rendered in a predetermined order. Specifically, since a 3D Gaussian model consists of a large number of discrete Gaussian points, to present correct occlusion relationships and semi-transparency visual effects on a 2D screen, all Gaussian points can be sorted according to certain rules, such as the distance values ​​of each Gaussian point to the virtual camera. The resulting rendering order list, from near to far or from far to near, is the rendering sequence. This ensures that the occlusion relationships and visibility between Gaussian points are correctly handled during the rendering process, thereby obtaining accurate 3D scene rendering results.

[0037] Target visual elements are visual elements that need to be embedded into a 3D scene, and can be used for interactive prompts, information display, or decoration. Target visual elements can include user interface controls such as buttons, sliders, text labels, and icons; they can also be specific identifiers, graphics, or patterns, such as a flashing arrow indicator or a highlighted outline of an object; in more complex scenes, they can also be another independent, simpler 3D model, such as a cuboid or a sphere. Specifically, target visual elements usually exist in the form of 2D images or vector graphics. During the embedding process into a 3D scene, they need to be converted and processed according to their expected position and depth information in 3D space to ensure that they can be correctly integrated with the 3D scene.

[0038] A 3D Gaussian point is the basic building block in a 3D Gaussian model, representing an ellipsoidal or spherical Gaussian distribution in space. Each 3D Gaussian point includes corresponding attributes, such as position coordinates, covariance matrix, opacity, and spherical harmonic coefficients, used to characterize the geometric and appearance features of a specific location in 3D space. 3D Gaussian points can be used to jointly fit the geometry and appearance of a target object. During rendering, these points are projected onto a 2D imaging plane. By adjusting the attribute parameters of each 3D Gaussian point, the color, transparency, and shape representation during rendering can be controlled, ultimately determining the color of pixels in the rendered image.

[0039] Depth information is numerical information describing the distance of a 3D Gaussian point relative to the virtual camera's viewpoint in 3D space. It can be represented as absolute depth values, relative depth values, or depth ranges. Depth information can be used to determine the order and occlusion relationships of 3D Gaussian points during the rendering process, providing spatial positional references for the correct rendering of 3D scenes. Specifically, depth information can typically be obtained by analyzing 2D images using a pre-trained depth estimation model, or calculated based on the coordinates of each 3D Gaussian point in the 3D Gaussian model. Furthermore, depth information can not only determine the order of primitives (including 3D Gaussian points and visual elements) in the rendering sequence, but also identify areas of depth discontinuity in the 3D Gaussian model—the gaps or edges between objects—thereby determining the insertion position of target visual elements to ensure they conform to the spatial hierarchy of the scene.

[0040] In practical applications, obtaining the rendering sequence of a 3D Gaussian model can be achieved through various methods.

[0041] One alternative approach is to directly read the rendering sequence of the 3D Gaussian model from a local storage system such as a hard drive or a cloud storage system such as a network database, or to transmit or receive the rendering sequence input from the front-end user interface via a data transmission interface over a network.

[0042] Another alternative approach is to first obtain the model image of the 3D Gaussian model, then determine the depth information of each 3D Gaussian point based on the model image, and further determine the rendering sequence of the 3D Gaussian model based on the depth information. Specifically, determining the depth information based on the model image can be done using a pre-trained depth estimation model to obtain an initial depth map, and then the depth information of each 3D Gaussian point can be determined based on the initial depth map.

[0043] Another option is to directly use the rendering sequence generated during the training and construction of the 3D Gaussian model, and further customize the rendering sequence according to the rendering requirements of specific application scenarios.

[0044] In practical applications, target visual elements can be acquired through various means.

[0045] One alternative approach is to directly obtain the target visual element from a predefined visual element library in the storage system. This visual element library can include various types of reusable visual element resources such as user interface controls, icons, and text templates. Appropriate elements can be selected from this visual element library as the target visual element according to actual needs.

[0046] Another alternative approach is to dynamically create target visual elements through programmatic generation. For example, text labels can be dynamically generated based on user-input text content, or custom graphic elements can be generated based on specific parameter configurations.

[0047] Furthermore, target visual elements can be imported through external data sources, such as vector graphics files exported from design tools or bitmap files exported from image processing software. These external files can be appropriately formatted and processed to be used as target visual elements.

[0048] In addition, after acquiring the target visual elements, they usually need to be preprocessed, such as size adjustment, color correction, and transparency settings, to ensure that they can be correctly integrated with the 3D scene.

[0049] In this step, by acquiring the rendering sequence of the 3D Gaussian model and the target visual elements, a data foundation is provided for embedding the target visual elements into the 3D scene. The rendering sequence of the 3D Gaussian model can contain the depth information and rendering order of each 3D Gaussian point, providing a reference for the accurate positioning of the target visual elements in 3D space, and ensuring that the target visual elements can be integrated with the 3D scene according to the correct depth level.

[0050] Step 104: Based on the depth information of multiple 3D Gaussian points, determine at least one depth gap in the 3D Gaussian model.

[0051] Depth gaps are regions of significant depth difference in the depth dimension between adjacent Gaussian points in a 3D Gaussian model. These gaps can include gaps between objects, transition areas between foreground and background, or depth discontinuities between different surfaces of the same object. Depth gaps characterize the spatial separation between different objects or different parts of the same object in a 3D scene, providing a reference for the insertion position of target visual elements in 3D space. Specifically, depth gaps are typically regions where depth values ​​abruptly change or depth gradients significantly alter the depth information. Identifying these regions helps determine suitable depth positions for inserting target visual elements into the 3D scene, allowing them to naturally integrate into the depth hierarchy of the 3D scene.

[0052] In practical applications, determining at least one depth gap in a 3D Gaussian model based on the depth information of multiple 3D Gaussian points can be achieved through various methods.

[0053] One alternative method is to divide the 3D Gaussian model based on the depth information of each 3D Gaussian point, obtaining multiple 3D Gaussian point depth intervals. Then, based on the depth information of the 3D Gaussian points within each depth interval, at least one depth gap in the 3D Gaussian model can be determined. Specifically, the 3D Gaussian points can first be sorted according to their depth information. Then, according to a preset depth step size, the sorted 3D Gaussian points can be sampled and grouped to form multiple depth intervals. The number of 3D Gaussian points within each depth interval can be counted, and the gradient of the change in the number of 3D Gaussian points between adjacent depth intervals can be calculated. If the gradient exceeds a preset gradient threshold, the location can be determined as a depth gap.

[0054] Another alternative approach is to determine the depth gradient between the depth intervals of the three-dimensional Gaussian points based on their depth information. Then, based on the depth gradient and a preset depth change threshold, the locations of steep depth drops and steep depth rises are determined. Finally, based on these locations, at least one depth gap in the three-dimensional Gaussian model is identified. Specifically, the difference in depth values ​​between adjacent depth intervals can be calculated. If the difference is greater than a preset depth difference threshold, the location is marked as a steep depth drop or a steep depth rise. If a steep depth drop location is adjacent to a steep depth rise location, the region between these two locations can be defined as a depth gap.

[0055] Another alternative method is to determine the depth gap based on the rendered depth map of a 3D Gaussian model. The rendered depth map can be obtained based on the 3D Gaussian model rendered from the current observation view. Then, by traversing each pixel in the rendered depth map, the depth difference between adjacent pixels is calculated. If the depth difference exceeds a preset depth difference threshold, the position can be recorded as the edge position of the depth jump. Then, along the depth distribution gradient direction at the edge position, samples are taken on both sides of the jump edge to obtain the near-end depth sample set and the far-end depth sample set. Based on the near-end depth sample set and the far-end depth sample set, the candidate depth layer boundary is determined as the depth gap.

[0056] In addition, pre-trained foreground / background separation models, subject separation models, or depth separation models can be used to directly analyze and separate the original or rendered images of the 3D Gaussian model. The depth sets corresponding to the separated different regions are compared, and the non-overlapping interval between the nearest depth of the foreground region and the farthest depth of the background region is determined as the depth gap. If there is an intersection in depth between the regions, it is considered that there is no depth gap.

[0057] In this step, by using depth information from multiple 3D Gaussian points, at least one depth gap in the 3D Gaussian model is determined. This provides crucial spatial reference information for the accurate positioning of the target visual element in the 3D scene, enabling the system to identify spatial separation regions between different objects or parts in the 3D scene. This provides an objective basis for selecting the insertion position of the target visual element and avoids spatial conflicts or occlusion between the target visual element and existing objects in the 3D scene. Through accurate identification of the depth gap, the system can insert the target visual element into the naturally existing spatial gap in the 3D scene, allowing the target visual element to form a correct spatial hierarchy with the 3D scene in the depth dimension, thereby achieving deep fusion between the target visual element and the 3D scene.

[0058] Step 106: Add the target visual element to the rendering sequence based on the depth gap.

[0059] In practical applications, adding target visual elements to the rendering sequence based on depth gaps can be achieved through various methods.

[0060] One alternative approach is to directly insert the target visual element into the rendering sequence at the depth position corresponding to the depth gap. Specifically, the depth value of the depth gap can be used as the depth reference value of the target visual element. Then, the position corresponding to this depth value is found in the rendering sequence, and the target visual element is inserted as a rendering unit into that position, so that the target visual element can be processed according to the correct depth order during the rendering process.

[0061] Another alternative approach is to calculate the insertion position of the target visual element in the rendering sequence based on the depth range of the depth gap and the expected display depth of the target visual element. Specifically, the near and far depths of the depth gap can be used as boundary conditions. Based on the expected position of the target visual element in 3D space, its corresponding depth value can be calculated. Then, the insertion point corresponding to this depth value can be found in the rendering sequence, and the target visual element can be inserted into that position to ensure that the target visual element can form the correct spatial hierarchy with other elements in the 3D scene.

[0062] Furthermore, when there are multiple target visual elements, these elements can be inserted into one or more different depth gaps. Specifically, based on the depth characteristics and spatial location of one or more depth gaps, multiple target visual element instances can be assigned to different depth gaps, allowing multiple target visual elements to be displayed simultaneously at different depth levels in the 3D scene.

[0063] Furthermore, based on the dynamic changes in depth gaps and the dynamic attributes of target visual elements, the dynamic insertion and adjustment of target visual elements in the rendering sequence can be achieved. Specifically, when objects in a 3D scene move or deform, causing changes in depth gaps, the insertion position of the target visual element in the rendering sequence can be dynamically updated. This allows the target visual element to dynamically adjust its display position and depth level as the 3D scene changes, maintaining spatial consistency with the 3D scene.

[0064] In this step, by adding the target visual element to the rendering sequence based on the depth gap, the target visual element and the 3D Gaussian model are deeply integrated during the rendering process. This allows the target visual element to be processed in a unified rendering order with the 3D Gaussian points, avoiding the separation and inefficiency of the rendering process caused by image overlay. By determining the insertion position of the target visual element based on the depth gap, the accurate positioning of the target visual element in 3D space is ensured, allowing the target visual element to naturally integrate into the depth hierarchy of the 3D scene and form the correct spatial relationship with other elements in the 3D scene. This ensures the accurate matching of the target visual element and the 3D Gaussian model in the depth dimension, resulting in a more natural and coherent 3D interactive effect in the rendering result. This improves the display quality of the target visual element in the target rendered image and also improves the execution efficiency of the entire rendering process.

[0065] Step 108: Render the rendering sequence to obtain the target rendered image.

[0066] Rendering is the process of converting 3D scene data into a 2D image. Specifically, it can include a series of graphics processing operations on each primitive in the rendering sequence, such as 3D Gaussian points and target visual elements, performing shading, blending, and depth testing according to their depth order. Rendering can be based on rasterization technology, projecting 3D Gaussian points onto the imaging plane of a virtual camera, and determining the color value of a pixel by calculating the color and opacity contributions of multiple Gaussian points affecting each pixel; it can also use ray tracing technology, calculating pixel color by simulating the propagation path of light in the scene; or it can utilize a graphics processing unit (GPU) to process all 3D Gaussian points and target visual elements in parallel, improving the rendering efficiency of the target image.

[0067] A target rendered image is the final two-dimensional image result obtained after rendering a rendering sequence. It can include the scene content of a three-dimensional Gaussian model and the target visual elements embedded within it. The target rendered image can be displayed directly on a display device, saved as image data in file format, or used as input frames for subsequent video encoding, thus presenting a complete three-dimensional scene and its interactive elements, providing users with an intuitive three-dimensional visual experience. Specifically, the color value of each pixel in the target rendered image can be jointly determined by the three-dimensional Gaussian point at the corresponding position in the rendering sequence and the target visual element. Through unified rendering processing, the correct fusion of the target visual element and the three-dimensional scene in the depth dimension is ensured, allowing the target visual element to naturally integrate into the depth hierarchy of the three-dimensional scene.

[0068] In practical applications, the rendering of a rendering sequence to obtain the target rendering image can be achieved through various methods.

[0069] One alternative approach is to use a rasterization-based rendering pipeline to process the rendering sequence. Specifically, each 3D Gaussian point and target visual element in the rendering sequence can be used as a rendering primitive, and vertex transformation, projection, rasterization, and fragment shading operations can be performed sequentially according to the order determined in the rendering sequence. The processing results are then written to the frame buffer to generate the target rendered image.

[0070] Another alternative approach is to use ray tracing-based rendering techniques to process the rendering sequence. Specifically, rays can be emitted from the virtual camera's viewpoint, and the intersection points of the rays with each rendering unit can be determined based on the depth information and spatial position of each rendering unit in the rendering sequence. Then, the color and lighting effects at each intersection point are calculated, and finally, the target rendered image is synthesized.

[0071] Another option is to use hybrid rendering techniques, combining the advantages of rasterization and ray tracing to process the rendering sequence. Specifically, rasterization can be used to process 3D Gaussian points with small depth variations, such as the background, to ensure rendering efficiency; simultaneously, ray tracing can be used to process target visual elements or 3D Gaussian points with large depth variations to improve rendering quality; finally, the two rendering results are fused to generate the target rendered image.

[0072] Another alternative approach is to use depth prediction and deferred rendering techniques. By pre-calculating the depth once, the most visible primitives at the foreground of each pixel in the rendered image are determined. Then, for the insertion position of the target visual element, the blending weights are dynamically adjusted using its depth information, so that the visual element can be correctly synthesized with the surrounding 3D Gaussian points.

[0073] In addition, an adaptive rendering strategy can be adopted based on the attribute information of each rendering unit in the rendering sequence. Specifically, the rendering parameters and processing methods can be dynamically adjusted based on the opacity of each 3D Gaussian point, the type of the target visual element, and other attributes. For example, for semi-transparent 3D Gaussian points, depth sorting and transparency (Alpha) mixing techniques can be used, and special shader programs can be used for the target visual elements, thereby obtaining high-quality target rendering images while ensuring rendering efficiency.

[0074] In this step, by performing unified rendering processing on the rendering sequence containing the target visual elements, the deep fusion of the target visual elements and the 3D Gaussian model in the rendering process is achieved. This allows the target visual elements to be processed with the 3D Gaussian points using the same rendering pipeline and parameters, avoiding the rendering quality loss and inefficiency caused by image overlay. Through unified rendering processing, the accurate matching of the target visual elements and the 3D scene in the depth dimension is ensured, allowing the target visual elements to naturally integrate into the depth hierarchy of the 3D scene and form the correct spatial relationship and visual effect with other elements in the 3D scene.

[0075] In the embodiments of this specification, by acquiring the rendering sequence of the 3D Gaussian model and the target visual elements, determining the depth gap based on the depth information of the 3D Gaussian points, and dynamically adding the target visual elements to the corresponding depth positions in the rendering sequence, the depth consistency fusion of the target visual elements and the 3D scene during the rendering process is achieved. By making the target visual elements naturally presented according to the depth hierarchy of the 3D scene, the depth misalignment and visual fragmentation problems caused by image overlay are avoided, ensuring the accurate position and hierarchical relationship of the target visual elements in 3D space. By directly adding the visual elements to the rendering sequence for unified rendering processing, the execution efficiency of the rendering process is improved, ensuring the accurate matching of the visual elements and the 3D Gaussian model in the depth dimension, making the rendering result present a more natural and coherent 3D interactive effect. Thus, the target visual elements can dynamically adjust their display positions according to changes in scene depth, improving the display quality of the target visual elements in the target rendering image of the 3D Gaussian model, as well as the rendering efficiency of the target rendering image.

[0076] In an optional embodiment of this specification, before obtaining the rendering sequence of the 3D Gaussian model, the method further includes: Obtain the model image of the 3D Gaussian model; Based on the model image of the 3D Gaussian model, determine the depth information of each 3D Gaussian point in the 3D Gaussian model; Based on the depth information of each 3D Gaussian point, determine the rendering priority of each 3D Gaussian point in the 3D Gaussian model; The rendering sequence of the 3D Gaussian model is determined by sorting the 3D Gaussian points based on rendering priority.

[0077] Model images are two-dimensional image data used to represent or generate three-dimensional Gaussian models. They can be images from a single viewpoint or collections of images from multiple viewpoints. Specifically, model images can include real-world scene photographs acquired by a camera, i.e., initial two-dimensional images, such as a set of object photographs taken from different angles; or virtual scene images generated by three-dimensional modeling software, i.e., initial rendered images, such as a snapshot of a three-dimensional scene output from a specific virtual camera viewpoint. Model images can be used to extract geometric, texture, and depth information of the scene, providing a basis for determining the properties of each three-dimensional Gaussian point in the three-dimensional Gaussian model. Each pixel in the model image can be mapped to one or more three-dimensional Gaussian points in three-dimensional space.

[0078] Rendering priority is a numerical indicator used to determine the order of 3D Gaussian points in the rendering sequence during the rendering process. Specifically, rendering priority is usually directly related to the depth information of 3D Gaussian points. The smaller the depth value of a 3D Gaussian point, the closer it is to the virtual camera of the current rendering viewpoint, and its corresponding rendering priority is usually higher, to ensure that it can correctly occlude points that are farther away in the final 2D image.

[0079] Furthermore, in addition to depth information, rendering priority can also take into account other properties of 3D Gaussian points, such as visibility information, i.e., opacity. For completely opaque points, their priority has a greater impact on occlusion relationships, while for semi-transparent points, they can be processed based on a more refined order to present the correct blending and overlay effects.

[0080] Sorting is a process in 3D rendering where Gaussian points are rearranged according to their rendering priority values. Sorting generates an ordered rendering sequence and determines the relative positions of Gaussian points within the sequence by comparing their rendering priority values. This ensures that higher-priority Gaussian points enter the rendering pipeline first, thus dominating blending and occlusion processing. Specifically, the sorting process can use ascending or descending order, but it is typically done in descending order of rendering priority values, placing higher-priority Gaussian points at the beginning of the rendering sequence.

[0081] In practical applications, obtaining the model image of a 3D Gaussian model can be achieved through various methods.

[0082] An alternative approach is to directly read pre-acquired model image files from a storage device. These image files can be real-world photographs taken with a camera or computer-generated rendered images.

[0083] Another alternative approach is to generate a model image from the 3D Gaussian model itself through real-time rendering, i.e., to quickly preview and render the model from the perspective of one or more virtual cameras, thereby obtaining a 2D image from the current perspective.

[0084] In addition, model images can be received from external data sources via a network interface, such as downloading view data associated with a 3D Gaussian model from a cloud server.

[0085] Once the model image of the 3D Gaussian model is obtained, the depth information of each 3D Gaussian point can be determined based on the model image.

[0086] One alternative approach is to analyze the model images using a pre-trained depth estimation model. Specifically, the depth estimation model can be a deep learning-based neural network that takes one or more 2D images as input and outputs a corresponding depth map, where the value of each pixel represents the distance of that point from the camera. Then, based on the mapping relationship between 3D Gaussian points and image pixels, the depth values ​​in the depth map are assigned to the corresponding 3D Gaussian points, thus determining the depth information of each 3D Gaussian point.

[0087] Another alternative approach is to obtain the depth value of each pixel directly during the rendering process if the model image is an initial rendered image obtained through rendering a 3D Gaussian model. This involves rendering a depth map and then mapping these depth values ​​back to the 3D Gaussian points to obtain the depth information of each 3D Gaussian point.

[0088] Furthermore, for a 3D Gaussian model reconstructed from multi-view images, the principle of triangulation can be used to calculate the spatial coordinates of each 3D Gaussian point through the geometric relationship of multiple views, and then calculate its depth value relative to a specified virtual camera.

[0089] Once the depth information of each 3D Gaussian point is determined, the rendering priority can be determined based on the depth information of each 3D Gaussian point.

[0090] Specifically, rendering priority is determined based on the depth information of each 3D Gaussian point. Depth information can be the primary criterion; that is, the smaller the depth value of a 3D Gaussian point, the closer it is to the virtual camera of the current rendering viewpoint, and therefore the higher its rendering priority, ensuring that nearby objects occlude distant objects. In practice, the reciprocal or negative value of the depth value can be used as the priority value, or the index of the sorted depth values ​​can be directly used as the priority.

[0091] Furthermore, priority can be adjusted by combining it with other attributes. For example, for 3D Gaussian points with opacity close to zero, their contribution to the final image is minimal, and their priority can be appropriately reduced to optimize rendering performance; for 3D Gaussian points designated as key regions by the user, their priority can be artificially increased to ensure rendering accuracy.

[0092] Furthermore, priority can be adjusted based on occlusion relationships. For example, if a 3D Gaussian point is located behind another 3D Gaussian point, meaning its normal direction is away from the camera, its priority can be appropriately reduced.

[0093] Once the rendering priority of each 3D Gaussian point is determined, the 3D Gaussian points can be sorted based on the rendering priority to determine the rendering sequence.

[0094] One alternative approach is to use standard sorting algorithms (such as quicksort, heapsort, etc.) to arrange all three-dimensional Gaussian points in descending order of priority. In actual sorting, since the number of three-dimensional Gaussian points can be enormous, parallel sorting algorithms can be prioritized, utilizing graphics processing units (GPUs) or parallel computing frameworks to accelerate the sorting process.

[0095] Another alternative approach is to directly construct the rendering sequence using depth information. For example, by maintaining a linked list segmented by depth, 3D Gaussian points can be inserted into the appropriate positions of the corresponding depth segments to achieve incremental sorting.

[0096] Furthermore, for dynamically changing scenarios, an adaptive sorting strategy can be adopted, which only performs local rearrangement on the parts with changing depth, while keeping the order of other parts unchanged, in order to improve sorting efficiency.

[0097] Furthermore, the sorting results can be further adjusted and optimized according to the needs of actual application scenarios. For example, for three-dimensional Gaussian points with the same rendering priority, they can be sorted a second time according to their position or other attributes in three-dimensional space to obtain a more reasonable rendering sequence.

[0098] In the embodiments of this specification, by acquiring the model image of a 3D Gaussian model, determining the depth information of each 3D Gaussian point based on the model image, determining the rendering priority based on the depth information, and sorting the 3D Gaussian points based on the rendering priority to determine the rendering sequence, the automatic generation of the 3D Gaussian model rendering sequence is realized. By fully utilizing the model image information of the 3D Gaussian model and accurately acquiring the depth information of each 3D Gaussian point using depth estimation technology, a reliable data foundation is provided for the generation of the rendering sequence. By determining the rendering priority based on the depth information, it is ensured that the arrangement order of each 3D Gaussian point in the rendering sequence conforms to the depth hierarchy relationship in 3D space, providing an accurate depth reference for the subsequent insertion of target visual elements. By arranging each 3D Gaussian point to generate the rendering sequence, the disordered point set is organized into an ordered rendering sequence, enabling the 3D Gaussian points to be processed according to the correct occlusion relationship during the rendering process, thereby improving the display quality of target visual elements in the target rendering image of the 3D Gaussian model and the rendering efficiency of the target rendering image.

[0099] In one optional embodiment of this specification, the model image includes an initial two-dimensional image; Based on the model image of the 3D Gaussian model, determine the depth information of each 3D Gaussian point in the 3D Gaussian model, including: Using a pre-trained depth estimation model, the depth of the initial two-dimensional image is estimated to obtain an initial depth map; Based on the initial depth map, the depth information of each 3D Gaussian point in the 3D Gaussian model is determined. The 3D Gaussian points are obtained by 3D mapping based on each initial pixel in the initial 2D image.

[0100] Initial 2D images are the raw 2D image data used to construct or represent 3D Gaussian models. They can be acquired through imaging devices or image acquisition systems, and may include real-world photographs taken from different angles, keyframes extracted from video sequences, and computer-generated virtual scene images. Initial 2D images can characterize the visual features of a 3D scene on a 2D plane, providing fundamental data for subsequent 3D model reconstruction and depth estimation. They typically contain visual information such as the appearance, texture, and color of objects in the scene. By analyzing this visual information, the 3D structure and spatial relationships of the scene can be determined, providing input data for the construction of the 3D Gaussian model.

[0101] A pre-trained depth estimation model is a machine learning or deep learning model trained on a large amount of training data. It can predict pixel depth information from a two-dimensional image and may include convolutional neural networks (CNNs), Transformer networks, or other types of neural network architectures. Pre-trained depth estimation models can be used to predict corresponding depth information from two-dimensional images, enabling the conversion from single or multiple two-dimensional images to depth maps. Specifically, during training, a pre-trained depth estimation model learns the mapping relationship between image features and depth information, and can quickly and accurately predict the depth distribution information of each pixel in a scene when given a new initial two-dimensional image.

[0102] Depth estimation is the process of determining the distance from each point in a scene to a virtual camera from an initial 2D image. It can include monocular image-based depth estimation, multi-view image-based depth estimation, and video sequence-based depth estimation. Depth estimation can be used to acquire 3D geometric information of a scene, providing spatial location references for tasks such as 3D reconstruction and scene understanding.

[0103] An initial depth map is a two-dimensional array of the same size as the initial two-dimensional image, generated after depth estimation of the initial two-dimensional image. The value of each pixel represents the distance (depth value) of the scene point corresponding to that pixel from the camera. Initial depth maps are typically presented as grayscale or pseudo-color images; brighter colors indicate closer distances, while darker colors indicate farther distances.

[0104] 3D mapping is the process of converting pixels in a 2D image into points in 3D space (i.e., 3D Gaussian points). By combining the 2D coordinates, depth values, and camera parameters of a pixel, its specific location in 3D space can be calculated. In constructing a 3D Gaussian model, feature points can be extracted from one or more initial 2D images, and an initial 3D point cloud can be generated using a 3D reconstruction algorithm. Each 3D point corresponds to pixels in multiple images. Further parameter optimization training of each 3D point yields the individual 3D Gaussian points.

[0105] In practical applications, the initial depth map can be obtained by using a pre-trained depth estimation model to estimate the depth of an initial two-dimensional image. This can be achieved through various methods.

[0106] One possible approach is to directly input the initial 2D image into a pre-trained depth estimation model, which outputs a corresponding initial depth map. Specifically, the pixel values ​​of the initial 2D image can be used as input, and the depth estimation model performs feature extraction and depth prediction through its internal neural network layers, ultimately outputting an initial depth map with the same size as the input image.

[0107] Another alternative approach is to utilize a multi-view stereo-based depth estimation model when multiple initial 2D images from different perspectives exist. This model can calculate a more accurate depth map by matching the pixel correspondences between different images. This process can combine a pre-trained feature extraction network and a stereo matching algorithm.

[0108] Optionally, the initial two-dimensional image can be preprocessed before being input into the depth estimation model. Specifically, preprocessing operations such as normalization, denoising, and enhancement can be performed on the initial two-dimensional image to improve the accuracy of depth estimation.

[0109] In addition, a multi-scale depth estimation strategy can be used. By scaling the initial two-dimensional image to different scales and inputting them into the depth estimation model, depth maps at multiple scales can be obtained. Then, the fused initial depth map can be obtained through upsampling and fusion operations, thereby improving the detail representation capability of depth estimation.

[0110] Once an initial depth map is obtained, the depth information of each 3D Gaussian point in the 3D Gaussian model can be determined based on the initial depth map.

[0111] Specifically, since 3D Gaussian points are typically reconstructed from feature points in an initial 2D image through 3D mapping and parameter optimization, each 3D Gaussian point corresponds to its initial pixel in the initial 2D image. Therefore, for each 3D Gaussian point, the depth value of its corresponding pixel location can be found in the initial depth map, and this depth value can be used as the depth information of that 3D Gaussian point. When a 3D Gaussian point corresponds to multiple pixels in multiple images, the depth values ​​of these pixels can be combined, for example, by taking the average or median, to obtain a more accurate depth estimate.

[0112] Furthermore, the depth information of each 3D Gaussian point can be determined through back projection calculation based on the initial depth map and camera parameters. Specifically, the 3D spatial coordinates corresponding to each pixel can be calculated based on the depth values ​​of each pixel in the initial depth map and the camera's intrinsic parameter matrix. Then, the depth information of each 3D Gaussian point can be determined based on the correspondence between the 3D Gaussian point and these 3D spatial coordinates.

[0113] In the embodiments of this specification, an initial depth map is obtained by using a pre-trained depth estimation model to estimate the depth of an initial two-dimensional image. Then, the depth information of each three-dimensional Gaussian point in the three-dimensional Gaussian model is determined based on the initial depth map. This achieves automatic conversion from two-dimensional image to three-dimensional depth information. By fully utilizing the advantages of deep learning technology in depth estimation, the pre-trained depth estimation model can accurately predict the depth distribution of the scene from the initial two-dimensional image, providing a reliable depth information foundation for the construction of the three-dimensional Gaussian model. By determining the depth information of each three-dimensional Gaussian point based on the initial depth map, it is ensured that the depth values ​​of each point in the three-dimensional Gaussian model are consistent with the spatial structure of the real scene. This provides an accurate depth reference for subsequent rendering sequence generation and target visual element insertion, achieving effective conversion from two-dimensional image information to three-dimensional spatial information. This improves the automation and accuracy of three-dimensional Gaussian model construction, thus providing a foundation for obtaining high-quality target rendering images.

[0114] In one optional embodiment of this specification, the model image includes an initial rendered image; Based on the model image of the 3D Gaussian model, determine the depth information of each 3D Gaussian point in the 3D Gaussian model, including: Based on the initial rendering viewpoint of the initial rendering image, the pixel neighborhood of each rendering pixel in the initial rendering image is traversed to determine the rendering depth map of the 3D Gaussian model. Based on the rendered depth map, the depth information of each 3D Gaussian point in the 3D Gaussian model is determined.

[0115] The initial rendered image is two-dimensional image data obtained through preliminary rendering operations on a 3D Gaussian model. It can include images obtained from a single rendering of the 3D Gaussian model from a specific virtual camera perspective, or a collection of images obtained from rendering the 3D Gaussian model from multiple different perspectives. The initial rendered image can characterize the appearance features and spatial structure information of the 3D Gaussian model from a specific perspective, providing a visual reference for subsequent depth information extraction and rendering sequence generation. Specifically, the initial rendered image typically contains the projection results of each 3D Gaussian point in the 3D Gaussian model onto a two-dimensional imaging plane. Each pixel in the image can correspond to the rendering contribution of one or more 3D Gaussian points in three-dimensional space. By analyzing the initial rendered image, the geometric features and appearance attributes of the 3D Gaussian model can be extracted.

[0116] The initial rendering viewpoint refers to the position and orientation parameters of the virtual camera in 3D space when generating the initial rendered image. It can include the virtual camera's position coordinates, viewing direction, and field of view angle. The initial rendering viewpoint determines the angle from which the 3D Gaussian model is viewed and which parts of the model appear in the rendered image. The initial rendering viewpoint can dynamically change based on user interaction or preset animation paths. In determining depth information, the initial rendering viewpoint serves as a reference for calculating the depth values ​​corresponding to each pixel, in order to calculate the distance from the virtual camera's position to scene points from that viewpoint.

[0117] A pixel neighborhood is a local region surrounding a central pixel in the initial rendered image. Its size can be set as needed; for example, it can be the area of ​​a single pixel surrounding that pixel, a 3×3 or 5×5 rectangular region, or a neighborhood of other shapes (such as a cross or a circle). Pixel neighborhoods can be used to analyze the features and trends of local regions in an image, providing local contextual information for image processing tasks such as edge detection, texture analysis, and depth estimation. During the determination of the rendered depth map, by traversing the neighborhood of each pixel, the depth difference between that pixel and its surrounding pixels can be analyzed, thereby determining whether the depth is continuous or has abrupt changes.

[0118] A rendering depth map is a two-dimensional image or data array that corresponds to the initial rendered image and records the depth value of each pixel. The rendering depth map typically has the same resolution as the initial rendered image, and the value of each pixel represents the distance from the camera position in the initial rendering view to the surface of the corresponding 3D Gaussian model. The rendering depth map can be generated during the rendering of the initial rendered image and directly reflects the spatial depth distribution of the 3D Gaussian model in the initial rendering view.

[0119] In practical applications, based on the initial rendering viewpoint of the initial rendered image, the pixel neighborhood of each rendered pixel in the initial rendered image is traversed to determine the rendering depth map of the 3D Gaussian model.

[0120] Specifically, while rendering the initial render image, the depth value corresponding to each pixel can be directly obtained using the depth buffer mechanism of the rendering pipeline, thereby generating a render depth map. In the rasterization-based rendering process, each 3D Gaussian point can cover a certain pixel area after projection. For each pixel, the depth of the 3D Gaussian point with the smallest depth value, that is, the depth closest to the camera, can be recorded to obtain the render depth map.

[0121] Another alternative approach is to perform depth back-projection calculations based on the initial rendered image and the initial rendered viewpoint. Specifically, based on the camera parameters of the initial rendered viewpoint and the position information of each rendered pixel in the initial rendered image, the 3D spatial position corresponding to each rendered pixel can be determined through back-projection calculations. Then, the distance from these 3D spatial positions to the virtual camera can be calculated as the depth value to generate a rendered depth map.

[0122] Given a rendering depth map of a 3D Gaussian model, the depth information of each 3D Gaussian point in the model can be determined based on the rendering depth map.

[0123] One alternative approach is to directly extract depth information from the rendering depth map based on the correspondence between 3D Gaussian points and rendered pixels. Specifically, during the rendering process, the 3D Gaussian points contributed by each pixel can be recorded. When the rendering pipeline determines the color for each pixel, the contributions of multiple 3D Gaussian points can be considered. To determine depth information, the identifier of the nearest 3D Gaussian point can usually be recorded. Based on this mapping relationship, the depth value of each pixel in the rendering depth map can be added to the corresponding 3D Gaussian point.

[0124] Another alternative approach is to perform interpolation calculations based on the rendered depth map and the spatial location of the 3D Gaussian point. For a 3D Gaussian point that does not have a direct corresponding pixel in the rendered image in the 3D Gaussian model, interpolation calculations can be performed based on the depth values ​​of the 3D Gaussian points with existing depth information around it, thereby obtaining the depth information of the 3D Gaussian point.

[0125] Another alternative method is to first transform the 3D coordinates of each 3D Gaussian point to the camera coordinate system based on the virtual camera parameters of the initial rendering viewpoint, and take its vertical distance coordinate value as the initial depth information of the point, and then further adjust the initial depth information based on the rendering depth map.

[0126] Furthermore, for rendering depth maps obtained from multiple different initial rendering perspectives, the depth information of each 3D Gaussian point can be determined through multi-view depth fusion. Specifically, the depth information in each rendering depth map can be weighted and averaged or the optimal value can be selected to obtain more reliable depth estimation results.

[0127] In the embodiments of this specification, the rendering depth map of the 3D Gaussian model is determined by traversing the pixel neighborhood of each rendered pixel in the initial rendered image based on the initial rendering viewpoint. Then, the depth information of each 3D Gaussian point in the 3D Gaussian model is determined based on the rendering depth map. This fully utilizes the depth calculation results during the rendering process of the 3D Gaussian model. The rendering depth map is obtained by directly acquiring or reverse calculating, providing an accurate data foundation for determining the depth information of the 3D Gaussian points. By determining the depth information of each 3D Gaussian point based on the rendering depth map, it is ensured that the depth information is consistent with the actual rendering result of the 3D Gaussian model. Through the traversal analysis of the pixel neighborhood, noise and discontinuous regions in the rendering depth map can be processed, improving the robustness and accuracy of depth information extraction. This provides a reliable depth reference for rendering sequence generation and target visual element insertion, thus providing a foundation for obtaining a high-quality target rendered image.

[0128] In one optional embodiment of this specification, the rendering priority of each 3D Gaussian point in the 3D Gaussian model is determined based on the depth information of each 3D Gaussian point, including: Based on the depth information of each 3D Gaussian point, determine the rendering parameters of each 3D Gaussian point in the 3D Gaussian model; Based on the rendering parameters of each 3D Gaussian point, the rendering priority of each 3D Gaussian point in the 3D Gaussian model is determined.

[0129] Rendering parameters are a set of attribute indicators used to control the rendering effect and processing order of each 3D Gaussian point during the rendering process of a 3D Gaussian model. Specifically, they can include attributes such as the position coordinates, size, opacity, spherical harmonic coefficient, depth, normal direction, and rendering weight of the 3D Gaussian point. Rendering parameters determine the requirements for rendering tasks such as the brightness and darkness representation of the 3D Gaussian point, occlusion handling, and blending weight allocation, directly affecting the visual effect and quality of the final rendered image.

[0130] For example, the depth value in the rendering parameters can determine the distance of the 3D Gaussian point from the virtual camera, thus affecting its arrangement order in the rendering sequence; the opacity can determine the degree of occlusion of the 3D Gaussian point on the background, affecting the presentation of the semi-transparent effect; the visibility information can characterize the visibility of the 3D Gaussian point in the current rendering viewpoint, which can optimize rendering performance; the normal direction can determine the lighting effect and brightness of the 3D Gaussian point; and the rendering weight can control the contribution ratio of the 3D Gaussian point to pixel color calculation in the rendered image, etc.

[0131] In practical applications, the rendering parameters of each 3D Gaussian point in a 3D Gaussian model can be determined based on the depth information of each 3D Gaussian point through various methods.

[0132] One alternative approach is to use depth information as the basis for rendering parameters. Specifically, the depth values ​​of each 3D Gaussian point can be used as the depth component in the rendering parameters, combined with other inherent properties of the 3D Gaussian point, such as opacity and spherical harmonics, to form a complete set of rendering parameters.

[0133] Another alternative approach is to adjust other properties of the 3D Gaussian points based on depth information to generate new rendering parameters. For example, the opacity of the 3D Gaussian points can be adjusted based on the depth value, making points farther from the virtual camera more transparent to simulate atmospheric perspective; or the color intensity of the 3D Gaussian points can be adjusted based on the depth value, making distant points lighter to enhance the sense of depth in the scene. The specific adjustments can be flexibly made according to the actual rendering task.

[0134] In addition, depth information can be used to calculate the relative occlusion relationship between three-dimensional Gaussian points, generating an occlusion coefficient for each three-dimensional Gaussian point to reflect the degree to which the point is occluded by other points, which can also be used as part of the rendering parameters.

[0135] Once the rendering parameters for each 3D Gaussian point are determined, the rendering priority of each 3D Gaussian point can be determined based on the rendering parameters.

[0136] One alternative approach is to use the depth component of the rendering parameters as the primary basis. Specifically, 3D Gaussian points with smaller depth values ​​can be assigned higher rendering priority to ensure that points closer to the virtual camera are rendered first, while points farther away are correctly occluded. This establishes an initial rendering priority, which is then adjusted based on other parameters to determine the sorting adjustments and refine the initial rendering priority.

[0137] Another alternative approach is to determine rendering priority based on a weighted average of multiple components in the rendering parameters. Specifically, parameters such as depth, opacity, and visibility can be assigned corresponding priority weights, and then further weighted and combined to calculate a comprehensive rendering priority. For example, a completely opaque 3D Gaussian point can be given a higher rendering priority, while a semi-transparent 3D Gaussian point can have its rendering priority appropriately reduced to optimize blending effects. Furthermore, the weight coefficients of each parameter can be dynamically adjusted according to the needs of the application scenario to achieve the optimal rendering effect.

[0138] Another alternative approach is to perform layered processing based on rendering parameters. Specifically, the 3D Gaussian points can be divided into multiple different rendering layers based on depth parameters, such as different depth layers, different lighting layers, and different visibility layers. Then, within each layer, further sorting can be performed based on other rendering parameters to obtain a more refined rendering priority allocation.

[0139] In the embodiments of this specification, the rendering parameters of each 3D Gaussian point in the 3D Gaussian model are determined based on the depth information of each 3D Gaussian point, and then the rendering priority of each 3D Gaussian point in the 3D Gaussian model is determined based on the rendering parameters of each 3D Gaussian point, thus realizing the refined and adaptive determination of the rendering priority. Through multi-dimensional parameter combination and calculation, a more comprehensive and accurate data foundation is provided for the determination of the rendering priority. Determining the rendering priority based on the rendering parameters ensures that the arrangement order of each 3D Gaussian point in the rendering sequence not only conforms to the depth hierarchy relationship, but also comprehensively considers other important factors such as opacity and visibility. This enables the rendering process to handle occlusion relationships and blending effects more accurately, providing a reliable depth reference and rendering order basis for the subsequent addition of target visual elements. This ensures that the target visual elements can be blended with the 3D Gaussian model at the correct depth hierarchy, thereby improving the display quality of the target visual elements in the target rendered image of the 3D Gaussian model and the rendering efficiency of the target rendered image.

[0140] In one optional embodiment of this specification, determining at least one depth gap in a three-dimensional Gaussian model based on depth information from multiple three-dimensional Gaussian points includes: Based on the depth information of multiple 3D Gaussian points, the 3D Gaussian model is divided to obtain multiple 3D Gaussian point depth intervals. Based on the depth information of the three-dimensional Gaussian points in the depth range of each three-dimensional Gaussian point, at least one depth gap in the three-dimensional Gaussian model is determined.

[0141] Partitioning, in the processing of a 3D Gaussian model, is the process of segmenting the entire 3D Gaussian model according to its depth dimension, based on the depth information of multiple 3D Gaussian points. Partitioning can allocate all 3D Gaussian points in the model to different depth levels or ranges according to their depth values, thus transforming the originally continuous depth distribution into a discrete set of depth intervals. Specifically, partitioning can be based on a preset step size for uniform division, such as dividing the depth range from the minimum to the maximum value into several equal segments; it can also be based on the statistical distribution of depth values ​​for adaptive partitioning, such as ensuring that each interval contains approximately an equal number of 3D Gaussian points; or it can be combined with other properties of the 3D Gaussian model, such as partitioning based on the natural layering of objects in the scene.

[0142] A depth interval is a range of depths containing a certain number of 3D Gaussian points, obtained by dividing a 3D Gaussian model. Each depth interval corresponds to a specific depth level or range within the 3D Gaussian model. A depth interval characterizes the set of 3D Gaussian points within a certain depth range of the 3D Gaussian model, reflecting the distribution density and depth characteristics of these points. Depth intervals can be used to statistically analyze the distribution of 3D Gaussian points at different depth levels, providing a data foundation for identifying depth gaps. Specifically, a depth interval is typically defined by the minimum and maximum depth values ​​of all 3D Gaussian points within that interval. The 3D Gaussian points within the interval are arranged in order of their depth values. There may be depth overlap or depth gaps between adjacent depth intervals. The number of depth intervals and the depth range of each interval can be flexibly adjusted according to the needs of the actual application scenario.

[0143] In practical applications, the 3D Gaussian model can be divided based on the depth information of multiple 3D Gaussian points to obtain multiple 3D Gaussian point depth intervals, which can be achieved through various methods.

[0144] One alternative approach is to use a constant depth step size method. Specifically, the minimum and maximum depth values ​​of all 3D Gaussian points in the 3D Gaussian model can be determined first, the total depth range can be calculated, and then the total depth range can be divided into several consecutive depth intervals according to a preset fixed step size. Each interval has the same depth width, and all 3D Gaussian points whose depth values ​​fall within the interval are assigned to the corresponding depth interval.

[0145] Another alternative method is to use the equal-number division method. Specifically, all three-dimensional Gaussian points can be sorted in ascending order of depth value. Then, based on the preset number of depth intervals, the sorted three-dimensional Gaussian point sequence can be divided into multiple subsequences, with each subsequence containing approximately the same number of three-dimensional Gaussian points, and each subsequence corresponding to a depth interval.

[0146] Another alternative approach is to use density-based clustering. Specifically, by analyzing the distribution density of three-dimensional Gaussian points in the depth dimension, continuous regions with high density can be divided into intervals, while regions with low density may form the boundaries between intervals.

[0147] In addition, a depth gradient-based partitioning method can be used. Specifically, the depth difference between adjacent 3D Gaussian points can be calculated. If the depth difference exceeds a preset threshold, the location can be used as the partition boundary to generate a depth interval.

[0148] After obtaining multiple 3D Gaussian point depth intervals, at least one depth gap in the 3D Gaussian model can be determined based on the depth information of the 3D Gaussian points in each 3D Gaussian point depth interval.

[0149] One alternative approach is based on statistical analysis of the number of three-dimensional Gaussian points within a depth interval. Specifically, the number of three-dimensional Gaussian points contained in each depth interval can be counted, and the gradient of the change in the number of three-dimensional Gaussian points between adjacent depth intervals can be calculated. When the gradient exceeds a preset gradient threshold, that location can be identified as a depth gap. That is, a depth gap can correspond to a region where the number of three-dimensional Gaussian points is significantly reduced.

[0150] Another alternative approach is to use comparative analysis based on depth values ​​at the boundaries of depth intervals. Specifically, the depth interval between adjacent depth intervals can be calculated, which is the difference between the minimum depth value of the later depth interval and the maximum depth value of the earlier depth interval. When this difference exceeds a preset depth interval threshold, the location can be identified as a depth gap. In other words, a depth gap can correspond to a significant jump in depth values.

[0151] Another alternative approach is to use variance analysis based on the depth distribution of three-dimensional Gaussian points within a depth interval. Specifically, the variance or standard deviation of the depth values ​​of three-dimensional Gaussian points within each depth interval can be calculated. When the variance of a certain depth interval is significantly less than the variance of adjacent depth intervals, that depth interval can be defined as a depth gap.

[0152] In addition, a multi-indicator comprehensive judgment method can be adopted. Specifically, multiple indicators such as the number of three-dimensional Gaussian points within the depth interval, the difference in depth values ​​at the depth interval boundaries, and the variance of the depth distribution within the depth interval can be considered simultaneously. By weighted combination or logical judgment, the location of the depth gap can be comprehensively determined, thereby improving the accuracy and robustness of depth gap identification.

[0153] In this embodiment, a 3D Gaussian model is divided into multiple 3D Gaussian point depth intervals based on the depth information of multiple 3D Gaussian points. Then, at least one depth gap in the 3D Gaussian model is determined based on the depth information of the 3D Gaussian points in each depth interval, thus realizing the systematic and structured identification of depth gaps. By dividing the 3D Gaussian model into depth dimensions, the continuous depth distribution is transformed into a discrete set of depth intervals, providing a data organization structure for the identification of depth gaps. By identifying depth gaps through depth intervals, the statistical characteristics and distribution patterns of 3D Gaussian points within the depth intervals can be fully utilized to accurately determine the regions of depth discontinuity in the 3D Gaussian model. This provides a clear depth position reference for the subsequent insertion of target visual elements, ensuring that target visual elements can be accurately inserted into the naturally existing spatial gaps in the 3D scene. This improves the display quality of target visual elements in the target rendering image of the 3D Gaussian model and the rendering efficiency of the target rendering image.

[0154] In one optional embodiment of this specification, determining at least one depth gap in the three-dimensional Gaussian model based on the depth information of the three-dimensional Gaussian points in each three-dimensional Gaussian point depth interval includes: Based on the depth information of the three-dimensional Gaussian points in each depth interval, the depth change gradient between each depth interval of the three-dimensional Gaussian points is determined. Based on the depth change gradient and the preset depth change threshold, the locations of steep depth descent and steep depth ascent are determined; Based on the steep descent position and the steep depth rise position, at least one depth gap in the three-dimensional Gaussian model is determined.

[0155] The depth gradient is a numerical indicator used to measure the degree of depth value change between adjacent depth intervals in a 3D Gaussian model. It characterizes the rate or magnitude of depth value change at the boundary of a depth interval. The depth gradient can be used to identify abrupt changes in depth distribution within a 3D Gaussian model, providing a quantitative basis for detecting depth gaps. Specifically, the depth gradient is obtained by calculating the depth difference between adjacent depth intervals and dividing by the number of intervals or the depth range. A larger gradient value indicates a more drastic depth change, while a smaller gradient value indicates a more gradual depth change. By analyzing the distribution characteristics of the depth gradient, locations where significant depth value changes can be accurately identified, thus providing a reference for locating depth gaps.

[0156] The depth change threshold is a pre-set critical value used to determine whether the depth change gradient reaches a significant level of change. It can serve as a criterion for distinguishing between normal and abnormal depth changes. The depth change threshold can filter out areas with small depth changes, preventing normal depth gradients from being misjudged as depth gaps. Specifically, the depth change threshold can be set based on the depth distribution characteristics of the 3D Gaussian model or the scene complexity, and can be dynamically adjusted based on actual application requirements. If the calculated depth change gradient exceeds the threshold, a significant depth change is considered to exist at that location, and further analysis is needed to determine if it is a depth gap. If the depth change gradient does not exceed the threshold, the depth change at that location is considered to be within the normal range and requires no special processing.

[0157] A depth drop point is a specific location in a 3D Gaussian model where the depth value decreases significantly, and can represent the boundary point where the depth value transitions from a closer depth region to a farther depth region; a depth rise point is a specific location in a 3D Gaussian model where the depth value increases significantly, and can represent the boundary point where the depth value transitions from a farther depth region to a closer depth region.

[0158] Depth descent and depth ascent locations can be used to identify the boundary regions between foreground and background objects in a 3D scene, providing depth references for the insertion of target visual elements. Specifically, depth descent locations typically correspond to positions where the depth gradient is negative and its absolute value exceeds a depth change threshold. The depth value of the preceding depth interval is significantly smaller than the depth value of the following depth interval, indicating a sudden increase in the depth of the 3D Gaussian point at that location, forming a "downhill" feature in depth. Depth ascent locations typically correspond to positions where the depth gradient is positive and exceeds a depth change threshold. The depth value of the preceding depth interval is significantly larger than the depth value of the following depth interval, indicating a sudden decrease in the depth of the 3D Gaussian point at that location, forming an "uphill" feature in depth.

[0159] In practical applications, the depth change gradient between different depth intervals of three-dimensional Gaussian points can be determined based on the depth information of the three-dimensional Gaussian points within each depth interval, which can be achieved through various methods.

[0160] One possible approach is to calculate the difference between the average depth values ​​of adjacent depth intervals as the depth change gradient. Specifically, you can first calculate the average depth value of all three-dimensional Gaussian points in each depth interval, and then calculate the difference between the average depth values ​​of adjacent depth intervals. This difference is the depth change gradient.

[0161] Another alternative approach is to calculate the difference between the depth values ​​at the boundaries of adjacent depth intervals as the depth change gradient. Specifically, the difference between the maximum depth value of the previous depth interval and the minimum depth value of the next depth interval can be used as the depth change gradient.

[0162] Another alternative method is to calculate the depth change gradient using a weighted average. Specifically, the depth values ​​can be weighted according to the number of three-dimensional Gaussian points in each depth interval, and then the weighted depth change gradient can be calculated to better reflect the actual depth distribution characteristics of the depth interval.

[0163] In addition, a sliding window approach can be used to calculate the depth gradient. Specifically, a sliding window containing multiple continuous depth intervals can be set, and the windows can overlap. The overall trend of the depth values ​​within the window is calculated as the depth gradient, thereby obtaining a smoother gradient calculation result.

[0164] After obtaining the depth change gradient, the locations of steep depth drops and steep depth rises can be determined based on the depth change gradient and the preset depth change threshold.

[0165] One alternative method is the direct comparison method. Specifically, each calculated depth change gradient can be compared with a preset depth change threshold. When the depth change gradient is positive and greater than the depth change threshold, the position is determined as a depth steep rise position; when the depth change gradient is negative and its absolute value is greater than the depth change threshold, the position is determined as a depth steep fall position.

[0166] Another alternative approach is to use a dual-threshold method. Specifically, a positive threshold and a negative threshold can be set. When the depth change gradient is greater than the positive threshold, it is determined as a location with a steep depth increase; when the depth change gradient is less than the negative threshold, it is determined as a location with a steep depth decrease. This method can more accurately distinguish depth changes in different directions.

[0167] Another option is to use the continuity verification method. Specifically, after initially determining the location of a steep depth drop or rise, it is necessary to further verify whether the depth change gradient within a certain range before and after the location maintains the same trend. Only when the trend is continuous can the location be finally confirmed as a steep depth drop or rise location to avoid misjudgment.

[0168] In addition, statistical analysis can be used. Specifically, statistical analysis can be performed on all depth change gradients to calculate their mean and standard deviation. Then, the depth change threshold can be dynamically determined by adding or subtracting a certain number of times the standard deviation from the mean, thereby adapting to the depth distribution characteristics under different scenarios.

[0169] Once the locations of steep depth drops and steep depth rises are determined, at least one depth gap in the three-dimensional Gaussian model can be determined based on these locations.

[0170] One alternative approach is to identify pairs of depth drops and rises. Specifically, when a depth drop is adjacent to a depth rise, the depth range between these two locations can be identified as a depth gap. For example, if there is a depth drop between the k-th and (k+1)-th depth intervals, and a depth rise between the (k+1)-th and (k+2)-th intervals, then the depth range corresponding to the (k+1)-th interval can be determined as a depth gap.

[0171] Another alternative approach is to combine multiple consecutive steep drops or rises in depth. Specifically, when a continuous region exhibits a depth gradient that consistently deviates significantly from zero, the start and end points of that region can be defined as a depth gap.

[0172] In addition, the presence of three-dimensional Gaussian points within the depth interval can be considered. If the number of points in a certain depth interval is extremely small or zero, and there are steep transition boundaries in the intervals before and after it, then the interval can be directly identified as a depth gap.

[0173] In an alternative embodiment, for example, depth information of three-dimensional Gaussian points can be determined based on an initial two-dimensional image, and at least one depth gap can be further determined.

[0174] Specifically, the step size for dividing the 3D Gaussian model can be determined based on the depth information of each 3D Gaussian point. And determine the total depth steps of the entire 3D Gaussian model. , that is, a total of A three-dimensional Gaussian point depth range.

[0175] For each depth range The number of three-dimensional Gaussian points included within its range is .

[0176] Then calculate the three-dimensional Gaussian points for each depth interval. Depth variation gradient , can be represented as:

[0177] for Above the depth change threshold That is, a location where the depth drops sharply. For depth changes below the threshold The location is where the depth rises sharply. If it exists... For a steep drop-off position and If the depth increases sharply, then it can be recorded. There is a depth gap between the depth intervals within the region.

[0178] Furthermore, the depth intervals within the region can be traversed in depth order. Calculate the depth difference between all pixels. If the depth difference between all pixels Maximum not exceeding Then the midpoint of that area can be selected. As a depth gap, otherwise depth difference can be used. The largest position is used as the depth gap.

[0179] For example, in another embodiment, the depth information of the three-dimensional Gaussian points can be determined based on the initial rendered image from the initial rendering view, and at least one depth gap can be further determined.

[0180] Specifically, it is possible to iterate through all pixels in the initial rendered image and calculate the position of each pixel. Single-domain depth difference:

[0181] Determine the location of the edge fault where the depth changes abruptly:

[0182] in, and It is a fixed empirical coefficient.

[0183] At the location where the depth changes abruptly, determine the direction of the normal to the depth distribution:

[0184]

[0185] Depth values ​​at both ends of the fault are obtained along the depth gradient direction. In actual calculations, a sampling radius can be specified. During the sampling process, it is possible to sample along the direction of the depth-varying gradient at intervals. Each pixel is sampled. Simultaneously, the area near the sampling point can be... A sampling window of a certain size is used to calculate the average or median depth values ​​to avoid blurred depth edges or local abrupt changes. The depth values ​​at the near end can then be recorded. The depth value at the far end is .

[0186] Furthermore, foreground and background separation can be performed on a 3D Gaussian model.

[0187] Specifically, it can be based on the depth value of the near end. and the depth value at the far end Build a deep set and ,in, This represents the threshold for the minimum depth difference, i.e., the preset depth change threshold.

[0188] Compute depth set The acceptable depth range of the gap for each 3D Gaussian point:

[0189] in, and It is the preset offset coefficient for depth distance.

[0190] Then, according to the given step size, traverse... and will fall within the range The location of the most numerous 3D Gaussian point is used as the depth segmentation location, and the two sides of the depth value corresponding to the 3D Gaussian point are marked as the foreground and background.

[0191] Furthermore, when it is necessary to determine multiple depth gaps, the range of depth gaps can be defined. By merging overlapping depth gaps, i.e. taking the union of the ranges, all non-overlapping areas can be identified as multiple depth gaps.

[0192] In the embodiments of this specification, the depth change gradient between the depth intervals of each 3D Gaussian point is determined based on the depth information of the 3D Gaussian points within each depth interval. The depth drop and rise positions are determined based on the depth change gradient and a preset depth change threshold. At least one depth gap in the 3D Gaussian model is then determined based on the drop and rise positions. This achieves accurate identification and location of depth gaps, transforming the problem of depth gap identification into the problem of depth change feature detection, making the depth gap identification process more quantifiable and objective. By setting a depth change threshold, normal depth gradient regions can be effectively filtered out, avoiding false judgments and improving depth accuracy. The accuracy of depth gap recognition is improved. By pairing and analyzing depth drop and rise locations, boundary regions where depth values ​​change abruptly can be accurately captured, thus precisely locating the depth gap. By adjusting the depth change threshold, the recognition requirements of depth gaps in different scenarios can be adapted, improving the flexibility and adaptability of depth gap determination. This provides reliable technical support for the accurate positioning of target visual elements in 3D scenes, ensuring that target visual elements can naturally integrate into the depth hierarchy of the 3D scene. This improves the display quality of target visual elements in the target rendering image of the 3D Gaussian model and the rendering efficiency of the target rendering image.

[0193] In one optional embodiment of this specification, rendering the rendering sequence to obtain the target rendered image includes: Render each 3D Gaussian point and target visual element in the rendering sequence to obtain the Gaussian point layer of each 3D Gaussian point and the visual element layer of the target visual element. Based on the sequence positions of each 3D Gaussian point and the target visual element in the rendering sequence, the Gaussian point layer and the visual element layer are overlaid to obtain the target rendered image.

[0194] The Gaussian point layer for each 3D Gaussian point is a 2D image layer generated by rendering each 3D Gaussian point in the rendering sequence. Specifically, it can include visual feature data such as the projection of each 3D Gaussian point onto the 2D imaging plane, color information, and transparency information. The Gaussian point layer for each 3D Gaussian point can characterize the spatial position, shape features, and visual attributes of each 3D Gaussian point during the rendering process from a specific rendering viewpoint. Specifically, the Gaussian point layer for each 3D Gaussian point typically projects the 3D Gaussian point onto the imaging plane of a virtual camera and determines its color contribution to the pixel based on the attribute parameters of the 3D Gaussian point.

[0195] The visual element layer of a target visual element is a two-dimensional image layer generated after rendering the target visual element in the rendering sequence. Specifically, it can include visual feature data such as the projection result of the target visual element onto the two-dimensional imaging plane, color information, transparency information, and shape contour. The visual element layer of a target visual element can characterize the embedding effect of the target visual element in a three-dimensional scene, reflecting the appearance and spatial relationship of the target visual element at a specific depth position. Specifically, the visual element layer of a target visual element is typically determined by projecting the target visual element onto the imaging plane of a virtual camera according to its depth position in the rendering sequence, and calculating its color contribution at each pixel based on the attribute parameters of the target visual element.

[0196] Sequence position refers to the relative positional information of each rendering unit (including 3D Gaussian points and target visual elements) in a rendering sequence, determined according to specific arrangement rules. Specifically, it can include spatial relationship data such as the index number, arrangement order, and depth level of each rendering unit in the rendering sequence. Sequence position can characterize the depth order and occlusion relationship of each rendering unit in 3D space, providing a basis for layer stacking during the rendering process. Specifically, sequence position can be determined based on parameters such as the depth information and rendering priority of each rendering unit. Rendering units with smaller depth values ​​are usually placed earlier in the sequence position to ensure that they can correctly occlude rendering units with larger depth values ​​during the rendering process.

[0197] Overlay is the process of merging multiple image layers according to a specific order and rules. This process can include graphics processing operations such as color blending, transparency calculation, and depth testing between layers. Overlay can be used to merge Gaussian point layers and visual element layers to generate the final target rendered image, achieving visual fusion and spatial relationship processing between layers. Specifically, the overlay process can calculate and blend the color and transparency values ​​of each layer sequentially according to the sequence position of each rendering unit in the rendering sequence, either from back to front or from front to back. For semi-transparent layers, alpha blending technology can be used; for opaque layers, they can be directly overlaid. Through overlay operations, the target visual elements and the 3D Gaussian model can be correctly fused in the depth dimension, generating a target rendered image with the correct spatial hierarchy.

[0198] In practical applications, rendering each 3D Gaussian point and target visual element in the rendering sequence, and obtaining the corresponding Gaussian point layer of each 3D Gaussian point and the visual element layer of the target visual element, can be achieved through various methods.

[0199] One alternative approach is to employ rasterization-based rendering techniques, treating each 3D Gaussian point and target visual element in the rendering sequence as independent rendering primitives. These primitives undergo vertex transformation, projection, rasterization, and fragment shading operations to generate corresponding Gaussian point layers and visual element layers. Specifically, for each 3D Gaussian point, its ellipsoidal or spherical Gaussian distribution can be projected onto the virtual camera's imaging plane. The coverage area on the pixel is determined based on the covariance matrix of the 3D Gaussian point. Then, the color contribution of each pixel is calculated based on the spherical harmonic coefficient and opacity to generate a Gaussian point layer. For the target visual element, it can be projected onto the imaging plane according to its depth position in the rendering sequence. The color contribution on the pixel is calculated based on the target visual element's attribute parameters to generate a visual element layer.

[0200] Another alternative approach is to use ray tracing-based rendering technology to emit rays from the virtual camera's perspective, perform intersection tests with each 3D Gaussian point and target visual element, calculate the color and lighting effects at each intersection point, and generate corresponding Gaussian point layers and visual element layers.

[0201] Another option is to use a layered rendering strategy. First, all 3D Gaussian points are rendered in batches to generate a Gaussian point layer. Then, the target visual elements are rendered separately to generate a visual element layer. By processing them separately, the rendering quality and performance of each layer can be better controlled.

[0202] In addition, GPU-based parallel rendering technology can be used to utilize the parallel computing capabilities of the graphics processor to simultaneously render multiple 3D Gaussian points and target visual elements, thereby improving the efficiency of layer generation.

[0203] Once you have a Gaussian point layer and a visual element layer, you can overlay these two layers based on the sequence positions of each 3D Gaussian point and the target visual element in the rendering sequence to obtain the target rendered image.

[0204] One alternative approach is to use a back-to-forward stacking order, following the sequence position of each rendering unit in the rendering sequence, starting with the rendering unit with the largest depth value, and sequentially stacking the corresponding layers onto the frame buffer. Specifically, the Gaussian point layer corresponding to the 3D Gaussian point with the largest depth value can be written to the frame buffer first, and then the Gaussian point layers with smaller depth values ​​and the visual element layers can be stacked sequentially according to their sequence positions. For semi-transparent layers, alpha blending technology is used for color mixing, and opaque layers are directly overlaid to finally generate the target rendered image.

[0205] Another alternative approach is to use a depth buffer technique. During the overlay process, a depth buffer is maintained to record the minimum depth value of each pixel. Color updates are only performed when the pixel depth value of the new layer is less than the depth value of the corresponding position in the depth buffer, thus ensuring the correct occlusion relationship.

[0206] Another option is to use a layered blending strategy. First, stack all opaque Gaussian point layers in depth order to generate a base layer. Then, perform alpha blending on the semi-transparent Gaussian point layers and visual element layers in depth order. Finally, merge the blending result with the base layer to generate the target rendering image.

[0207] In addition, an adaptive overlay strategy can be adopted, which dynamically adjusts the overlay order and blending method according to the transparency attributes and depth relationships of each layer. For example, an overlay method can be used for completely opaque layers, and a weighted average method can be used for semi-transparent layers, thereby improving overlay efficiency while ensuring rendering quality.

[0208] In this embodiment, Gaussian point layers and visual element layers are obtained by rendering each 3D Gaussian point and target visual element in the rendering sequence. The Gaussian point layers and visual element layers are then overlaid based on the sequence positions of each 3D Gaussian point and target visual element in the rendering sequence to obtain the target rendered image. This achieves deep fusion and unified processing of the target visual element and the 3D Gaussian model during the rendering process. By rendering each 3D Gaussian point and target visual element as an independent layer, the system can flexibly control the rendering parameters and processing methods of each layer, providing a more refined data foundation for subsequent layer overlay. Overlaying the Gaussian point layers and visual element layers based on sequence positions ensures correct fusion of the target visual element and the 3D Gaussian model in the depth dimension, allowing the target visual element to naturally integrate into the depth hierarchy of the 3D scene and form correct spatial relationships and visual effects with other elements in the 3D scene. The layered rendering and overlay processing avoids the rendering quality loss and inefficiency problems caused by traditional image overlay methods, improving the execution efficiency of the entire rendering process and the display quality of the target rendered image.

[0209] In one optional embodiment of this specification, based on the sequence positions of each 3D Gaussian point and the target visual element in the rendering sequence, the Gaussian point layer and the visual element layer are overlaid to obtain the target rendered image, including: Based on the sequence positions of each 3D Gaussian point and the target visual element in the rendering sequence, the occlusion relationship between the Gaussian point layer and the visual element layer is determined. Based on the occlusion relationship and the visibility information of each 3D Gaussian point and the target visual element, the Gaussian point layer and the visual element layer are overlaid to obtain the target rendered image.

[0210] Occlusion relationships, in the 3D rendering process, refer to the spatial relationships in which different rendering units occlude or are occluded in 3D space. They characterize whether each 3D Gaussian point and target visual element can be observed from a specific viewpoint, and the degree to which it is occluded by other objects. Occlusion relationships can be used to determine the order and visibility rules of layers during rendering, providing a spatial hierarchy for layer stacking. Specifically, occlusion relationships are usually determined based on the depth information and spatial position of each rendering unit. For example, when the depth value of one rendering unit is less than that of another, the former will occlude the latter. Determining occlusion relationships ensures that in the final rendered image, objects closer to the virtual camera correctly occlude objects farther away, thus presenting a visual effect consistent with realistic 3D spatial perception.

[0211] Visibility information describes the visibility of 3D Gaussian points and target visual elements from the current rendering viewpoint, and the degree of visibility. Specifically, it can include parameters such as opacity, transparency, visibility flags, and occlusion status. Visibility information can be used to control the presentation effect and contribution of each rendering unit in the final image, providing a basis for decision-making regarding color blending and depth testing during layer overlay. Specifically, within visibility information, the opacity parameter determines the degree of occlusion a rendering unit places on the background; a completely opaque unit will completely occlude objects behind it, while a semi-transparent unit will blend with the background. The visibility flag indicates whether a rendering unit should participate in the rendering process of the current frame, thereby optimizing rendering performance. The occlusion status records whether a rendering unit is completely occluded by other objects; if completely occluded, the rendering process of that unit can be skipped.

[0212] In practical applications, the occlusion relationship between the Gaussian point layer and the visual element layer can be determined based on the sequence position of each 3D Gaussian point and the target visual element in the rendering sequence, which can be achieved through various methods.

[0213] One alternative approach is to directly compare the depth information of each rendering unit in the rendering sequence. Specifically, adjacent rendering units in the rendering sequence can be compared. If the depth value of the preceding rendering unit is less than the depth value of the following rendering unit, it is assumed that the preceding rendering unit will occlude the following rendering unit, thereby determining the occlusion relationship between the two.

[0214] Another alternative approach is to determine the order based on the sequence position of the rendering sequence. Specifically, since the rendering sequence is usually arranged in order of depth from near to far or from far to near, the occlusion relationship can be directly determined based on the order of the sequence positions, with rendering units earlier in the sequence occluding rendering units later in the sequence.

[0215] Another alternative approach is to perform occlusion testing based on a depth buffer. Specifically, a depth buffer is maintained during the rendering process to record the minimum depth value of each pixel. When a new rendering unit is processed, its depth value is compared with the depth value at the corresponding position in the depth buffer. If the depth value of the new unit is less than the depth value in the buffer, the unit is visible and will occlude the previous unit; otherwise, the unit is occluded.

[0216] In addition, spatial segmentation-based occlusion culling techniques can be used to divide the three-dimensional space into multiple regions and pre-calculate the occlusion relationships between each region, thereby quickly determining the occlusion relationships between different rendering units.

[0217] Once the occlusion relationship is determined, the Gaussian point layer and the visual element layer can be overlaid based on the occlusion relationship and the visibility information of each 3D Gaussian point and the target visual element to obtain the target rendered image.

[0218] One alternative approach is to employ a depth-based overlay strategy, where each rendering unit in the rendering sequence is overlaid sequentially, starting with the unit with the smallest depth value. For each rendering unit, the overlay method is determined based on the opacity parameter in its visibility information. If the unit is completely opaque, the pixel color at the corresponding position is directly overwritten. If the unit is semi-transparent, an alpha blending technique is used to perform a weighted average with the existing pixel color.

[0219] Another alternative approach is to employ a conditional overlay strategy based on occlusion relationships. This involves first determining which rendering units are visible based on occlusion, then only overlaying the visible units, skipping completely occluded units to improve efficiency. Specifically, this can be achieved by maintaining a visibility list that records all visible rendering units in the current frame, and then overlaying the visible units in depth order.

[0220] Another option is to use a layered blending strategy, first stacking all opaque rendering units in depth order to generate a base layer, then performing alpha blending on the semi-transparent rendering units from back to front in depth order, and finally merging the blending result with the base layer.

[0221] In addition, an adaptive overlay strategy can be adopted to dynamically adjust the overlay order and blending method according to the visibility information of each rendering unit. For example, units with high opacity can be processed first, while units with low opacity can be processed with a more refined blending algorithm, thereby improving overlay efficiency while ensuring rendering quality.

[0222] In the embodiments of this specification, the occlusion relationship between the Gaussian point layer and the visual element layer is determined based on the sequential position of each 3D Gaussian point and the target visual element in the rendering sequence. Based on the occlusion relationship and the visibility information of each 3D Gaussian point and the target visual element, the Gaussian point layer and the visual element layer are superimposed to obtain the target rendered image. This achieves precise spatial relationship processing and visual fusion between the target visual element and the 3D Gaussian model during the rendering process. By determining the occlusion relationship based on the sequential position, it is ensured that the spatial hierarchy relationship between each rendering unit can be accurately identified and processed. By combining visibility information for layer superposition, the actual spatial relationship between each rendering unit can be optimized. The visibility adjustment overlay strategy employs an efficient overlay method for completely opaque units and a refined blending algorithm for semi-transparent units, thereby optimizing rendering performance while ensuring rendering quality. Through the coordinated processing of occlusion relationships and visibility information, it ensures that target visual elements can be integrated with the 3D Gaussian model at the correct depth level, allowing target visual elements to naturally blend into the depth hierarchy of the 3D scene and form correct spatial relationships and visual effects with other elements in the 3D scene. This improves the display quality of target visual elements in the target rendering image of the 3D Gaussian model, as well as the rendering efficiency of the target rendering image.

[0223] In one optional embodiment of this specification, adding the target visual element to the rendering sequence based on the depth gap includes: Based on the depth gap, the depth information of the target visual element is determined; Based on the depth information of the target visual elements, the target visual elements are converted into renderable entities; Based on the depth gap and the depth information of the target visual elements, renderable entities are added to the rendering sequence.

[0224] Depth information of a target visual element describes its distance from the virtual camera's viewpoint in 3D space. It can be expressed as absolute depth, relative depth, or depth range. This depth information can be used to determine the rendering order and occlusion relationships of target visual elements, providing a spatial reference for their accurate positioning in the 3D scene. The depth information can also be determined based on depth reference values ​​from depth gaps, ensuring that the target visual element blends correctly with the 3D scene according to the appropriate depth hierarchy.

[0225] A renderable entity is a graphical object formed by transforming a target visual element according to its depth position and attribute parameters in three-dimensional space, and capable of participating in the 3D rendering pipeline. A renderable entity can include visual feature data such as the target visual element's position coordinates, depth information, color information, transparency information, and shape contour in three-dimensional space. It can be used to characterize the spatial position, shape features, and visual attributes of the target visual element from a specific rendering viewpoint. Specifically, the original two-dimensional form of the target visual element (such as bitmaps or vector graphics) or a simple 3D model needs to be converted into an entity with geometric definitions, material properties, and rendering parameters before it can participate in the unified rendering process of the rendering sequence. Converting a target visual element into a renderable entity allows the target visual element to be projected onto the imaging plane of a virtual camera according to its depth information, and its color contribution at each pixel is calculated based on the target visual element's attribute parameters. This entity can then participate in subsequent rendering processes as an independent rendering primitive.

[0226] In practical applications, the depth information of target visual elements can be determined based on depth gaps through various methods.

[0227] One alternative approach is to use the depth value of the depth gap as a depth reference value for the target visual element. Specifically, the midpoint between the near and far depths of the depth gap can be taken as the depth information of the target visual element, allowing the target visual element to be positioned in the middle of the depth gap.

[0228] Another alternative approach is to calculate the depth value of the target visual element based on its expected position in three-dimensional space and the depth range of the depth gap. For example, the near-end depth of the depth gap can be used as a boundary condition to calculate the depth information of the target visual element based on its expected display position.

[0229] Another alternative approach is to dynamically adjust the depth information of the target visual element based on the depth distribution characteristics of the depth gap and the type of the target visual element. For example, for a target visual element that needs to be highlighted, its depth value can be set to a depth close to the near end of the depth gap, and for a target visual element that needs to be weakened, its depth value can be set to a depth close to the far end of the depth gap.

[0230] Once the depth information of the target visual element is determined, the target visual element can be converted into a renderable entity based on the depth information.

[0231] One alternative approach is to convert the 2D image or vector graphics data of the target visual element, combined with its depth information, into a renderable entity with a 3D spatial position. Specifically, for a 2D image-type target visual element, a rectangular planar mesh consisting of two triangles can be generated, and the image can be applied as a texture map to the mesh, while setting the position of the plane in 3D space according to the depth information; for vector graphics, they can first be rasterized into a texture with an alpha channel, and then a renderable entity can be generated in a similar manner.

[0232] Another alternative approach is to create a virtual 3D plane or surface for the target visual element, apply the texture map of the target visual element to the plane, and determine the position and orientation of the plane in 3D space based on the depth information of the target visual element, thereby forming a renderable entity.

[0233] Another alternative approach is to convert the target visual element into a set of one or more three-dimensional Gaussian points, each of which has color, transparency, and depth information corresponding to the target visual element. A renderable entity is formed by superimposing and projecting these three-dimensional Gaussian points.

[0234] In addition, specific rendering shader programs can be assigned to target visual elements based on their type and attributes, so that renderable entities can present the expected visual effects during the rendering process.

[0235] Once the target visual element is converted into a renderable entity, the renderable entity can be added to the rendering sequence based on the depth gap and the depth information of the target visual element.

[0236] One alternative approach is to directly insert the renderable entity into the rendering sequence at the depth position corresponding to the depth gap. Specifically, the position corresponding to the depth information of the target visual element can be found in the rendering sequence, and the renderable entity can be inserted as a rendering unit into that position, so that the renderable entity can be processed in the correct depth order during the rendering process.

[0237] Another alternative approach is to calculate the insertion position of the renderable entity in the rendering sequence based on the depth range of the depth gap and the depth information of the target visual element. Specifically, the near and far depths of the depth gap can be used as boundary conditions to determine the precise insertion point of the target visual element in the rendering sequence based on its depth information.

[0238] Another alternative approach is to perform depth sorting of renderable entities with their adjacent 3D Gaussian points in the rendering sequence to ensure that renderable entities are processed together with other rendering units at the correct depth level.

[0239] Furthermore, the position and processing method of renderable entities in the rendering sequence can be dynamically adjusted based on their attribute information, such as opacity and visibility. For example, completely opaque renderable entities can be processed first to ensure correct occlusion relationships; semi-transparent renderable entities can be processed using more refined blending algorithms.

[0240] In the embodiments of this specification, the depth information of the target visual element is determined based on the depth gap, the target visual element is converted into a renderable entity based on the depth information of the target visual element, and the renderable entity is added to the rendering sequence based on the depth gap and the depth information of the target visual element. This achieves depth fusion and unified processing of the target visual element and the 3D Gaussian model in the rendering process. By converting the target visual element into a renderable entity, the target visual element can participate in the processing of the 3D rendering pipeline as an independent rendering primitive, providing a data foundation for unified rendering. By adding the renderable entity to the rendering sequence based on the depth gap and the depth information of the target visual element, it is ensured that the target visual element can be processed with the 3D Gaussian point in a unified rendering order. Through the precise matching of the depth information of the renderable entity and the depth gap, the accurate positioning of the target visual element in 3D space is ensured, allowing the target visual element to naturally integrate into the depth hierarchy of the 3D scene and form a correct spatial relationship with other elements in the 3D scene. This ensures the accurate matching of the target visual element and the 3D Gaussian model in the depth dimension, improves the display quality of the target visual element in the target rendered image, and improves the execution efficiency of the entire rendering process.

[0241] In one optional embodiment of this specification, there are multiple target visual elements; Based on depth gaps, target visual elements are added to the rendering sequence, including: Determine the depth information of each target visual element based on at least one depth gap; Based on the depth information corresponding to multiple target visual elements, the multiple target visual elements are converted into multiple renderable entities; Based on the depth information of at least one depth gap and multiple target visual elements, multiple renderable entities are added to the rendering sequence.

[0242] Multiple target visual elements refer to a collection of visual elements that need to be embedded in the same 3D scene. These can be of the same type or different types. Specifically, they can include multiple user interface controls, multiple logo graphics, or multiple decorative patterns. Multiple target visual elements can be used to simultaneously present various interactive prompts, information displays, or visual decorative effects in a 3D scene, enriching the interactive content and visual presentation of the 3D scene.

[0243] In practical applications, the depth information of each target visual element can be determined based on at least one depth gap in a variety of ways.

[0244] One alternative approach is to assign multiple target visual elements to a depth gap and assign different depth information to each target visual element based on their relative position within the depth gap. For example, if two labels arranged one after the other need to be inserted into the same gap between the foreground and background, the label in front can be assigned a shallower depth value in the depth gap, and the label behind can be assigned a deeper depth value.

[0245] Another alternative approach is to assign different target visual elements to different depth gaps when multiple depth gaps exist. For example, a user interface button can be inserted into the first gap between a foreground object and the background, while an information tooltip can be inserted into a second gap between a background object and a more distant wall. The depth level of the visual element can be determined based on its semantics or display priority.

[0246] In addition, the precise three-dimensional coordinates and depth values ​​of each element in the corresponding depth gap can be determined by back-projection calculations, based on the expected position of the target visual element on the two-dimensional projection plane.

[0247] Once the depth information of multiple target visual elements is determined, these target visual elements can be converted into multiple renderable entities based on this depth information.

[0248] Specifically, the conversion process can be performed independently for each target visual element, and the method is similar to that used for processing a single visual element. Please refer to the aforementioned embodiments, which will not be repeated here.

[0249] With multiple renderable entities available, these renderable entities can be added to the rendering sequence based on at least one depth gap and the depth information of multiple target visual elements.

[0250] One alternative approach is to find the corresponding insertion point in the depth-sorted rendering sequence based on the depth value of each renderable entity, and insert them one by one. This can be viewed as performing multiple insertion operations on the rendering sequence, ultimately resulting in an ordered sequence containing all 3D Gaussian points and all target visual elements. During the addition process, if multiple renderable entities are assigned to the same depth gap, their arrangement order within that gap can be further determined based on their relative depth relationships (e.g., front-to-back order).

[0251] Another option is to reserve a position range for each depth gap when generating the rendering sequence, and then place the renderable entities belonging to that gap into the corresponding range in sequence.

[0252] In addition, a dynamic insertion strategy can be adopted, which adjusts the position of multiple renderable entities in the sequence in real time according to the changes in the current viewpoint and depth gap during the rendering process to adapt to the dynamic changes in the scene. For example, when the user moves the viewpoint and the occlusion relationship changes, the insertion position of the visual elements is recalculated and updated.

[0253] In the embodiments of this specification, the depth information of each target visual element is determined based on at least one depth gap. Multiple target visual elements are converted into multiple renderable entities based on their corresponding depth information. These renderable entities are then added to the rendering sequence based on at least one depth gap and the depth information of the multiple target visual elements. This achieves depth fusion and unified processing of multiple target visual elements and the 3D Gaussian model during the rendering process. By adding multiple renderable entities to the rendering sequence based on at least one depth gap and the depth information of the multiple target visual elements, it ensures that multiple target visual elements can be processed with the 3D Gaussian points in a unified rendering order. This avoids depth conflicts and occlusion problems between multiple target visual elements, ensures accurate positioning of multiple target visual elements in 3D space, and allows multiple target visual elements to naturally integrate into the depth hierarchy of the 3D scene, forming correct spatial relationships with other elements in the 3D scene. This guarantees accurate matching of multiple target visual elements with the 3D Gaussian model in the depth dimension, improves the display quality of multiple target visual elements in the target rendered image, and simultaneously improves the execution efficiency of the entire rendering process.

[0254] Corresponding to the above method embodiments, this specification also provides embodiments of an image rendering apparatus. Figure 3 A schematic diagram of the structure of an image rendering apparatus according to one embodiment of this specification is shown. Figure 3 As shown, the device includes: The acquisition module 202 is configured to acquire the rendering sequence of the 3D Gaussian model and the target visual elements, wherein the 3D Gaussian model includes multiple 3D Gaussian points, and any 3D Gaussian point includes depth information; The determination module 204 is configured to determine at least one depth gap in a three-dimensional Gaussian model based on depth information from multiple three-dimensional Gaussian points. Add module 206, configured to add target visual elements to the rendering sequence based on depth gaps; Rendering module 208 is configured to render the rendering sequence to obtain the target rendered image.

[0255] The acquisition module is a functional unit within the image rendering device responsible for acquiring the rendering sequence of the 3D Gaussian model and target visual elements. It may include a data reading unit, a data transmission unit, and a data generation unit. The acquisition module can retrieve the rendering sequence of the 3D Gaussian model from storage devices or external data sources, and acquire target visual elements from a visual element library or through programmatic generation. Specifically, the acquisition module can read pre-generated 3D Gaussian model rendering sequences from local storage systems or cloud storage systems, and can also receive rendering sequence data transmitted externally via a network interface. Simultaneously, the acquisition module can select appropriate target visual elements from a predefined visual element library, or dynamically generate target visual elements based on user input, providing a data foundation for embedding target visual elements in the 3D scene.

[0256] The determination module is a functional unit in the image rendering device responsible for determining at least one depth gap in a 3D Gaussian model based on depth information from multiple 3D Gaussian points. It may include a depth analysis unit, a gap identification unit, and a depth gradient calculation unit. The determination module can analyze the depth distribution characteristics of each 3D Gaussian point in the 3D Gaussian model, identify regions of depth discontinuity, and thus determine suitable depth gaps for inserting target visual elements. Specifically, the determination module can divide the 3D Gaussian model based on the depth information of each 3D Gaussian point, obtaining multiple 3D Gaussian point depth intervals. Then, based on the depth information of the 3D Gaussian points in each depth interval, it determines at least one depth gap in the 3D Gaussian model through statistical analysis or depth gradient calculation, providing a spatial reference for the accurate positioning of target visual elements in the 3D scene.

[0257] The adding module is a functional unit in the image rendering apparatus responsible for adding target visual elements to the rendering sequence based on depth gaps. It may include a depth mapping unit, an entity conversion unit, and a sequence insertion unit. The adding module converts target visual elements into renderable entities and inserts these entities into their corresponding positions in the rendering sequence based on the depth information from the depth gaps. Specifically, the adding module determines the depth information of the target visual element based on the depth gaps, then converts the target visual element into a renderable entity with a three-dimensional spatial position. Finally, based on the depth gaps and the depth information of the target visual element, it adds the renderable entity to the rendering sequence, ensuring that the target visual element and the three-dimensional Gaussian points are processed in a consistent rendering order.

[0258] The rendering module is a functional unit in an image rendering device responsible for rendering a rendering sequence containing target visual elements to obtain the target rendered image. It may include layer generation units, layer overlay units, and image output units. The rendering module processes each 3D Gaussian point and target visual element in the rendering sequence as independent rendering primitives, generating corresponding Gaussian point layers and visual element layers. Then, based on the sequence position of each rendering unit in the rendering sequence, these layers are overlaid to finally generate the target rendered image. Specifically, the rendering module can use rasterization or ray tracing techniques to render each rendering unit, generating Gaussian point layers and visual element layers. Then, based on occlusion relationships and visibility information, these layers are overlaid to ensure the correct fusion of the target visual elements and the 3D Gaussian model in the depth dimension, generating a target rendered image with correct spatial hierarchy.

[0259] The image rendering apparatus provided in the embodiments of this specification acquires the rendering sequence of a 3D Gaussian model and target visual elements through an acquisition module, providing a data foundation for subsequent processing; determines at least one depth gap in the 3D Gaussian model based on the depth information of multiple 3D Gaussian points through a determination module, providing a spatial reference for the accurate positioning of target visual elements in the 3D scene; adds target visual elements to the rendering sequence based on the depth gaps through an addition module, achieving depth fusion between target visual elements and the 3D Gaussian model during the rendering process; and performs unified rendering processing on the rendering sequence containing target visual elements through a rendering module, ensuring accurate matching of target visual elements and the 3D Gaussian model in the depth dimension. Through the collaborative work of each module, a consistent depth fusion between target visual elements and the 3D scene is achieved during the rendering process, enabling target visual elements to naturally integrate into the depth hierarchy of the 3D scene, forming correct spatial relationships with other elements in the 3D scene, avoiding depth misalignment and visual fragmentation problems caused by image overlay methods, improving the display quality of target visual elements in the target rendered image of the 3D Gaussian model, and increasing the rendering efficiency of the target rendered image.

[0260] Optionally, the image rendering apparatus further includes a generation module configured to: acquire a model image of a three-dimensional Gaussian model; determine the depth information of each three-dimensional Gaussian point in the three-dimensional Gaussian model based on the model image of the three-dimensional Gaussian model; determine the rendering priority of each three-dimensional Gaussian point in the three-dimensional Gaussian model based on the depth information of each three-dimensional Gaussian point; and sort each three-dimensional Gaussian point based on the rendering priority to determine the rendering sequence of the three-dimensional Gaussian model.

[0261] Optionally, the model image includes an initial two-dimensional image; the generation module is further configured to: use a pre-trained depth estimation model to perform depth estimation on the initial two-dimensional image to obtain an initial depth map; and based on the initial depth map, determine the depth information of each three-dimensional Gaussian point in the three-dimensional Gaussian model, wherein the three-dimensional Gaussian points are obtained by three-dimensional mapping based on each initial pixel in the initial two-dimensional image.

[0262] Optionally, the model image includes an initial rendered image; the generation module is further configured to: based on the initial rendering view of the initial rendered image, traverse the pixel neighborhood of each rendered pixel in the initial rendered image to determine the rendering depth map of the 3D Gaussian model; and based on the rendering depth map, determine the depth information of each 3D Gaussian point in the 3D Gaussian model.

[0263] Optionally, the generation module is further configured to: determine the rendering parameters of each three-dimensional Gaussian point in the three-dimensional Gaussian model based on the depth information of each three-dimensional Gaussian point; and determine the rendering priority of each three-dimensional Gaussian point in the three-dimensional Gaussian model based on the rendering parameters of each three-dimensional Gaussian point.

[0264] Optionally, the determining module is further configured to: divide the three-dimensional Gaussian model based on the depth information of multiple three-dimensional Gaussian points to obtain multiple three-dimensional Gaussian point depth intervals; and determine at least one depth gap in the three-dimensional Gaussian model based on the depth information of the three-dimensional Gaussian points in each three-dimensional Gaussian point depth interval.

[0265] Optionally, the determining module is further configured to: determine the depth change gradient between the depth intervals of the three-dimensional Gaussian points based on the depth information of the three-dimensional Gaussian points in the depth intervals of each three-dimensional Gaussian point; determine the depth drop position and the depth rise position based on the depth change gradient and the preset depth change threshold; and determine at least one depth gap in the three-dimensional Gaussian model based on the drop position and the depth rise position.

[0266] Optionally, the rendering module is further configured to: sequentially render each 3D Gaussian point and target visual element in the rendering sequence, thereby obtaining a Gaussian point layer for each 3D Gaussian point and a visual element layer for the target visual element; and overlay the Gaussian point layer and the visual element layer based on the sequence position of each 3D Gaussian point and target visual element in the rendering sequence to obtain the target rendered image.

[0267] Optionally, the rendering module is further configured to: determine the occlusion relationship between the Gaussian point layer and the visual element layer based on the sequence position of each 3D Gaussian point and the target visual element in the rendering sequence; and overlay the Gaussian point layer and the visual element layer based on the occlusion relationship and the visibility information of each 3D Gaussian point and the target visual element to obtain the target rendered image.

[0268] Optionally, the adding module is further configured to: determine the depth information of the target visual element based on the depth gap; convert the target visual element into a renderable entity based on the depth information of the target visual element; and add the renderable entity to the rendering sequence based on the depth gap and the depth information of the target visual element.

[0269] Optionally, there are multiple target visual elements; the adding module is further configured to: determine the depth information of each target visual element based on at least one depth gap; convert the multiple target visual elements into multiple renderable entities based on the depth information corresponding to the multiple target visual elements; and add the multiple renderable entities to the rendering sequence based on at least one depth gap and the depth information of the multiple target visual elements.

[0270] The above is a schematic scheme of an image rendering apparatus according to this embodiment. It should be noted that the technical solution of this image rendering apparatus and the technical solution of the image rendering method described above belong to the same concept. For details not described in detail in the technical solution of the image rendering apparatus, please refer to the description of the technical solution of the image rendering method described above.

[0271] Figure 3 A structural block diagram of a computing device 300 according to one embodiment of this specification is shown. The components of the computing device 300 include, but are not limited to, a memory 310 and a processor 320. The processor 320 is connected to the memory 310 via a bus 330, and a database 350 is used to store data.

[0272] The computing device 300 also includes an access device 340, which enables the computing device 300 to communicate via one or more networks 360. Examples of these networks include a Public Switched Telephone Network (PSTN), a Local Area Network (LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), or a combination of communication networks such as the Internet. The access device 340 may include one or more of any type of wired or wireless network interface (e.g., a network interface controller (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, or a Near Field Communication (NFC) interface.

[0273] In one embodiment of this specification, the aforementioned components of the computing device 300 and Figure 3 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 3 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0274] The computing device 300 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 300 can also be a mobile or stationary server.

[0275] The processor 320 is used to execute the following computer program / instructions, which, when executed by the processor, implement the steps of the above-described image rendering method.

[0276] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the image rendering method described above belong to the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the image rendering method described above.

[0277] An embodiment of this specification also provides a computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the above-described image rendering method.

[0278] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium and the technical solution of the image rendering method described above belong to the same concept. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the image rendering method described above.

[0279] An embodiment of this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described image rendering method.

[0280] The above is an illustrative scheme of a computer program according to this embodiment. It should be noted that the technical solution of this computer program and the technical solution of the image rendering method described above belong to the same concept. For details not described in detail in the technical solution of the computer program, please refer to the description of the technical solution of the image rendering method described above.

[0281] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0282] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0283] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0284] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0285] The preferred embodiments disclosed above are merely illustrative of this specification. Optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. An image rendering method, characterized in that, include: Obtain the rendering sequence of a 3D Gaussian model and the target visual elements, wherein the 3D Gaussian model includes multiple 3D Gaussian points, and any 3D Gaussian point includes depth information; Based on the depth information of the plurality of three-dimensional Gaussian points, at least one depth gap in the three-dimensional Gaussian model is determined; Based on the depth gap, the target visual element is added to the rendering sequence; The rendering sequence is rendered to obtain the target rendered image.

2. The method according to claim 1, characterized in that, Before obtaining the rendering sequence of the 3D Gaussian model, the following is also included: Obtain the model image of the three-dimensional Gaussian model; Based on the model image of the three-dimensional Gaussian model, determine the depth information of each three-dimensional Gaussian point in the three-dimensional Gaussian model; Based on the depth information of each three-dimensional Gaussian point, the rendering priority of each three-dimensional Gaussian point in the three-dimensional Gaussian model is determined; The rendering sequence of the three-dimensional Gaussian model is determined by sorting the three-dimensional Gaussian points based on the rendering priority.

3. The method according to claim 2, characterized in that, The model image includes an initial two-dimensional image; The determination of depth information of each 3D Gaussian point in the 3D Gaussian model based on the model image of the 3D Gaussian model includes: Using a pre-trained depth estimation model, the initial two-dimensional image is depth estimated to obtain an initial depth map; Based on the initial depth map, the depth information of each three-dimensional Gaussian point in the three-dimensional Gaussian model is determined, wherein the three-dimensional Gaussian points are obtained by three-dimensional mapping based on each initial pixel in the initial two-dimensional image.

4. The method according to claim 2, characterized in that, The model image includes the initial rendered image; The determination of depth information of each 3D Gaussian point in the 3D Gaussian model based on the model image of the 3D Gaussian model includes: Based on the initial rendering viewpoint of the initial rendered image, the pixel neighborhood of each rendered pixel in the initial rendered image is traversed to determine the rendering depth map of the three-dimensional Gaussian model. Based on the rendered depth map, the depth information of each three-dimensional Gaussian point in the three-dimensional Gaussian model is determined.

5. The method according to claim 2, characterized in that, The step of determining the rendering priority of each 3D Gaussian point in the 3D Gaussian model based on the depth information of each 3D Gaussian point includes: Based on the depth information of each three-dimensional Gaussian point, the rendering parameters of each three-dimensional Gaussian point in the three-dimensional Gaussian model are determined; Based on the rendering parameters of each 3D Gaussian point, the rendering priority of each 3D Gaussian point in the 3D Gaussian model is determined.

6. The method according to claim 1, characterized in that, Determining at least one depth gap in the three-dimensional Gaussian model based on the depth information of the plurality of three-dimensional Gaussian points includes: Based on the depth information of the multiple three-dimensional Gaussian points, the three-dimensional Gaussian model is divided to obtain multiple three-dimensional Gaussian point depth intervals; Based on the depth information of the three-dimensional Gaussian points in each three-dimensional Gaussian point depth interval, at least one depth gap in the three-dimensional Gaussian model is determined.

7. The method according to claim 6, characterized in that, The step of determining at least one depth gap in the three-dimensional Gaussian model based on the depth information of the three-dimensional Gaussian points in each depth interval includes: Based on the depth information of the three-dimensional Gaussian points in each depth interval, the depth change gradient between each depth interval of the three-dimensional Gaussian points is determined. Based on the depth change gradient and the preset depth change threshold, the locations of steep depth drop and steep depth rise are determined; Based on the steep drop position and the steep depth rise position, at least one depth gap in the three-dimensional Gaussian model is determined.

8. The method according to any one of claims 1-7, characterized in that, The step of rendering the rendering sequence to obtain the target rendered image includes: The three-dimensional Gaussian points and the target visual elements in the rendering sequence are rendered sequentially to obtain the Gaussian point layer of each three-dimensional Gaussian point and the visual element layer of the target visual element. Based on the position of each 3D Gaussian point and the target visual element in the rendering sequence, the Gaussian point layer and the visual element layer are superimposed to obtain the target rendered image.

9. The method according to claim 8, characterized in that, The step of overlaying the Gaussian point layer and the visual element layer based on the sequence positions of the three-dimensional Gaussian points and the target visual elements in the rendering sequence to obtain the target rendered image includes: Based on the sequence positions of the three-dimensional Gaussian points and the target visual element in the rendering sequence, the occlusion relationship between the Gaussian point layer and the visual element layer is determined. Based on the occlusion relationship and the visibility information of each 3D Gaussian point and the target visual element, the Gaussian point layer and the visual element layer are superimposed to obtain the target rendered image.

10. The method according to claim 1, characterized in that, Adding the target visual element to the rendering sequence based on the depth gap includes: Based on the depth gap, the depth information of the target visual element is determined; Based on the depth information of the target visual element, the target visual element is converted into a renderable entity; Based on the depth gap and the depth information of the target visual element, the renderable entity is added to the rendering sequence.

11. The method according to claim 1, characterized in that, The target visual elements are multiple; Adding the target visual element to the rendering sequence based on the depth gap includes: Based on the at least one depth gap, the depth information of each target visual element is determined; Based on the depth information corresponding to the multiple target visual elements, the multiple target visual elements are converted into multiple renderable entities; Based on the depth information of the at least one depth gap and the multiple target visual elements, the multiple renderable entities are added to the rendering sequence.

12. An image rendering apparatus, characterized in that, include: The acquisition module is configured to acquire the rendering sequence of a 3D Gaussian model and the target visual elements, wherein the 3D Gaussian model includes multiple 3D Gaussian points, and any 3D Gaussian point includes depth information. The determination module is configured to determine at least one depth gap in the three-dimensional Gaussian model based on the depth information of the plurality of three-dimensional Gaussian points. An add module is configured to add the target visual element to the rendering sequence based on the depth gap; The rendering module is configured to render the rendering sequence to obtain the target rendered image.

13. A computing device, characterized in that, include: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the image rendering method according to any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, It stores a computer program / instruction that, when executed by a processor, implements the steps of the image rendering method according to any one of claims 1 to 11.

15. A computer program product, characterized in that, Includes a computer program / instructions that, when executed by a processor, implement the steps of the image rendering method according to any one of claims 1 to 11.

Images