Hybrid rendering method and device and storage medium

Abstract

The invention provides a hybrid rendering method and device and a storage medium. The method relates to the technical field of image processing, and comprises the following steps: processing an original image through two-dimensional Gaussian sputtering 2DGS to obtain a two-dimensional Gaussian sputtering image, and extracting a plurality of first feature images; performing extraction processing on the two-dimensional Gaussian sputtering image through an unreal engine to obtain a plurality of second feature images; performing hybrid processing on each first feature image and the corresponding second feature image based on depth to obtain a plurality of target feature images; and calling a delay rendering pipeline of an unreal engine to execute illumination and shadow calculation based on the plurality of target feature images, and generating a target rendering image. According to the method, the rendering effect of the output image can be remarkably improved while the real-time rendering performance is kept by fusing the detail enhancement of the 2DGS and the efficient architecture of the unreal engine.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a hybrid rendering method, device and storage medium. Background Technology

[0002] In the development and testing of intelligent driving systems, the visual realism and physical accuracy of the simulation scenario have a decisive impact on the reliability and safety of the entire system. Visual realism directly determines the quality and distribution of image data processed by the perception algorithm, while physical accuracy relates to the effectiveness of vehicle dynamics models, sensor simulations, and decision-making control algorithms in the virtual environment.

[0003] In related technologies, when rendering high-precision simulation scenes in real time, Unreal Engine, based on meshes, is typically used. To improve rendering speed, larger meshes are usually used. However, larger meshes cannot render detailed images, resulting in poor rendering quality. Summary of the Invention

[0004] This application provides a hybrid rendering method, device, and storage medium to resolve the contradiction between the lack of rendering details and rendering efficiency caused by using a single large-scale mesh when rendering high-precision simulation scenes in real time, thereby improving the rendering effect.

[0005] In a first aspect, this application provides a hybrid rendering method, the method comprising:

[0006] The original image is processed by 2DGS sputtering to obtain a 2D Gaussian sputtered image, and multiple first feature images are extracted from the 2D Gaussian sputtered image.

[0007] The two-dimensional Gaussian sputtering image is extracted and processed using Unreal Engine to obtain multiple second feature images;

[0008] Based on depth, each first feature image and its corresponding second feature image are mixed to obtain multiple target feature images;

[0009] Based on multiple target feature images, the Unreal Engine's deferred rendering pipeline is invoked to perform lighting and shadow calculations, generating the target rendered image.

[0010] In one possible implementation, the two-dimensional Gaussian sputtering image has color features, depth features, and normal features;

[0011] The plurality of first feature images include a first color image, a first depth image, and a first normal image;

[0012] The plurality of second feature images include a second color image, a second depth image, and a second normal image;

[0013] The plurality of target feature images include a target depth image, a target color image, and a target normal image;

[0014] The depth-based mixing process for each first feature image and its corresponding second feature image yields multiple target feature images, including:

[0015] The target depth image is obtained based on the first depth image and the second depth image, wherein the depth of each pixel in the target depth image is the minimum value between the depth of the pixel in the first depth image and the depth in the second depth image;

[0016] Based on the depth of each pixel in the target depth image, a color is selected from the first color image and the second color image in the depth-corresponding color image to obtain the target color image;

[0017] Based on the depth of each pixel in the target depth image, the normals in the normal images corresponding to the depths are selected from the first normal image and the second normal image to obtain the target normal image.

[0018] In one possible implementation, obtaining the target depth image based on the first depth image and the second depth image includes:

[0019] For any pixel, a first depth corresponding to the pixel is determined in the first depth image, a second depth corresponding to the pixel is determined in the second depth image, and a target depth of the pixel is determined based on the first depth and the second depth.

[0020] The target depth image is obtained based on the target depth of multiple pixels.

[0021] In one possible implementation, determining the target depth of the pixel based on the first depth and the second depth includes:

[0022] If the first depth is less than or equal to the second depth, then the first depth is determined as the target depth corresponding to the pixel.

[0023] If the first depth is greater than the second depth, then the second depth is determined as the target depth corresponding to the pixel.

[0024] In one possible implementation, the step of selecting a color from the depth-corresponding color image in the first color image and the second color image based on the depth of each pixel in the target depth image to obtain the target color image includes:

[0025] For any pixel in the target depth image, if the target depth of the pixel is a first depth, then the target color corresponding to the pixel is determined to be a first color, and the first color is the color corresponding to the pixel in the first color image; if the target depth of the pixel is a second depth, then the target color corresponding to the pixel is determined to be a second color, and the second color is the color corresponding to the pixel in the second color image.

[0026] A target color image is obtained based on the target color of multiple pixels.

[0027] In one possible implementation, the step of selecting a normal from the first normal image and the second normal image corresponding to the depth of each pixel in the target depth image to obtain the target normal image includes:

[0028] For any pixel in the target depth image, if the target depth of the pixel is a first depth, then the target normal corresponding to the pixel is determined to be a first normal, and the first normal is the normal corresponding to the pixel in the first normal image; if the target depth of the pixel is a second depth, then the target normal corresponding to the pixel is determined to be a second normal, and the second normal is the normal corresponding to the pixel in the second normal image.

[0029] The target normal image is obtained based on the target normals of multiple pixels.

[0030] In one possible implementation, the method further includes:

[0031] The two-dimensional Gaussian sputtering image is sent to the cloud server so that the cloud server generates an identifier corresponding to the two-dimensional Gaussian sputtering image, and the identifier is used to obtain the two-dimensional Gaussian sputtering image.

[0032] In one possible implementation, the method further includes:

[0033] Send an image loading request to the cloud server, the image loading request including a target identifier;

[0034] Receive the target two-dimensional Gaussian sputtering image corresponding to the target identifier sent by the cloud server.

[0035] Secondly, this application provides a hybrid rendering apparatus, the apparatus comprising:

[0036] The first processing module is used to process the original image to obtain a two-dimensional Gaussian sputtered image by two-dimensional Gaussian sputtering 2DGS, and to extract multiple first feature images from the two-dimensional Gaussian sputtered image.

[0037] The second processing module is used to extract and process the two-dimensional Gaussian sputtering image using Unreal Engine to obtain multiple second feature images;

[0038] The third processing module is used to perform a mixing process on each first feature image and the corresponding second feature image based on depth to obtain multiple target feature images;

[0039] The generation module is used to generate target rendered images by calling the Unreal Engine's deferred rendering pipeline to perform lighting and shadow calculations based on multiple target feature images.

[0040] Thirdly, this application provides an electronic device, including: a memory and a processor;

[0041] The memory stores computer-executed instructions;

[0042] The processor executes computer execution instructions stored in the memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0043] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.

[0044] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.

[0045] The hybrid rendering method, device, and storage medium provided in this application include processing the original image using 2DGS (Two-Dimensional Gaussian Sputtering) to obtain a 2D Gaussian sputtered image, and extracting features from the image to obtain multiple first feature images; extracting features from the 2D Gaussian sputtered image using Unreal Engine to obtain multiple second feature images; performing adaptive blending processing on each first feature image and its corresponding second feature image based on depth information to obtain multiple target feature images; and using the multiple target feature images, calling the Unreal Engine's deferred rendering pipeline to perform lighting and shadow calculations to generate a target rendered image. By combining the detail enhancement capabilities of 2DGS with the efficient rendering architecture of Unreal Engine, the rendering effect of the final output image is significantly improved while maintaining real-time rendering performance. Attached Figure Description

[0046] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0047] Figure 1 A flowchart illustrating an embodiment of the hybrid rendering method provided in this application;

[0048] Figure 2 A flowchart illustrating Embodiment 2 of the hybrid rendering method provided in this application;

[0049] Figure 3 A schematic diagram illustrating the principle of acquiring a target depth image as provided in an embodiment of this application;

[0050] Figure 4 A flowchart illustrating Embodiment 3 of the hybrid rendering method provided in this application;

[0051] Figure 5 This is a schematic diagram of the structure of the hybrid rendering apparatus provided in the embodiments of this application;

[0052] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0053] The accompanying drawings have illustrated specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to specific embodiments. Detailed Implementation

[0054] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0055] First, let me explain the terms used in this application:

[0056] Unreal Engine (UE): A powerful real-time 3D creation tool that supports a dynamic global illumination system for highly realistic lighting rendering; it has a built-in Chaos physics engine that supports vehicle dynamics and soft / rigid body collision detection, enabling the construction of high-precision intelligent driving simulation systems; Unreal Engine is cross-platform and highly scalable, supporting custom development to meet simulation needs from single-machine to distributed collaborative models.

[0057] 3D Gaussian Splatting (3DGS): A point cloud rendering technique that achieves real-time high-fidelity simulation by optimizing the Gaussian distribution in 3D space. Unlike traditional methods that use meshes or point clouds to represent a 3D scene, it uses thousands of tiny, learnable "3D Gaussian spheres" for representation.

[0058] Two-dimensional Gaussian Splatting (2DGS) is a variant of 3DGS. Its aggregated surfaces are deterministic and do not become ambiguous due to changes in the viewing angle. It has high geometric consistency and high accuracy of collision bodies.

[0059] In the critical process of intelligent driving simulation and synthetic data production, the visual realism of the simulation scene is the most core and fundamental evaluation indicator. Its realism and physical accuracy directly determine the credibility and effectiveness of the intelligent driving system's perception, decision-making and control algorithm testing in the simulation environment. At the same time, it also fundamentally affects the diversity, labeling accuracy and the quality and generalization ability of the generated synthetic data used for model training.

[0060] In related technologies, to achieve real-time rendering of high-precision simulation scenes, traditional mesh-based rendering pipelines are commonly relied upon, such as using Unreal Engine for scene construction and rendering. To meet real-time requirements, large meshes with low polygon counts and simplified structures are often used to improve rendering efficiency. However, this approach struggles to accurately represent the subtle geometric features and material details of complex surfaces, significantly limiting the visual realism and expressiveness of the final rendering result, leading to poor rendering quality.

[0061] To address the aforementioned issues, the inventors proposed a hybrid rendering method integrating 2DGS and Unreal Engine. This method involves using 2DGS to perform high-frequency detail reconstruction and enhancement on the original image, generating a 2D Gaussian sputtering image, and extracting multiple first feature images containing rich details from the 2D Gaussian sputtering image. Simultaneously, Unreal Engine performs low-level feature extraction on the 2D Gaussian sputtering image to obtain multiple second feature images. Based on this, a depth-based adaptive feature fusion mechanism is used to blend the multiple first and second feature images, generating a series of enhanced target feature images. Finally, the Unreal Engine's deferred rendering pipeline is invoked to perform lighting and shadow calculations, outputting a visually detailed and realistic target rendered image. In this process, by offloading the computationally intensive detail generation task to the 2DGS preprocessing stage, the geometric and texture load on the Unreal Engine rendering pipeline is significantly reduced, thereby greatly improving rendering efficiency. This method effectively overcomes the limitations of traditional single-mesh rendering in terms of detail representation, achieving high-fidelity restoration of the scene's fine structure without sacrificing rendering efficiency.

[0062] The application scenarios of the embodiments of this application will be described below first.

[0063] The hybrid rendering method provided in this application can be applied to real-time graphics rendering fields that require both high scene rendering efficiency and visual realism. It is particularly suitable for scenarios such as intelligent driving simulation testing, high-fidelity digital twins, real-time 3D interactive simulation, and high-quality real-time visualization in film and games. For example, in an intelligent driving simulation testing scenario, the target rendered image generated based on the technical solution of this application can provide training and testing data containing rich geometric details and high-precision lighting effects for vehicle perception algorithms. These highly realistic images can accurately simulate visual conditions under different weather, lighting, and complex road environments, thereby significantly improving the generalization ability and reliability of the perception model in the real world.

[0064] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0065] Figure 1 This is a flowchart illustrating an embodiment of the hybrid rendering method provided in this application. Please refer to [link / reference]. Figure 1 The method includes:

[0066] S101. The original image is processed by 2DGS sputtering to obtain a 2D Gaussian sputtered image, and the 2D Gaussian sputtered image is further processed to obtain multiple first feature images.

[0067] The execution subject of the method embodiments of this application can be an electronic device or a hybrid rendering device disposed in an electronic device. The hybrid rendering device can be implemented by software or by a combination of software and hardware. The hybrid rendering device can be a processor in the electronic device. For ease of understanding, the technical solution of this application will be described below using an electronic device as an example.

[0068] In this step, the original image can be processed using 2DGS to generate a two-dimensional Gaussian sputtering image composed of multiple 2DGS patches. Subsequently, a feature extraction network can be used to extract multiple feature maps from the two-dimensional Gaussian sputtering image to obtain multiple first feature images.

[0069] Optionally, the multiple first feature images may include a first color image, a first depth image, and a first normal image. The first color image is used to represent the basic color information of the scene after 2DGS processing; the first depth image is used to represent the relative or absolute spatial distance between pixels in the scene; and the first normal image is used to represent the orientation and micro-geometric details of the object surfaces in the scene.

[0070] Specifically, a 2DGS patch is a differentiable rendering primitive based on a two-dimensional Gaussian distribution in 2DGS technology. It has optimizable parameters including position parameters, covariance matrix, opacity, and appearance attributes. The position parameter determines the patch's specific location in the image plane using two-dimensional coordinates. The covariance matrix controls the shape of the Gaussian distribution corresponding to the patch, including its scale and orientation rotation. The opacity parameter adjusts the patch's visibility during image compositing: higher opacity typically indicates an entity or opaque surface, while lower opacity often represents transparent, semi-transparent, or background areas. Appearance attributes are generally expressed using spherical harmonic coefficients, which can exhibit different emissivity depending on the viewing direction, effectively simulating complex lighting effects and material appearance changes.

[0071] The shape of a 2DGS patch is controlled by the covariance matrix, and it is mainly characterized by an elliptical shape that can be arbitrarily rotated, scaled, and is anisotropic.

[0072] Optionally, the original images can be obtained by acquiring a sequence of real-scene images using high-precision sensors (such as multi-view cameras, LiDAR, or infrared sensors mounted on intelligent vehicles).

[0073] In one alternative implementation, the following steps can be taken. Two-dimensional Gaussian sputtering images are obtained from the original image.

[0074] step The input raw image is preprocessed to convert it into a tensor format to suit the computational requirements of electronic devices (such as image processors). Optionally, preprocessing may include steps such as resizing the input image to a fixed size and normalizing pixel values.

[0075] step Initialize a set of 2DGS patches, each patch being assigned a set of optimizable parameters, including position parameters, covariance matrix, opacity, and appearance attributes.

[0076] step Each 2DGS patch is projected onto the image plane, and the weights of each pixel within the coverage area are calculated based on its covariance matrix. Colors are then synthesized pixel-by-pixel using an alpha blending formula. This rendering process is fully differentiable, allowing optimization of the 2DGS parameters via gradient descent. Finally, by minimizing the photometric loss (such as L1 or L2 loss) and structural similarity loss between the synthesized image and the original image, all patch parameters are iteratively optimized until a visually enhanced, edge-sharp, and semantically preserved 2D Gaussian sputtered image is generated.

[0077] In one alternative implementation, a multi-task feature extraction network based on an encoder-decoder architecture can be used to extract features from a two-dimensional Gaussian sputtering image to obtain multiple first feature images. Specifically, the network takes the generated two-dimensional Gaussian sputtering image as input and first performs multi-scale feature extraction via a shared encoder backbone network. This encoder backbone network can employ structures such as a Feature Pyramid Network (FPN) or a Residual Network (ResNet). Subsequently, the network is divided into three parallel decoding branches: a color feature decoding branch, which outputs a first color image; a depth feature decoding branch, which outputs a first depth image; and a normal feature decoding branch, which outputs a first normal image.

[0078] S102. Extract and process the two-dimensional Gaussian sputtering image using Unreal Engine to obtain multiple second feature images.

[0079] In this step, multi-channel feature extraction can be performed on the 2D Gaussian sputtering image using Unreal Engine to obtain multiple second feature images. These multiple second feature images may include a second color image, a second depth image, and a second normal image.

[0080] In one specific implementation, the generation of multiple second feature images essentially relies on a core component of the Unreal Engine rendering system: the geometry buffer (G-Buffer). The geometry buffer is a crucial data layer in the deferred rendering pipeline, consisting of a set of intermediate render targets precisely aligned within screen space. These render targets store different attributes of the scene object surfaces during the geometry processing phase. These include: the base color channel, which stores surface albedo information; the world normal channel, which primarily records surface orientation data; and the scene depth channel, which specifically stores the depth value of each pixel. This multi-channel parallel storage mechanism provides a complete data foundation for feature extraction.

[0081] By configuring the Unreal Engine's rendering module, you can explicitly specify the rendering channels to be output, thereby extracting the corresponding feature images from the G-Buffer. In specific implementations, you can obtain a second depth image from the scene depth channel, a second color image from the base color channel, and a second normal image from the world normal channel.

[0082] Optionally, the two-dimensional Gaussian sputtering image can be extracted and processed using Unreal Engine to obtain multiple second feature images, including not only the second color image, the second depth image, and the second normal image, but also high-level semantic and physical property images such as roughness image, metallicity image, and ambient occlusion image. These images are used to enhance the performance of downstream computer vision tasks, provide richer scene editing capabilities, and achieve higher levels of visual effects and physically realistic rendering and compositing.

[0083] S103. Based on depth, each first feature image and its corresponding second feature image are mixed to obtain multiple target feature images.

[0084] In this step, based on the depth of each pixel, each set of corresponding feature images from different sources can be blended to generate a series of enhanced target feature images. These target feature images can include target depth images, target color images, and target normal images.

[0085] In the field of image processing technology, depth is generally defined as the vertical distance from a real-world 3D scene point corresponding to a pixel in an image to a specified observation plane (usually the camera's imaging plane or the near-end projection plane of a virtual camera). This geometric quantity establishes a one-to-one mapping relationship between 2D image pixels and 3D spatial points, and is a core geometric parameter for realizing tasks such as 3D reconstruction, stereo vision, and virtual-real fusion.

[0086] In one alternative implementation, each obtained target feature image can be stored in a predefined corresponding channel in the G-Buffer. For example, the target depth image can be stored in the scene depth channel, the target color image in the base color channel, and the target normal image in the world normal channel, to provide structured, multi-channel input data for the subsequent deferred shading stage.

[0087] S104. Based on multiple target feature images, call the Unreal Engine's deferred rendering pipeline to perform lighting and shadow calculations and generate target rendering images.

[0088] In this step, the deferred rendering pipeline built into Unreal Engine can be invoked to perform complete lighting and shadow calculations on the feature data (such as target depth image, target color image, and target normal image) stored in each channel of the G-Buffer, and generate the target rendered image.

[0089] In one alternative implementation, the lighting and weather systems of Unreal Engine can be reused to perform unified lighting and shadow calculations on multiple target feature images that have been incorporated into the G-Buffer. The geometric representation generated by 2DGS and incorporated into the G-Buffer is used as the rendering object, enabling it to receive shadows cast by other objects in the scene and generate realistic self-shadow effects. The final output is a target rendering image with high detail and strictly conforms to the laws of physical lighting.

[0090] In another alternative implementation, a high-performance physics simulation system based on Unreal Engine can be used, combined with the generated target rendered image, to achieve differentiable simulation interaction between the vehicle and the virtual scene. The target rendered image incorporates geometric details and physical properties enhanced by 2DGS, providing the simulation system with high-precision scene geometric representation and semantic information. In this environment, the vehicle model can not only perform optically calibrated visual interaction with the scene, but also achieve vehicle dynamic response, real-time collision detection, and sensor simulation through a differentiable physics engine, significantly improving the realism, closed-loop efficiency, and data validity of the simulation test.

[0091] In this embodiment, the original image can be processed using 2DGS to generate a two-dimensional Gaussian sputtered image with enhanced detail, and multiple first feature images containing high-frequency detail information can be extracted from it. Simultaneously, the same two-dimensional Gaussian sputtered image is subjected to low-level feature extraction using Unreal Engine to obtain multiple second feature images with stable geometric structure and spatial consistency. Subsequently, adaptive blending processing can be performed on each corresponding first and second feature image based on depth to generate a series of target feature images with rich detail and structural integrity. Then, Unreal Engine's deferred rendering pipeline is invoked to perform physically accurate lighting and shadow calculations based on the aforementioned target feature images, outputting a target rendered image with high visual realism and excellent detail. In the above process, by explicitly enhancing high-frequency detail features using 2DGS and combining it with a depth-based adaptive blending mechanism, the output target rendered image can present subtle geometric structures and surface textures that are difficult to achieve with traditional mesh rendering. Furthermore, through Unreal Engine's physically based lighting and shadow calculations, a rendering result rich in detail, with accurate light and shadow relationships and a high degree of visual realism is generated, significantly improving the overall rendering effect.

[0092] Furthermore, the hybrid rendering method provided in this application, by introducing 2DGS, can significantly improve the efficiency of 3D model reconstruction and the realism of the generated results. Compared with traditional mesh modeling, this method greatly reduces the time and economic cost of the reconstruction process, while effectively solving the problem of insufficient simulation of complex materials, fine structures and dynamic lighting in traditional mesh modeling, providing a reliable data foundation for achieving high realism and high efficiency in real-time rendering.

[0093] Figure 2 This is a flowchart illustrating a second embodiment of the hybrid rendering method provided in this application. Please refer to [link / reference]. Figure 2 Based on the above embodiments, the two-dimensional Gaussian sputtering image has color features, depth features, and normal features; multiple first feature images include a first color image, a first depth image, and a first normal image; multiple second feature images include a second color image, a second depth image, and a second normal image; multiple target feature images include a target depth image, a target color image, and a target normal image; in the specific implementation of this hybrid rendering method, step S103 performs a hybrid processing on each first feature image and the corresponding second feature image based on depth to obtain multiple target feature images, and the specific implementation also includes the following steps:

[0094] S201. Obtain the target depth image based on the first depth image and the second depth image.

[0095] In this step, the first depth image and the second depth image can be blended to obtain the target depth image. The depth of each pixel in the target depth image is the minimum of the pixel's depth in the first depth image and its depth in the second depth image.

[0096] In one optional implementation, for any given pixel, a first depth corresponding to the pixel can be determined in a first depth image, a second depth corresponding to the pixel can be determined in a second depth image, and a target depth of the pixel can be determined based on the first and second depths. Furthermore, a target depth image can be obtained based on the target depths of multiple pixels.

[0097] In one optional implementation, the specific process of determining the target depth of a pixel based on a first depth and a second depth may include: if the first depth is less than or equal to the second depth, then the first depth is determined as the target depth corresponding to the pixel; if the first depth is greater than the second depth, then the second depth is determined as the target depth corresponding to the pixel.

[0098] It should be noted that if the first depth and the second depth are the same, either depth (such as the first depth) can be selected as the target depth of the pixel to ensure the clarity and efficiency of the depth fusion process.

[0099] Figure 3 This is a schematic diagram illustrating the principle of acquiring a target depth image as provided in an embodiment of this application. Please refer to [link / reference]. Figure 3For example, both the first depth image and the second depth image include N pixels, namely pixel 1, pixel 2, ..., and pixel n. For any pixel i (1 ≤ i ≤ N), firstly, its corresponding first depth Ai is obtained from the first depth image, and its corresponding second depth Bi is obtained from the second depth image. Then, by comparing the magnitudes of Ai and Bi, the smaller value is selected as the target depth Hi of the pixel. After traversing all N pixels and performing the above operations in sequence, the target depths of all pixels constitute the final target depth image.

[0100] S202. Based on the depth of each pixel in the target depth image, select the color from the depth-corresponding color image from the first color image and the second color image to obtain the target color image.

[0101] In one optional implementation, for any pixel in the target depth image, if the target depth of the pixel is a first depth, then the target color corresponding to the pixel is determined to be a first color, and the first color is the color corresponding to the pixel in the first color image; if the target depth of the pixel is a second depth, then the target color corresponding to the pixel is determined to be a second color, and the second color is the color corresponding to the pixel in the second color image; furthermore, a target color image can be obtained based on the target colors of multiple pixels.

[0102] For example, both the first color image and the second color image contain N pixels, namely pixel 1, pixel 2, ..., and pixel N. For any pixel i (1 ≤ i ≤ N), firstly, its corresponding first color Ci is obtained from the first color image, and its corresponding second color Di is obtained from the second color image. At the same time, based on the target depth image obtained in the aforementioned steps, the target depth Hi corresponding to the pixel and its source flag (indicating whether Hi originates from the first depth or the second depth) are obtained.

[0103] If the target depth Hi of pixel i comes from the first depth image (i.e., Hi=Ai), then the target color of the pixel is determined to be Ci; if Hi comes from the second depth image (i.e., Hi=Bi), then its target color is determined to be Di. After traversing all N pixels and performing the above color selection operation in sequence, the target colors of all pixels constitute the final target color image.

[0104] S203. Based on the depth of each pixel in the target depth image, select the normals in the normal images corresponding to the depth from the first normal image and the second normal image to obtain the target normal image.

[0105] In one optional implementation, for any pixel in the target depth image, if the target depth of the pixel is a first depth, then the target normal corresponding to the pixel is determined as the first normal, and the first normal is the normal corresponding to the pixel in the first normal image; if the target depth of the pixel is a second depth, then the target normal corresponding to the pixel is determined as the second normal, and the second normal is the normal corresponding to the pixel in the second normal image; then, based on the target normals of multiple pixels, a target normal image is obtained.

[0106] For example, both the first normal image and the second normal image contain N pixels, namely pixel 1, pixel 2, ..., and pixel N. For any pixel i (1 ≤ i ≤ N), firstly, the corresponding first normal Ei is obtained from the first normal image, and the corresponding second normal Fi is obtained from the second normal image. At the same time, based on the target depth image obtained in the aforementioned steps, the target depth Hi corresponding to the pixel and its source flag (indicating whether Hi originates from the first depth or the second depth) are obtained.

[0107] If the target depth Hi of pixel i comes from the first depth image (i.e., Hi=Ai), then the target normal of that pixel is determined to be Ei; if Hi comes from the second depth image (i.e., Hi=Bi), then its target normal is determined to be Fi. After traversing all N pixels and performing the above normal selection operation in sequence, the target normals of all pixels constitute the final target normal image.

[0108] In one optional implementation, the obtained target depth image, target color image, and target normal image can be uniformly integrated into a G-Buffer to construct a standardized scene representation that combines high-precision geometry with rich semantics. Based on this enhanced G-Buffer, depth ground truth and semantic segmentation ground truth can be extracted quickly and accurately, significantly improving the quality and efficiency of synthetic data generation. Furthermore, compared to 3DGS, 2DGS has the characteristics of well-defined surfaces and high geometric consistency, enabling the generation of high-precision scene geometric representations. This characteristic allows it to better support downstream tasks such as physical simulation, collision detection, and LiDAR sensor simulation, significantly improving the reliability and realism of applications such as autonomous driving simulation.

[0109] In this embodiment, a target depth image can be obtained based on a first depth image and a second depth image, wherein the depth value of each pixel in the target depth image is the minimum of its corresponding depth values ​​in the first and second depth images. Finally, based on the depth of each pixel in the target depth image, corresponding color values ​​can be selected from the first and second color images to generate a target color image; simultaneously, corresponding normal data can be selected from the first and second normal images to obtain a target normal image. In this process, by selecting the minimum depth value pixel by pixel, the problem of misestimation of distant backgrounds that may exist with a single depth source can be effectively avoided, making the final geometric representation closer to the nearest surface of the real scene, reducing penetration artifacts and rendering errors.

[0110] In one possible design, after processing the original image using 2DGS to obtain a two-dimensional Gaussian sputtering image, the two-dimensional Gaussian sputtering image can be sent to a cloud server so that the cloud server generates an identifier corresponding to the two-dimensional Gaussian sputtering image. The identifier is used to obtain the two-dimensional Gaussian sputtering image.

[0111] The identifier is a unique code generated by the cloud server, used to efficiently and securely index and retrieve two-dimensional Gaussian sputtering images stored on the cloud server.

[0112] In one alternative implementation, a dedicated material library for storing two-dimensional Gaussian sputtering images can be pre-established in the cloud server. When the cloud server receives a two-dimensional Gaussian sputtering image uploaded by an electronic device, it can store it in the material library and assign a unique identifier (e.g., a hash value generated based on the image content) to the image.

[0113] In this embodiment of the application, 2DGS-generated two-dimensional Gaussian sputtering images can be uniformly stored and efficiently managed through a cloud server, and a unique identifier can be assigned to each image to ensure the security and traceability of image resources.

[0114] Figure 4 This is a flowchart illustrating Embodiment 3 of the hybrid rendering method provided in this application. Please refer to... Figure 4 Based on the above embodiments, the hybrid rendering method further includes:

[0115] S401. Send an image loading request to the cloud server. The image loading request includes a target identifier.

[0116] In this step, the electronic device can send an image loading request to the cloud server, which explicitly includes the target identifier corresponding to the required image.

[0117] Optionally, the target identifier can be a unique code assigned and returned by the cloud server when the image was previously uploaded, used to precisely specify the two-dimensional Gaussian sputtering image to be acquired.

[0118] For example, in intelligent driving simulation testing, electronic devices (such as vehicle-mounted simulation terminals or simulators) need to obtain a two-dimensional Gaussian sputtering image of a road that was previously uploaded in order to synthesize the next frame of data. At this time, the electronic device can send an image loading request to the cloud server, and the image loading request can contain a target identifier (such as the hash value e5b3).

[0119] S402, Receive the target two-dimensional Gaussian sputtering image corresponding to the target identifier sent by the cloud server.

[0120] In this step, the electronic device receives the corresponding two-dimensional Gaussian sputtering image retrieved and returned by the cloud server based on the target identifier. Specifically, after receiving the image loading request, the cloud server can quickly locate the storage address of the two-dimensional Gaussian sputtering image corresponding to the target identifier by querying its internal material library, and then send the two-dimensional Gaussian sputtering image to the requesting electronic device.

[0121] For example, a cloud server can quickly retrieve the corresponding 2D Gaussian sputtered image of a road scene from its material library based on the received target identifier (such as the hash value e5b3) and return the 2D Gaussian sputtered image to an electronic device.

[0122] In one alternative implementation, the two-dimensional Gaussian sputtering image can be stored as a scene file in a cloud server's media library. This scene file not only contains the image's pixel data but also integrates related metadata, such as acquisition viewpoint, spatial location identifier, and timestamp. By encapsulating the two-dimensional Gaussian sputtering image into a structured scene file for unified storage and management, multi-dimensional retrieval and version management can be better supported.

[0123] For example, users can quickly filter the scene files they need based on spatial location, time, or viewpoint attributes, while the server can automatically match image assets under different scenes based on metadata, improving the efficiency and consistency of large-scale scene construction.

[0124] In this embodiment, the electronic device can send an image loading request containing a target identifier to a cloud server and receive a target 2D Gaussian sputtering image corresponding to the target identifier returned by the cloud server. By centrally storing and managing the image assets generated by 2DGS through the cloud server, the electronic device can lightweightly call relevant resources through the target identifier, thereby effectively promoting the large-scale application of 2DGS in simulation and synthetic data generation. In addition, it can significantly reduce the local storage footprint and computational load of the electronic device, and significantly reduce the resource dependence and system size of the electronic device.

[0125] Figure 5 This is a schematic diagram of the hybrid rendering apparatus provided in an embodiment of this application. Please refer to... Figure 5 The hybrid rendering apparatus 10 includes:

[0126] The first processing module 11 is used to process the original image through two-dimensional Gaussian sputtering (2DGS) to obtain a two-dimensional Gaussian sputtered image, and to extract multiple first feature images from the two-dimensional Gaussian sputtered image.

[0127] The second processing module 12 is used to extract and process the two-dimensional Gaussian sputtering image through Unreal Engine to obtain multiple second feature images;

[0128] The third processing module 13 is used to perform a mixing process on each first feature image and the corresponding second feature image based on depth to obtain multiple target feature images;

[0129] The generation module 14 is used to generate target rendering images by calling the Unreal Engine's deferred rendering pipeline to perform lighting and shadow calculations based on multiple target feature images.

[0130] The hybrid rendering apparatus provided in this application embodiment can execute the technical solutions shown in the above method embodiments. Its implementation principle and beneficial effects are similar, and will not be described again here.

[0131] In one possible implementation, the two-dimensional Gaussian sputtering image has color features, depth features, and normal features;

[0132] Multiple first feature images include a first color image, a first depth image, and a first normal image;

[0133] Multiple second feature images include a second color image, a second depth image, and a second normal image;

[0134] Multiple target feature images include target depth image, target color image, and target normal image;

[0135] The third processing module 13 is specifically used for:

[0136] Based on the first depth image and the second depth image, a target depth image is obtained. The depth of each pixel in the target depth image is the minimum value between the depth of the pixel in the first depth image and the depth in the second depth image.

[0137] Based on the depth of each pixel in the target depth image, the color corresponding to the depth is selected from the first color image and the second color image to obtain the target color image;

[0138] Based on the depth of each pixel in the target depth image, the normals in the normal images corresponding to the depths are selected from the first normal image and the second normal image to obtain the target normal image.

[0139] In one possible implementation, the third processing module 13 is specifically used for:

[0140] For any pixel, determine the first depth corresponding to the pixel in the first depth image, determine the second depth corresponding to the pixel in the second depth image, and determine the target depth of the pixel based on the first depth and the second depth.

[0141] Obtain a target depth image based on the target depth of multiple pixels.

[0142] In one possible implementation, the third processing module 13 is specifically used for:

[0143] If the first depth is less than or equal to the second depth, then the first depth is determined as the target depth corresponding to the pixel.

[0144] If the first depth is greater than the second depth, then the second depth is determined as the target depth corresponding to the pixel.

[0145] In one possible implementation, the third processing module 13 is specifically used for:

[0146] For any pixel in the target depth image, if the target depth of the pixel is the first depth, then the target color corresponding to the pixel is determined to be the first color, and the first color is the color corresponding to the pixel in the first color image; if the target depth of the pixel is the second depth, then the target color corresponding to the pixel is determined to be the second color, and the second color is the color corresponding to the pixel in the second color image.

[0147] A target color image is obtained based on the target color of multiple pixels.

[0148] In one possible implementation, the third processing module 13 is specifically used for:

[0149] For any pixel in the target depth image, if the target depth of the pixel is the first depth, then the target normal corresponding to the pixel is determined to be the first normal, and the first normal is the normal corresponding to the pixel in the first normal image; if the target depth of the pixel is the second depth, then the target normal corresponding to the pixel is determined to be the second normal, and the second normal is the normal corresponding to the pixel in the second normal image.

[0150] The target normal image is obtained based on the target normals of multiple pixels.

[0151] In one possible implementation, the first processing module 11 is further configured to:

[0152] Send a two-dimensional Gaussian sputtering image to the cloud server so that the cloud server generates an identifier corresponding to the two-dimensional Gaussian sputtering image. The identifier is used to obtain the two-dimensional Gaussian sputtering image.

[0153] In one possible implementation, the first processing module 11 is further configured to:

[0154] Send an image loading request to the cloud server. The image loading request includes a target identifier.

[0155] Receive the target two-dimensional Gaussian sputtering image corresponding to the target identifier sent by the cloud server.

[0156] The hybrid rendering apparatus provided in this application embodiment can execute the technical solutions shown in the above method embodiments. Its implementation principle and beneficial effects are similar, and will not be described again here.

[0157] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Please refer to... Figure 6 The electronic device 20 provided in this embodiment includes at least one processor 21 and a memory 22. Optionally, the device 20 also includes a communication component 23. The processor 21, the memory 22, and the communication component 23 are connected via a bus 24.

[0158] In a specific implementation, at least one processor 21 executes computer execution instructions stored in memory 22, causing at least one processor 21 to perform the above-described method.

[0159] The specific implementation process of processor 21 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0160] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0161] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0162] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0163] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0164] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0165] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0166] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0167] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0168] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0169] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0170] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0171] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0172] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A hybrid rendering method, characterized in that, The method includes: The original image is processed by 2DGS sputtering to obtain a 2D Gaussian sputtered image, and multiple first feature images are extracted from the 2D Gaussian sputtered image. The two-dimensional Gaussian sputtering image is extracted and processed using Unreal Engine to obtain multiple second feature images; Based on depth, each first feature image and its corresponding second feature image are mixed to obtain multiple target feature images; Based on multiple target feature images, the Unreal Engine's deferred rendering pipeline is invoked to perform lighting and shadow calculations, generating the target rendered image.

2. The method according to claim 1, characterized in that, The two-dimensional Gaussian sputtering image has color features, depth features, and normal features; The plurality of first feature images include a first color image, a first depth image, and a first normal image; The plurality of second feature images include a second color image, a second depth image, and a second normal image; The plurality of target feature images include a target depth image, a target color image, and a target normal image; The depth-based mixing process for each first feature image and its corresponding second feature image yields multiple target feature images, including: The target depth image is obtained based on the first depth image and the second depth image, wherein the depth of each pixel in the target depth image is the minimum value between the depth of the pixel in the first depth image and the depth in the second depth image; Based on the depth of each pixel in the target depth image, a color is selected from the first color image and the second color image in the depth-corresponding color image to obtain the target color image; Based on the depth of each pixel in the target depth image, the normals in the normal images corresponding to the depths are selected from the first normal image and the second normal image to obtain the target normal image.

3. The method according to claim 2, characterized in that, The step of obtaining the target depth image based on the first depth image and the second depth image includes: For any pixel, a first depth corresponding to the pixel is determined in the first depth image, a second depth corresponding to the pixel is determined in the second depth image, and a target depth of the pixel is determined based on the first depth and the second depth. The target depth image is obtained based on the target depth of multiple pixels.

4. The method according to claim 3, characterized in that, Determining the target depth of the pixel based on the first depth and the second depth includes: If the first depth is less than or equal to the second depth, then the first depth is determined as the target depth corresponding to the pixel. If the first depth is greater than the second depth, then the second depth is determined as the target depth corresponding to the pixel.

5. The method according to any one of claims 2-4, characterized in that, The step of selecting a color from the first color image and the second color image corresponding to the depth of each pixel in the target depth image to obtain the target color image includes: For any pixel in the target depth image, if the target depth of the pixel is a first depth, then the target color corresponding to the pixel is determined to be a first color, and the first color is the color corresponding to the pixel in the first color image; if the target depth of the pixel is a second depth, then the target color corresponding to the pixel is determined to be a second color, and the second color is the color corresponding to the pixel in the second color image. A target color image is obtained based on the target color of multiple pixels.

6. The method according to any one of claims 2-4, characterized in that, The step of selecting the normal from the first normal image and the second normal image corresponding to the depth of each pixel in the target depth image to obtain the target normal image includes: For any pixel in the target depth image, if the target depth of the pixel is a first depth, then the target normal corresponding to the pixel is determined to be a first normal, and the first normal is the normal corresponding to the pixel in the first normal image; if the target depth of the pixel is a second depth, then the target normal corresponding to the pixel is determined to be a second normal, and the second normal is the normal corresponding to the pixel in the second normal image. The target normal image is obtained based on the target normals of multiple pixels.

7. The method according to claim 1, characterized in that, The method further includes: The two-dimensional Gaussian sputtering image is sent to the cloud server so that the cloud server generates an identifier corresponding to the two-dimensional Gaussian sputtering image, and the identifier is used to obtain the two-dimensional Gaussian sputtering image.

8. The method according to claim 7, characterized in that, The method further includes: Send an image loading request to the cloud server, the image loading request including a target identifier; Receive the target two-dimensional Gaussian sputtering image corresponding to the target identifier sent by the cloud server.

9. An electronic device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-8.

Images