Three-dimensional rendering method and device, storage medium and program product

Abstract

The embodiment of the invention provides a three-dimensional rendering method and device, a storage medium and a program product. In the embodiment of the invention, for a three-dimensional Gaussian sputtering model of a three-dimensional object, a camera pose and a display pose of the three-dimensional object are input, a Gaussian point is adaptively selected according to an observation distance between a virtual camera and the three-dimensional object reflected by the camera pose and the display pose, Gaussian sputtering is carried out for the selected Gaussian point, and the three-dimensional object is obtained. And obtaining a two-dimensional image corresponding to the three-dimensional object. Wherein the number of the Gaussian points is in negative correlation with the distance, the number of the Gaussian points is adaptively adjusted along with the observation distance, a large number of Gaussian points do not need to be rendered, and the rendering performance is improved. Furthermore, the Gaussian point appearing on the polygonal surface is adaptively adjusted from the dimension of the polygonal surface corresponding to the three-dimensional model according to the distance between the polygonal surface and the virtual camera, the distance information with the virtual camera does not need to be determined from the dimension of the Gaussian point, the calculation amount is reduced, and the rendering performance is improved.

Description

Technical Field

[0001] The present application relates to the field of three-dimensional rendering technology, and in particular to a three-dimensional rendering method, device, storage medium and program product. Background Art

[0002] 3D Gaussian Splatting (3DGS) is a high-performance photorealistic 3D rendering technology that is widely used in mixed reality (MR), augmented reality (AR) or virtual reality (VR) technology. A 3D scene includes a 3D object, which can be rendered into a 2D image through 3DGS technology, and the 2D image is presented on the screen of an MR device, VR device or AR device to display the 3D scene.

[0003] However, 3DGS technology consumes a lot of rendering resources, which limits the use of 3DGS technology. Summary of the invention

[0004] Multiple aspects of the present application provide a three-dimensional rendering method, device, storage medium and program product to reduce resources consumed by three-dimensional rendering and improve the performance of three-dimensional rendering.

[0005] An embodiment of the present application provides a three-dimensional rendering method, which includes: determining a first camera pose in a target three-dimensional scene and a first display pose corresponding to a three-dimensional object to be rendered in the target three-dimensional scene; wherein the three-dimensional object corresponds to a target three-dimensional Gaussian sputtering model, and the target three-dimensional Gaussian sputtering model includes a plurality of first Gaussian points for describing the three-dimensional object; selecting a target Gaussian point from the plurality of first Gaussian points according to distance information reflected by the first camera pose and the first display pose, wherein the number of the target Gaussian points is negatively correlated with the distance information; and performing Gaussian sputtering on the target Gaussian point according to the first display pose to obtain a two-dimensional image corresponding to the three-dimensional object.

[0006] An embodiment of the present application also provides an electronic device, including: a memory and a processor; the memory is used to store a computer program; the processor is coupled to the memory and is used to execute the computer program to implement each step in the three-dimensional rendering method provided in the embodiment of the present application.

[0007] The embodiment of the present application also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor implements each step of the three-dimensional rendering method provided in the embodiment of the present application.

[0008] The embodiment of the present application also provides a computer program product, including a computer program / instruction. When the computer program / instruction is executed by a processor, the processor is caused to implement each step of the three-dimensional rendering method provided in the embodiment of the present application.

[0009] In an embodiment of the present application, a three-dimensional object is represented by a three-dimensional Gaussian sputtering model, and a target Gaussian point is adaptively selected from the Gaussian points included in the three-dimensional Gaussian sputtering model according to the observation distance between the virtual camera and the three-dimensional object reflected by the camera posture and the display posture, and Gaussian sputtering is performed on the selected target Gaussian point to obtain a two-dimensional image corresponding to the three-dimensional object. Among them, the number of Gaussian points is negatively correlated with the distance, and the number of Gaussian points can be adaptively adjusted with the observation distance. Under the condition of ensuring the visual effect of the image, there is no need to render a large number of Gaussian points, thereby improving the rendering performance.

[0010] Furthermore, based on the dimension of the polygonal surface corresponding to the three-dimensional model and the distance between the polygonal surface and the virtual camera, the Gaussian points appearing on the polygonal surface are adaptively adjusted. There is no need to determine the distance information from the virtual camera from the dimension of the Gaussian point, thereby reducing the amount of calculation and improving rendering performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0012] Figure 1 A schematic diagram of a flow chart of a three-dimensional rendering method provided for an exemplary embodiment of the present application;

[0013] Figure 2a A schematic diagram of the change in the number of Gaussian points with respect to the change in distance provided for an exemplary embodiment of the present application;

[0014] Figure 2b A schematic diagram of projecting a triangular surface onto a screen provided for an exemplary embodiment of the present application;

[0015] Figure 2c A schematic diagram of a polygonal surface, a polygonal mesh, and a plurality of first Gaussian points provided for an exemplary embodiment of the present application;

[0016] Figure 3a A schematic diagram of the reasoning process of a target three-dimensional Gaussian sputtering model provided for an exemplary embodiment of the present application;

[0017] Figure 3b A schematic diagram of a training process of a target three-dimensional Gaussian sputtering model provided for an exemplary embodiment of the present application;

[0018] Figure 4A schematic structural diagram of a three-dimensional rendering device provided for an exemplary embodiment of the present application;

[0019] Figure 5 A schematic structural diagram of an electronic device provided for an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0021] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0022] The various models involved in this application (including but not limited to language models or large models) are in compliance with relevant laws and standards.

[0023] In this embodiment, the rendering performance of the three-dimensional Gaussian sputtering (3DGS) model is highly correlated with the number of Gaussian points rendered. Generally, in order to ensure the rendering effect, all Gaussian points of the three-dimensional object are rendered through the 3DGS model, so that the image quality of the rendered two-dimensional image is high. When the observation distance is far, the texture information of the three-dimensional object can be reduced and the rendering details can be simplified. However, the 3DGS model still needs to render all Gaussian points, which seriously affects the rendering efficiency of the 3DGS model. Among them, the observation distance refers to the relative distance between the virtual camera and the three-dimensional object in the virtual space, that is, the observation distance of the virtual camera observing the three-dimensional object.

[0024] In an embodiment of the present application, a scheme is proposed for performing three-dimensional rendering on a three-dimensional Gaussian sputtering model according to the level of detail (LOD). Among them, the level of detail is a technology for optimizing the rendering of three-dimensional graphics. It dynamically adjusts the level of detail of the mesh model corresponding to the three-dimensional object according to the distance between the three-dimensional object and the observer (such as a camera or a video camera). When the three-dimensional object is far away from the observer, the level of detail technology can reduce the number of Gaussian points, reduce the texture resolution, and simplify the rendering details. When the three-dimensional object is close to the observer, the number of Gaussian points is increased to display models and textures with higher details to ensure the visual effect. This method can effectively reduce the consumption of rendering resources and improve the rendering efficiency of three-dimensional scenes.

[0025] In the embodiment of the present application, for the three-dimensional Gaussian sputtering model of the three-dimensional object, Gaussian points are adaptively selected according to the observation distance between the virtual camera and the three-dimensional object reflected by the camera posture and the display posture, and Gaussian sputtering is performed on the selected Gaussian points to obtain a two-dimensional image corresponding to the three-dimensional object. Among them, the number of Gaussian points is negatively correlated with the distance. The number of Gaussian points can be adaptively adjusted with the distance of observation. Under the condition of ensuring the visual effect of the image, there is no need to render a large number of Gaussian points, thereby improving the rendering performance.

[0026] Furthermore, based on the dimension of the polygonal surface corresponding to the three-dimensional model and the distance between the polygonal surface and the virtual camera, the Gaussian points appearing on the polygonal surface are adaptively adjusted. There is no need to determine the distance information from the virtual camera from the dimension of the Gaussian point, thereby reducing the amount of calculation and improving rendering performance.

[0027] Furthermore, the display level of the polygonal surface is determined according to the distance between the polygonal surface and the virtual camera, and the Gaussian points appearing on the polygonal surface are adaptively adjusted based on the display level.

[0028] A solution provided by an embodiment of the present application is described in detail below in conjunction with the accompanying drawings.

[0029] Figure 1 The following is a flow chart of a three-dimensional rendering method provided by an exemplary embodiment of the present application. Figure 1 As shown, the method includes:

[0030] 101. Determine a first camera pose in a target three-dimensional scene and a first display pose corresponding to a three-dimensional object to be rendered in the target three-dimensional scene; wherein the three-dimensional object corresponds to a target three-dimensional Gaussian sputtering model, and the target three-dimensional Gaussian sputtering model includes a plurality of first Gaussian points for describing the three-dimensional object;

[0031] 102. Selecting a target Gaussian point from a plurality of first Gaussian points according to the distance information reflected by the first camera pose and the first display pose, wherein the number of the target Gaussian points is negatively correlated with the distance information;

[0032] 103. Perform Gaussian sputtering on the target Gaussian points according to the first display posture to obtain a two-dimensional image corresponding to the three-dimensional object.

[0033] In this embodiment, the target three-dimensional scene may include but is not limited to: MR scene, VR scene and AR scene, etc. The target three-dimensional scene includes virtual three-dimensional objects. For example, for MR scene, the virtual three-dimensional object is seamlessly integrated with the real world environment. The characteristic of MR is that the boundary between virtual and reality is blurred, and the two are integrated together. Users can interact with virtual three-dimensional objects in the real world. For example, in medical education scenes, students can wear MR devices in the real world to observe virtual human anatomical structures. For another example, for VR scenes, the target three-dimensional scene is a completely virtual three-dimensional environment. Users wear VR devices and immerse themselves in the virtual three-dimensional environment to interact with virtual three-dimensional objects, such as game scenes. For another example, for AR scenes, the virtual environment is superimposed on the real world, and users can interact with virtual three-dimensional objects in the real environment. The characteristics of AR are that virtual information supplements and enhances the real world.

[0034] The target three-dimensional scene also corresponds to a virtual camera, and the three-dimensional rendering process is the process of converting the three-dimensional objects in the target three-dimensional scene into a two-dimensional image through a virtual camera. The two-dimensional image can be displayed on the screen of the MR device, VR device, AR device and terminal device for users to view. If the screen is divided into a left-eye screen and a right-eye screen, the left-eye image and the right-eye image can be generated respectively, and the left-eye image and the right-eye image can be projected onto the left-eye screen and the right-eye screen respectively for users to view, producing a three-dimensional stereoscopic effect. Among them, this process involves the calculation of various effects such as lighting, shadows, reflections, and refractions to achieve realistic visual effects. Among them, the left-eye screen and the right-eye screen can be implemented as VR glasses, MR glasses, AR glasses, or, can also be implemented as VR helmets, etc., without limitation.

[0035] Among them, 3D objects are the basic elements that make up the target 3D scene. 3D objects can be of any shape, from simple geometric bodies such as cubes, spheres and cylinders, to complex models such as people, cars and buildings. Each 3D object is composed of vertices, edges and faces. These vertices have X, Y and Z coordinates in 3D space, which define the spatial position and shape of the 3D object. Texture maps can be applied to the surface of 3D objects to increase details and realism. 3D objects can be created manually through modeling software or automatically using program algorithms.

[0036] Among them, the virtual camera is an observer in the target 3D scene, which can determine the viewing angle and view. In the target 3D scene, the position, direction and lens properties of the virtual camera (such as focal length, field of view) can be freely adjusted to simulate different shooting conditions and visual effects. The movement and rotation of the virtual camera are similar to the real camera used by photographers in the real world. The parameter setting of the virtual camera is crucial for rendering the 2D image corresponding to the 3D object. The parameters of the virtual camera control the perspective projection, determine which parts of the 3D object are visible and which parts are not visible, and how to deal with the effects of depth and distance.

[0037] In this embodiment, the target three-dimensional scene may include one virtual camera or two virtual cameras, for example, a left-eye camera and a right-eye camera. Wherein, in the case where the target three-dimensional scene includes one virtual camera, the first camera pose may be the pose information of the virtual camera. In the case where the target three-dimensional scene includes a left-eye camera and a right-eye camera, the first camera pose includes the pose information of the left-eye camera and the pose information of the right-eye camera. In either case, the first camera pose may include: position information (e.g., spatial coordinates) and posture information (e.g., direction or orientation).

[0038] In this embodiment, the three-dimensional object to be rendered corresponds to a display posture, and the display posture includes position information (such as spatial coordinates) and posture information (such as direction or orientation) of the three-dimensional object in the target three-dimensional scene.

[0039] In this embodiment, the implementation method for determining the first camera pose is not limited. For example, the user can manually position and orient the virtual camera directly in the target three-dimensional scene. For example, through the pose setting interface, the three-dimensional coordinates and rotation angle of the virtual camera can be specified, and the three-dimensional coordinates and rotation angle are used as the first camera pose. For another example, the pose of the virtual camera may be constrained by physical laws, such as gravity or collision, etc. In this case, the physical engine can calculate the dynamic pose of the virtual camera in the target three-dimensional scene and use the dynamic pose as the first camera pose. For another example, in a virtual device, such as a virtual reality (VR) helmet or augmented reality (AR) glasses, built-in sensors (such as gyroscopes, accelerometers, magnetometers) can capture the user's head movement, thereby updating the pose information of the virtual camera in real time. These sensor data usually need to be corrected and optimized by sensor fusion algorithms to reduce latency and improve accuracy. For another example, the user can control the pose of the virtual camera through input devices such as a controller, keyboard, mouse, or touch screen. For example, in a game, a player can change the first camera pose of the virtual camera by moving the mouse or manipulating the game controller.

[0040] In this embodiment, the implementation method for determining the first display posture is also not limited. For example, the user can manually position and orient the three-dimensional object directly in the target three-dimensional scene. For example, through the posture setting interface, the three-dimensional coordinates and rotation angle of the three-dimensional object can be specified, and the three-dimensional coordinates and rotation angle are used as the first display posture. For another example, the posture of the three-dimensional object may be constrained by physical laws, such as gravity or collision. In this case, the physical engine can calculate the dynamic posture of the three-dimensional object in the target three-dimensional scene, and use the dynamic posture as the first display posture. For another example, the user can control the posture of the three-dimensional object through an input device such as a controller, keyboard, mouse or touch screen. For example, in the process of displaying a three-dimensional model, the user can change the first display posture of the three-dimensional object by moving the mouse or manipulating the handle.

[0041] In this embodiment, a target three-dimensional Gaussian sputtering (3DGS) model for representing a three-dimensional object is maintained in advance. The target three-dimensional Gaussian sputtering model can be used for three-dimensional reconstruction and three-dimensional rendering, and can efficiently represent three-dimensional scenes and render complex three-dimensional scenes without losing details. The target three-dimensional Gaussian sputtering model belongs to the field of computer graphics, is a three-dimensional model for three-dimensional expression of three-dimensional objects, and can be used for three-dimensional reconstruction and visualization of three-dimensional objects.

[0042] The target three-dimensional scene may include one or more three-dimensional objects, each three-dimensional object may correspond to a target three-dimensional Gaussian sputtering model, or the target three-dimensional scene may correspond to a target three-dimensional Gaussian sputtering model, which is not limited.

[0043] Wherein, the target three-dimensional Gaussian sputtering model includes a plurality of Gaussian points for describing the three-dimensional object. For the convenience of description and distinction, the plurality of Gaussian points maintained by the target three-dimensional Gaussian sputtering model are referred to as first Gaussian points. Wherein, the plurality of first Gaussian points are used to describe the point cloud data corresponding to the three-dimensional object through Gaussian distribution. The plurality of Gaussian points can describe the position and shape of the three-dimensional object in the target three-dimensional scene. Wherein, the parameters of the Gaussian points include but are not limited to: the center position, shape, size, direction, color and transparency of the point cloud, etc., wherein the mean of the Gaussian points represents the center position of the point cloud, and the covariance matrix of the Gaussian points describes the shape, size and direction of the point cloud, etc.

[0044] In this embodiment, a target Gaussian point can be selected from the multiple first Gaussian points according to the distance information reflected by the first camera posture and the first display posture, wherein the number of target Gaussian points is not limited, and can be 1 or multiple, for example, 10 or 50. The distance information reflected by the first camera posture and the first display posture can be understood as the distance information of the virtual camera observing the three-dimensional object, and the number of target Gaussian points is negatively correlated with the distance information.

[0045] An increase in the number of Gaussian points usually means that there is more information to describe the three-dimensional object, which can increase the detail richness of the rendered two-dimensional image, referred to as detail. More Gaussian points can more finely depict the shape, texture, and surface characteristics of the three-dimensional object. Especially for complex and dynamic application scenarios, an increase in the number of Gaussian points helps to capture and reproduce more subtle features. Therefore, when the distance information indicates that the virtual camera is far away from the three-dimensional object, the detail of the three-dimensional object can be reduced and the number of target Gaussian points can be reduced; when the distance information indicates that the virtual camera is close to the three-dimensional object, the detail of the three-dimensional object can be increased and the number of target Gaussian points can be increased, so as to achieve adaptive adjustment of the number of rendered Gaussian points with the distance information and improve rendering performance. Figure 2a As shown, the process of how the number of Gaussian points changes with the distance between the virtual camera and the three-dimensional object is exemplarily demonstrated, wherein the three-dimensional object is implemented as a "moon" shape as an example for illustration, but is not limited to this.

[0046] Further, in Gaussian sputtering, a Gaussian point is a point with a Gaussian distribution, and the size of the Gaussian point is determined by the standard deviation of the Gaussian distribution. A larger standard deviation produces a wider Gaussian distribution, which means that the Gaussian point has a wider range of influence and it blends more smoothly with the surrounding pixels. Detail refers to the fineness of the smallest feature or texture that can be distinguished in the rendered image. When the size of the Gaussian point increases, the transition between the Gaussian point and other Gaussian points becomes smoother, which causes the originally sharp boundaries to become blurred and the detail information to be averaged, thereby reducing the detail. On the contrary, if the size of the Gaussian point is reduced, the influence range of each Gaussian point is limited, which can maintain the clarity of the feature edges, retain more details, and make the rendering result look sharper and clearer.

[0047] Furthermore, when selecting a target Gaussian point from multiple first Gaussian points, in addition to considering the number of Gaussian points, the size information of the Gaussian points can also be considered. The farther the distance information reflected by the first camera pose and the first display pose is, the larger the size of the selected target Gaussian point is, and the smaller the distance information is, the smaller the size of the selected target Gaussian point is.

[0048] It should be noted that the distance information reflected by the first camera pose and the first display pose is not limited. For example, the distance information can be the overall distance from the three-dimensional object to the screen, or the average distance from each Gaussian point on the three-dimensional object to the screen, or the distance from each Gaussian point to the screen, or the distance from the polygonal surface on the target mesh model of the three-dimensional object to the screen. For a detailed introduction to the target mesh model and the polygonal surface, please refer to the subsequent embodiments, which will not be described in detail here.

[0049] In this embodiment, Gaussian sputtering is performed on the target Gaussian points according to the first display posture to obtain a two-dimensional image corresponding to the three-dimensional object. Among them, Gaussian sputtering is a technology used for real-time radiation field rendering, which describes the three-dimensional object by Gaussian points, and uses these Gaussian points for efficient rasterization processing, and projects the three-dimensional object onto the screen by sputtering to achieve fast image rendering. Among them, in the target three-dimensional Gaussian sputtering model, sputtering specifically refers to projecting the Gaussian points onto the screen through the intrinsic and extrinsic parameters of the virtual camera. The intrinsic parameters of the virtual camera assist in understanding and controlling the formation of the two-dimensional image, while the extrinsic parameters of the virtual camera assist in mapping the Gaussian points corresponding to the three-dimensional object onto the screen.

[0050] Among them, the pose information of the target Gaussian point corresponding to the three-dimensional object can be determined according to the first display pose, and according to the pose information of the target Gaussian point and the first camera pose of the virtual camera, the target Gaussian point can be projected onto the imaging plane of the virtual camera to obtain a two-dimensional image corresponding to the three-dimensional object, so as to realize three-dimensional rendering.

[0051] In the embodiment of the present application, for the three-dimensional Gaussian sputtering model of the three-dimensional object, according to the observation distance between the virtual camera and the three-dimensional object reflected by the camera posture and the display posture, the target Gaussian point is adaptively selected from the Gaussian points contained in the model, and Gaussian sputtering is performed on the selected target Gaussian point to obtain a two-dimensional image corresponding to the three-dimensional object. Among them, the number of Gaussian points is negatively correlated with the distance. The number of Gaussian points is adaptively adjusted with the observation distance. Under the condition of ensuring the visual effect of the image, there is no need to render a large number of Gaussian points, thereby improving the rendering performance.

[0052] In an optional embodiment, the implementation method of selecting the target Gaussian point from multiple first Gaussian points according to the distance information reflected by the first camera pose and the first display pose is not limited. Considering that the number of first Gaussian points corresponding to the three-dimensional object is large, the amount of calculation involved in selecting the target Gaussian point from the dimension of the Gaussian point is large. In order to reduce the amount of calculation, a target mesh model is constructed for the three-dimensional object, and the target mesh model includes multiple polygonal faces, each of which includes at least two Gaussian points. The target Gaussian point can be selected from the dimension of the polygonal face to reduce the amount of calculation. That is, for the polygonal faces included in the target mesh model, some polygonal faces are closer to the virtual camera, so more detailed information is displayed, and a larger number of target Gaussian points are selected, and other polygonal faces are closer to the virtual camera, so less detailed information is displayed, and a smaller number of target Gaussian points are selected. In other words, in the target mesh model, the number of target Gaussian points displayed by different polygonal faces can be different, which is specifically related to the distance between the polygonal face and the virtual camera.

[0053] Specifically, based on the first display posture, the posture information of the polygonal surface contained in the target mesh model is generated; based on the posture information of the polygonal surface and the first camera posture, the projection area of ​​the polygonal surface on the screen is determined, and the size of the projection area reflects the distance information from the polygonal surface to the screen, and the projection area is negatively correlated with the distance information; based on the size of the projection area, the target Gaussian point is selected from at least two first Gaussian points corresponding to the polygonal surface.

[0054] Among them, according to the principle of perspective projection, the straight line from the projection center (i.e., the virtual camera) to each vertex of the target mesh model can determine the position of the vertex on the screen. If the size of the polygonal surface remains unchanged, when the polygonal surface is far away from the virtual camera, the distance between the virtual camera and the polygonal surface increases, and the projection of the polygonal surface on the screen will be reduced, that is, the projection area becomes smaller; when the polygonal surface is close to the virtual camera, the distance between the virtual camera and the polygonal surface decreases, and the projection of the polygonal surface on the screen will be enlarged, that is, the projection area becomes smaller.

[0055] Among them, the size of the projection area is positively correlated with the number of target Gaussian points. For example, the larger the projection area, the closer the distance between the polygonal surface and the virtual camera is, the more detailed information can be displayed, the more the number of target Gaussian points is, and further, the smaller the size of the target Gaussian points is; the smaller the projection area, the farther the distance between the polygonal surface and the virtual camera is, the less detailed information can be displayed, the fewer the number of target Gaussian points is, and further, the larger the size of the target Gaussian points is.

[0056] Among them, the implementation method of generating the pose information of the polygonal faces contained in the target mesh model according to the first display pose is not limited. The target mesh model is usually composed of vertices, edges and polygonal faces (such as triangles), and the pose (position and posture) of each polygonal face is determined by the positions of multiple vertices constituting the polygonal face. Therefore, according to the overall pose information of the target network model, the position information of each vertex in the target network model is determined; according to the correspondence between the polygonal faces and the vertices, the pose information of each polygonal face is determined in combination with the position information of each vertex. When the first display pose of the entire target mesh model changes, the positions of the vertices contained in the target network model will change accordingly, and accordingly, the pose of the polygonal faces will also change accordingly.

[0057] Among them, the three-dimensional object can be divided into three-dimensional voxels, and the Gaussian points contained in each three-dimensional voxel are merged. This is not suitable for dynamic 3DGS scenes. For example, in a digital human, the points of the upper lip and the lower lip are connected together (similar to Lego) and cannot be moved, which may result in abnormal display effects. Compared with dividing a three-dimensional object into three-dimensional voxels, the method provided in this embodiment of dividing a three-dimensional object into multiple polygonal faces through a target grid model makes the texture information of the three-dimensional object more consistent and the effect more natural and vivid. For example, when the three-dimensional object is a person, the person's expression, such as joy, anger, sorrow, and happiness, can be simulated, and the opening and closing of the upper lip and the lower lip can also be simulated.

[0058] Among them, the implementation method of determining the projection area of ​​the polygonal surface on the screen according to the posture information of the polygonal surface and the posture of the first camera is not limited. For example, according to the posture information of the polygonal surface and the posture information of the first camera, the polygonal surface is projected onto the imaging plane of the virtual camera, so that the projection area of ​​the polygonal surface can be calculated on the imaging plane. Among them, the imaging plane is a reference plane for the virtual camera to capture the target three-dimensional scene for perspective projection, and the screen is a physical or virtual surface that finally displays the two-dimensional image, usually a two-dimensional display device, such as a display or a projection surface. The imaging plane of the virtual camera is the "behind the scenes" for generating the image, and the screen is the "front stage" seen by the user. The conversion from the imaging plane to the screen involves multiple steps, including projection, rendering, adjustment and display, to ensure that the two-dimensional image presents a better visual effect on the screen. For example, the image on the imaging plane can be scaled, cropped or filled to adapt to the size and aspect ratio of the screen, or the image resolution of the imaging plane can be adjusted.

[0059] The implementation method of calculating the projection area is not limited. The following takes the polygonal surface as a triangular surface as an example to illustrate the process of calculating the projection area of ​​the triangular surface. Figure 2b In the figure, the three vertices of the triangle in the target three-dimensional scene are: vertex v0, vertex v1 and vertex v2. The coordinates of the three vertices of the triangle projected onto the two-dimensional screen space are: vertex v ′ 0. Vertex v1 ′ and Vertex v2 ′ The projection area (coverage) of the triangle can be calculated by the following formula:

[0060]

[0061] The implementation method of selecting the target Gaussian point from at least two first Gaussian points corresponding to the polygonal surface according to the size of the projection area is not limited. Optionally, the polygonal surface corresponds to multiple display levels, and the higher the display level, the higher the display detail. Figure 2aAs shown, one Gaussian point is displayed in display level 0, three Gaussian points are displayed in display level 1, and more than three Gaussian points are displayed in display level 2, indicating that as the display level increases, the number of Gaussian points increases and the degree of detail also increases. Among them, multiple display levels can be 3, 5 or more levels, and there is no limitation on this. For example, a polygonal surface corresponds to 3 display levels, and in the order of increasing display levels, the three display levels are respectively represented as display level 0 (detail level = 0), display level 1 (detail level = 1), and display level 2 (detail level = 2). Of course, in the order of increasing display levels, the three display levels can be respectively represented as display level 3, display level 2, and display level 1, and there is no limitation on this. Any first Gaussian point corresponding to a three-dimensional object has a display level range, which indicates the display level in which the first Gaussian point can appear. The display level range of the first Gaussian point may be: the lowest display level (detail_level_min) to the highest display level (detail_level_max). If the first Gaussian point appears in a display level, the lowest display level in the display level range of the first Gaussian point is equal to the highest display level.

[0062] For example, in Figure 2c , the display level range of Gaussian point 1, Gaussian point 2, Gaussian point 3 and Gaussian point 4 includes: display level 2; the display level range of Gaussian point 5 includes: display level 1 to display level 2; the display level range of Gaussian point 6 includes: display level 0 to display level 2; the display level of Gaussian point 7 includes: display level 1; the display level of Gaussian point 8 includes: display level 0.

[0063] Based on this, a specific implementation method of selecting a target Gaussian point from at least two first Gaussian points corresponding to a polygonal surface according to the size of the projection area includes: determining a target display level from multiple display levels corresponding to the polygonal surface according to the size of the projection area; and selecting a display level including the first Gaussian point of the target display level as the target Gaussian point from at least two first Gaussian points corresponding to the polygonal surface.

[0064] For example, the target display level could be Wherein, coverage represents the projection area, and γ is a parameter used to control the rendering complexity, for example, γ = 2. The larger the projection area of ​​the polygonal surface, the higher the detail of the target display level.

[0065] If the display level range of any first Gaussian point is: the lowest display level (detail_level_min) to the highest display level (detail_level_max), match the target display level (detail_level) with the display level range of at least two first Gaussian points corresponding to the polygonal surface; the display level range includes the first Gaussian point of the target display level as the target Gaussian point, that is, detail_level_min≤detail_level≤detail_level_max, such as Figure 3a shown.

[0066] In this embodiment, it is not necessary to traverse all the first Gaussian points to calculate the pixel coverage, but to calculate the pixel coverage of each polygonal surface. In a high-precision three-dimensional Gaussian sputtering model, the number of first Gaussian points is generally much greater than the number of polygonal surfaces, so the calculation overhead can be effectively reduced.

[0067] In an optional embodiment, the three-dimensional object corresponds to a second Gaussian point, which may be an initial Gaussian point contained in the three-dimensional object, and the second Gaussian point corresponds to an initial scale. The three-dimensional object corresponds to a target three-dimensional model, and the target three-dimensional model includes multiple polygonal faces, and each polygonal face corresponds to multiple second Gaussian points. Considering that when the distance between the virtual camera and the three-dimensional object is far, Gaussian points of larger scales can be displayed. If the scale of the Gaussian points corresponding to the three-dimensional object is smaller, it will not be displayed. At this time, multiple second Gaussian points can be merged to obtain new Gaussian points of larger scales. It can be understood that the first Gaussian point includes: the second Gaussian point and the new Gaussian point. When the distance between the three-dimensional object and the virtual camera changes, the target Gaussian point can be adaptively selected from the first Gaussian points of different scales according to the target display level of the polygonal face to improve rendering performance.

[0068] It should be noted that, for each polygonal surface, multiple first Gaussian points corresponding to the polygonal surface may be stored, for example, Figure 2c As shown, Gaussian points 1 to 8 are stored for the polygonal surface. Compared with storing the first Gaussian point for each display level of the polygonal surface, the number of new points can be reduced and storage space can be saved. Figure 2c In the case of storing the first Gaussian point for the polygonal face, Gaussian point 6 is stored sequentially. If Gaussian point 6 is stored for each display level, Gaussian point 6 is stored three times, thus saving storage space. For example, for a target mesh model with 470,000 second Gaussian points, the number of new Gaussian points can be 70,000.

[0069] Specifically, for any polygonal face, multiple candidate Gaussian points corresponding to any display level are determined. In this case, multiple candidate Gaussian points are determined for each display level in descending order of display level details. For the first display level, i.e., the display level with the highest detail (e.g. Figure 2c For display level 2 in the display level, multiple second Gaussian points with an initial scale may be used as multiple candidate Gaussian points. For a non-first display level, multiple first Gaussian points that can appear in the previous display level may be used as multiple candidate Gaussian points for the non-first display level. For example, the previous display level of display level 1 is display level 2. At least two candidate Gaussian points in any display level whose scale is lower than the scale threshold are merged into at least one new Gaussian point, wherein different display levels correspond to different scale thresholds. For candidate Gaussian points whose scale is lower than the scale threshold, two candidate Gaussian points may be merged into a new Gaussian point, or four candidate Gaussian points may be merged into a new Gaussian point, without limitation. Candidate Gaussian points in any display level whose scale is higher than or equal to the scale threshold are referred to as other Gaussian points, and at least one new Gaussian point and other Gaussian points in the any display level are used as multiple first Gaussian points that can appear in the any display level; according to the number of first Gaussian points that can appear in multiple display levels,

[0070] For example, in Figure 2c , Gaussian point 1, Gaussian point 2, Gaussian point 3, Gaussian point 4, Gaussian point 5 and Gaussian point 6 are the second Gaussian points with initial scales. For display level 2, Gaussian points 1 to 6 are taken as candidate Gaussian points, and the scale threshold of display level 2 is lower than or equal to the scales of the above 6 Gaussian points. Therefore, it can be considered that the above 6 Gaussian points can appear in display level 2. For display level 1, Gaussian points 1 to 6 are taken as candidate Gaussian points, and the scales of Gaussian points 1 to 4 are lower than the scale threshold of display level 1. Therefore, Gaussian points 1, Gaussian points 2, Gaussian points 3 and Gaussian points 4 can be merged to obtain Gaussian point 7; the scales of Gaussian points 7, Gaussian points 5 and Gaussian points 6 are higher than or equal to the scale threshold of display level 1. Therefore, Gaussian points 7, Gaussian points 5 and Gaussian points 6 can appear in display level 1. For display level 0, Gaussian point 7, Gaussian point 5 and Gaussian point 6 are taken as candidate Gaussian points. The scales of Gaussian point 7 and Gaussian point 5 are both lower than the scale threshold of display level 0. Then Gaussian point 7 and Gaussian point 5 can be merged to obtain Gaussian point 8. The scales of Gaussian point 8 and Gaussian point 6 are both higher than or equal to the scale threshold of display level 0. Then Gaussian point 8 and Gaussian point 6 can appear in display level 0. Among them, Gaussian point 1 to Gaussian point 8 are all the first Gaussian points of the polygonal surface. According to the Gaussian points that can appear in each display level, the display level range of each Gaussian point can be obtained.

[0071] Optionally, in order to improve the accuracy of generating the first Gaussian point, an implementation method of merging at least two candidate Gaussian points in any display level whose initial scale is lower than any display level scale threshold into at least one new Gaussian point is as follows: according to the preset number of grids corresponding to any display level, multiple polygonal grids are divided on any polygonal surface, and multiple candidate Gaussian points corresponding to the polygonal surface at any display level are divided into different polygonal grids, and the polygonal grid may contain zero, one or more candidate Gaussian points. The shape of the polygonal grid is the same as the shape of the polygonal surface, or the shape of the polygonal grid is different from the shape of the polygonal surface. For example, the polygonal grid is a quadrilateral, and the polygonal surface is a triangle, such as Figure 2c As shown in the figure, the higher the display level, the smaller the polygon mesh area and the greater the number of polygon meshes. For example, when the polygon face is a triangle and the polygon mesh is a quadrilateral, for any display level, the triangle face can be divided into 2 level ×2 level For example, at level 2, the triangles can be divided into 2 2 ×2 2 = 16 quadrilaterals. For the display level (level = 1), the triangle surface can be divided into 2 1 ×2 1 = 4 quadrilaterals. For the display level (level = 0), the triangle surface can be divided into 2 0 ×2 0 =1 quadrilateral, for example, the entire triangular face is placed inside the quadrilateral. Wherein, in the case where any polygonal mesh contains multiple candidate Gaussian points, at least two candidate Gaussian points in any polygonal mesh whose initial scale is lower than the scale threshold of any display level are merged into at least one new Gaussian point. It should be noted that, in the case where any polygonal mesh contains a candidate Gaussian point, if the scale of the candidate Gaussian point is higher than or equal to the scale threshold of the display level, the candidate Gaussian point is used as the first Gaussian point, and if the scale of the candidate Gaussian point is lower than the scale threshold of the display level, the candidate Gaussian point can be discarded, or the scale of the first Gaussian point can be enlarged to obtain a new Gaussian point whose scale is higher than the scale threshold of the display level, and the new Gaussian point is used as the first Gaussian point.

[0072] Among them, when the polygonal surface is divided into multiple display levels, the target three-dimensional Gaussian sputtering model can also be called a target three-dimensional Gaussian sputtering model based on the detail level (3DGS LOD), which can merge the first Gaussian points in the polygonal surface based on the polygonal mesh (also called the mesh body). If the polygonal mesh can be driven (that is, the shape or position can be changed), for example, the polygonal mesh corresponding to display level 2 is changed to the polygonal mesh corresponding to display level 1, then the target Gaussian points appearing in the polygonal surface can also change with the change of the polygonal mesh. This means that at different display levels, the target three-dimensional Gaussian sputtering model can still maintain the consistency of animation or deformation, and there will be no problems with sudden changes or discontinuities in the animation effect.

[0073] In an optional embodiment, the implementation method of Gaussian sputtering the target Gaussian points according to the first display posture to obtain the two-dimensional image corresponding to the three-dimensional object is not limited. Depending on the different implementation methods of the virtual camera, the implementation method of Gaussian sputtering the target Gaussian points is also different. For example, if the virtual camera is a monocular camera, and the first camera posture is the posture information of the monocular camera, then for the viewing angle in the monocular camera, Gaussian sputtering is performed on the target Gaussian points according to the first display posture to obtain the left-view two-dimensional image and the right-view two-dimensional image corresponding to the three-dimensional object. For another example, if the virtual camera is a binocular camera, according to the first camera posture, combined with the camera parameters of the binocular camera, the left posture information and the right posture information corresponding to the binocular camera under the left and right viewing angles are determined; according to the left posture information and the right posture information, Gaussian sputtering is performed on the target Gaussian points respectively to obtain the left-view two-dimensional image and the right-view two-dimensional image corresponding to the three-dimensional object. Among them, the camera parameters can be the relative positions of the left camera and the right camera in the binocular camera, and the first camera posture is the overall posture information of the binocular camera, which can be represented by a rotation matrix and a translation matrix. According to the first camera posture combined with the relative position between the left camera and the right camera, the posture information corresponding to the left camera and the right camera respectively can be determined, that is, the left posture information and the right posture information corresponding to the left and right viewing angles respectively.

[0074] In the case where the virtual camera is implemented as a binocular camera, the left-view two-dimensional image and the right-view two-dimensional image can be synthesized to obtain a three-dimensional image.

[0075] Among them, the left-view two-dimensional image and the right-view two-dimensional image can be synthesized by using the left-by-Side format (SBS) or multi-view video coding (MV-HEVC) to obtain a three-dimensional image.

[0076] SBS is a way of presenting stereoscopic video. For the SBS format, at the sending end, the left and right viewpoint images are horizontally spliced ​​together to form a single SBS format image frame, which is convenient for the transmission and display of stereoscopic video; at the receiving end, the SBS format image frame will be decoded and separated into two single-view images on the left and right, and then presented to the user's left and right eyes respectively, thus creating a stereoscopic effect.

[0077] MV-HEVC is an efficient video coding format, and MV-HEVC is an extension of High Efficiency Video Coding (HEVC). MV-HEVC is mainly used for multi-view video coding. It effectively eliminates redundant information between different perspectives by introducing an inter-layer reference mode in the coding process, thereby improving coding efficiency. MV-HEVC has higher performance than traditional coding methods when processing multi-view or stereoscopic videos. In an optional embodiment, in the process of displaying a three-dimensional object, the three-dimensional object itself can be rotated or translated, and the user can also perform user operations on the three-dimensional object so that the three-dimensional object rotates or translates with the user operation. The above two situations may cause the first display posture of the three-dimensional object to change, and during the change process, the distance between the three-dimensional object and the virtual camera may change. For example, some triangular faces of a three-dimensional object are close to the virtual camera, while other triangular faces are far away from the virtual camera, and the distance of some other triangular faces relative to the virtual camera has not changed. In this case, when the first camera pose of the virtual camera remains unchanged, the change of the first display pose corresponding to the three-dimensional object in the target three-dimensional scene can be responded to to determine the changed second display pose; according to the second display pose and the first camera pose, the target Gaussian points are re-selected from multiple first Gaussian points, and Gaussian sputtering is performed on the re-selected target Gaussian points according to the second display pose to re-render the two-dimensional image corresponding to the three-dimensional object.

[0078] In another optional embodiment, during the process of displaying a three-dimensional object, the position of the virtual camera may change. For example, the user moves while wearing a VR device, causing the first camera posture to change. When the first display posture does not change, the changed second camera posture can be determined in response to the change of the first camera posture in the target three-dimensional scene; based on the second camera posture and the first display posture, target Gaussian points are re-selected from multiple first Gaussian points, and Gaussian sputtering is performed on the re-selected target Gaussian points according to the first display posture to re-render the two-dimensional image corresponding to the three-dimensional object.

[0079] In another optional embodiment, if both the first camera pose and the first display pose change, the changed second camera pose and the second display pose are determined in response to the change of the first camera pose in the target three-dimensional scene and the change of the first display pose corresponding to the three-dimensional object; according to the second camera pose and the second display pose, the target Gaussian points are re-selected from the multiple first Gaussian points, and Gaussian sputtering is performed on the re-selected target Gaussian points according to the second display pose to re-render the two-dimensional image corresponding to the three-dimensional object.

[0080] Among them, regarding the implementation of reselecting the target Gaussian point from the plurality of first Gaussian points according to the second camera pose and / or the second display pose in the above embodiment, and performing Gaussian sputtering on the reselected target Gaussian point according to the first display pose or the second display pose to re-render the two-dimensional image corresponding to the three-dimensional object, please refer to the aforementioned Figure 1 The description in the illustrated embodiment will not be repeated here.

[0081] In an optional embodiment, the model training method of the target three-dimensional Gaussian sputtering model is not limited, and an example is provided below. A plurality of sample data are obtained, wherein each sample data includes: a sample camera pose, a sample display pose, and a reference two-dimensional image corresponding to the three-dimensional object, and the reference image includes the three-dimensional object; in addition, the three-dimensional object corresponds to an initial three-dimensional Gaussian sputtering model, and the initial three-dimensional Gaussian sputtering model includes a plurality of sample first Gaussian points for describing the three-dimensional object; based on the distance information reflected by the sample camera pose and the sample display pose, a sample target Gaussian point is selected from the plurality of sample first Gaussian points; Gaussian sputtering is performed on the sample target Gaussian point according to the sample display pose to obtain a training two-dimensional image corresponding to the three-dimensional object; with the goal of the training two-dimensional image and the reference two-dimensional image meeting the set pixel difference threshold, the initial three-dimensional Gaussian sputtering model is trained, and the parameters of the first Gaussian point are continuously updated until the model converges to obtain a target three-dimensional Gaussian sputtering model, such as Figure 3b As shown. The parameters of the first Gaussian point include: position (mean), covariance matrix (shape and direction), transparency and color, etc. The loss function may include but is not limited to: mean square error (MSE) and structural similarity index (SSIM).

[0082] Furthermore, in order to improve the robustness of the model during model training, after obtaining the target display level of each polygonal surface, a random global offset can be added to the target display level to obtain the offset display level of the polygonal surface, such as Figure 3bBased on the display level after the shift, a sample target Gaussian point is selected from multiple sample first Gaussian points, and Gaussian sputtering is performed on the sample target Gaussian point to obtain a training two-dimensional image corresponding to the three-dimensional object; with the training two-dimensional image and the reference two-dimensional image meeting the set pixel difference threshold as the goal, the initial three-dimensional Gaussian sputtering model is trained until the model converges to obtain a target three-dimensional Gaussian sputtering model.

[0083] The generation method of the initial three-dimensional Gaussian sputtering model is not limited. For example, the initial three-dimensional Gaussian sputtering model maintains a target mesh model corresponding to the three-dimensional object, and the target mesh model includes multiple polygonal faces, each of which corresponds to multiple second Gaussian points. The second Gaussian points describe the location and shape of the polygonal faces and other attribute information. For any display level, the first Gaussian points corresponding to the display level are generated, such as Figure 3b As shown. For the implementation method of obtaining the first Gaussian point of any display level, reference may be made to the above-mentioned embodiment, which will not be described in detail here.

[0084] Regarding the embodiments of the present application Figure 1 The detailed implementation and beneficial effects of each step in the method have been described in detail in the aforementioned embodiments and will not be elaborated here.

[0085] It should be noted that the execution subject of each step of the method provided in the above embodiment can be the same device, or the method can be executed by different devices. For example, the execution subject of steps 101 to 103 can be device A; for another example, the execution subject of steps 101 and 102 can be device A, and the execution subject of step 103 can be device B; and so on.

[0086] In addition, in some of the processes described in the above embodiments and the accompanying drawings, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or executed in parallel, and the sequence numbers of the operations, such as 101, 102, etc., are only used to distinguish between different operations, and the sequence numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit the "first" and "second" to be different types.

[0087] Figure 4 A schematic diagram of a three-dimensional rendering device provided by an exemplary embodiment of the present application is shown in FIG. Figure 4 As shown, the device includes: a determination module 41, a selection module 42 and a sputtering module 43.

[0088] A determination module 41 is used to determine a first camera pose in a target three-dimensional scene and a first display pose corresponding to a three-dimensional object to be rendered in the target three-dimensional scene; wherein the three-dimensional object corresponds to a target three-dimensional Gaussian sputtering model, and the target three-dimensional Gaussian sputtering model includes a plurality of first Gaussian points for describing the three-dimensional object;

[0089] A selection module 42, configured to select a target Gaussian point from a plurality of first Gaussian points according to the first camera pose and the distance information reflected by the first display pose, wherein the number of the target Gaussian points is negatively correlated with the distance information;

[0090] The sputtering module 43 is used to perform Gaussian sputtering on the target Gaussian points according to the first display posture to obtain a two-dimensional image corresponding to the three-dimensional object.

[0091] In an optional embodiment, the selection module 42 is specifically used to: generate posture information of the polygonal face contained in the target mesh model according to the first display posture, the target mesh model is a mesh model corresponding to the three-dimensional object, and the polygonal face corresponds to at least two first Gauss points; determine the projection area of ​​the polygonal face on the screen according to the posture information of the polygonal face and the first camera posture, the size of the projection area reflects the distance information from the polygonal face to the screen and is negatively correlated; select the target Gaussian point from the at least two first Gaussian points corresponding to the polygonal face according to the size of the projection area, and the size of the projection area is positively correlated with the number of target Gaussian points.

[0092] Optionally, the polygonal surface corresponds to multiple display levels, and the higher the display level, the higher the display detail, and any first Gaussian point corresponds to a display level range; the selection module 42 is specifically used to: determine the target display level from the multiple display levels corresponding to the polygonal surface according to the size of the projection area; from at least two first Gaussian points corresponding to the polygonal surface, select the first Gaussian point including the target display level in the display level range as the target Gaussian point.

[0093] Further optionally, the device also includes: a merging module and a processing module; the determination module 41 is also used to determine, for any polygonal face, in order of detail from high to low, multiple candidate Gaussian points corresponding to any display level, the multiple candidate Gaussian points corresponding to any display level include multiple first Gaussian points that can appear in the previous display level, or multiple pre-set second Gaussian points; the merging module is used to merge at least two candidate Gaussian points in any display level whose scale is lower than the scale threshold of any display level into at least one new Gaussian point, and different display levels correspond to different scale thresholds; the processing module is used to use at least one new Gaussian point and other Gaussian points in any display level as multiple first Gaussian points that can appear in any display level; wherein the other Gaussian points are candidate Gaussian points in any display level whose scale is higher than or equal to the scale threshold of any display level; the determination module is also used to determine the display level range of each first Gaussian point based on the first Gaussian points that can appear in multiple display levels.

[0094] Further optionally, the merging module is specifically used to: divide any polygonal face into multiple polygonal meshes according to a preset number of meshes corresponding to any display level, the higher the detail level of any display level, the smaller the area of ​​the polygonal mesh and the greater the number of polygonal meshes; merge at least two candidate Gaussian points in any polygonal mesh whose initial scale is lower than a scale threshold of any display level into at least one new Gaussian point.

[0095] In an optional embodiment, the sputtering module 43 is specifically used to: when the first camera posture corresponds to a binocular camera, determine the left posture information and the right posture information corresponding to the binocular camera in the left and right perspectives respectively according to the first camera posture in combination with the camera parameters of the binocular camera; and perform Gaussian sputtering on the target Gaussian points according to the left posture information and the right posture information, so as to obtain a left-perspective two-dimensional image and a right-perspective two-dimensional image corresponding to the three-dimensional object.

[0096] In an optional embodiment, the determination module 41 is further used to: determine the changed second camera pose and / or second display pose in response to the change of the first camera pose in the target three-dimensional scene and / or the change of the first display pose corresponding to the three-dimensional object; the selection module 42 is further used to: re-select the target Gaussian point from multiple first Gaussian points according to the second camera pose and the first display pose, or the first camera pose and the second display pose, or according to the second camera pose and the second display pose; the sputtering module 43 is further used to: perform Gaussian sputtering on the reselected target Gaussian point according to the first display pose or the second display pose to re-render the two-dimensional image corresponding to the three-dimensional object.

[0097] In an optional embodiment, the device also includes: a training module; a training module for acquiring multiple sample data, any sample data including: a sample camera pose, a sample display pose, and a reference two-dimensional image; wherein the three-dimensional object corresponds to an initial three-dimensional Gaussian sputtering model, and the initial three-dimensional Gaussian sputtering model includes multiple sample first Gaussian points for describing the three-dimensional object; according to the distance information reflected by the sample camera pose and the sample display pose, a sample target Gaussian point is selected from the multiple sample first Gaussian points; Gaussian sputtering is performed on the sample target Gaussian point according to the sample display pose to obtain a training two-dimensional image corresponding to the three-dimensional object; with the goal of the training two-dimensional image and the reference two-dimensional image meeting the set pixel difference threshold, the initial three-dimensional Gaussian sputtering model is trained to obtain a target three-dimensional Gaussian sputtering model.

[0098] Regarding the embodiments of the present application Figure 4 The detailed implementation and beneficial effects of each step in the illustrated device have been described in detail in the aforementioned embodiments and will not be elaborated on here.

[0099] Figure 5 A schematic diagram of the structure of an electronic device provided by an exemplary embodiment of the present application is shown in FIG. Figure 5 As shown, the device includes: a memory 54 and a processor 55.

[0100] The memory 54 is used to store computer programs and can be configured to store various other data to support operations on the electronic device. Examples of such data include instructions for any application or method used to operate on the electronic device.

[0101] The processor 55 is coupled to the memory 54 and is used to execute the computer program in the memory 54, so as to: determine a first camera pose in a target three-dimensional scene and a first display pose corresponding to a three-dimensional object to be rendered in the target three-dimensional scene; wherein the three-dimensional object corresponds to a target three-dimensional Gaussian sputtering model, and the target three-dimensional Gaussian sputtering model includes a plurality of first Gaussian points for describing the three-dimensional object; according to the distance information reflected by the first camera pose and the first display pose, select a target Gaussian point from the plurality of first Gaussian points, and the number of the target Gaussian points is negatively correlated with the distance information; and perform Gaussian sputtering on the target Gaussian point according to the first display pose to obtain a two-dimensional image corresponding to the three-dimensional object.

[0102] In an optional embodiment, when the processor 55 selects a target Gauss point from multiple first Gauss points based on the distance information reflected by the first camera pose and the first display pose, it is specifically used to: generate the pose information of the polygonal face contained in the target mesh model according to the first display pose, the target mesh model is a mesh model corresponding to the three-dimensional object, and the polygonal face corresponds to at least two first Gauss points; determine the projection area of ​​the polygonal face on the screen according to the pose information of the polygonal face and the first camera pose, the size of the projection area reflects the distance information from the polygonal face to the screen and is negatively correlated; select the target Gauss point from the at least two first Gauss points corresponding to the polygonal face according to the size of the projection area, and the size of the projection area is positively correlated with the number of target Gauss points.

[0103] Optionally, the polygonal surface corresponds to multiple display levels, and the higher the display level, the higher the display detail, and any first Gauss point corresponds to a display level range; when the processor 55 selects a target Gauss point from at least two first Gauss points corresponding to the polygonal surface according to the size of the projection area, it is specifically used to: determine the target display level from the multiple display levels corresponding to the polygonal surface according to the size of the projection area; select the first Gauss point including the target display level in the display level range from the at least two first Gauss points corresponding to the polygonal surface as the target Gauss point.

[0104] Further optionally, the processor 55 is also used to: for any polygonal face, determine in sequence a plurality of candidate Gaussian points corresponding to any display level in order of detail from high to low of the display level, the plurality of candidate Gaussian points corresponding to any display level include a plurality of first Gaussian points that can appear in the previous display level, or a plurality of pre-set second Gaussian points; merge at least two candidate Gaussian points in any display level whose scale is lower than a scale threshold of any display level into at least one new Gaussian point, and different display levels correspond to different scale thresholds; use at least one new Gaussian point and other Gaussian points in any display level as a plurality of first Gaussian points that can appear in any display level; wherein the other Gaussian points are candidate Gaussian points in any display level whose scale is higher than or equal to the scale threshold of any display level; and determine the display level range of each first Gaussian point based on the first Gaussian points that can appear in multiple display levels.

[0105] Further optionally, when the processor 55 merges at least two candidate Gaussian points in any display level whose initial scale is lower than the scale threshold of any display level into at least one new Gaussian point, it is specifically used to: divide any polygonal face into multiple polygonal grids according to the preset number of grids corresponding to any display level, the higher the detail level of any display level, the smaller the area of ​​the polygonal grid and the greater the number of polygonal grids; merge at least two candidate Gaussian points in any polygonal grid whose initial scale is lower than the scale threshold of any display level into at least one new Gaussian point.

[0106] In an optional embodiment, when the processor 55 performs Gaussian sputtering on the target Gaussian points according to the first display posture to obtain a two-dimensional image corresponding to the three-dimensional object, it is specifically used to: when the first camera posture corresponds to a binocular camera, determine the left posture information and right posture information corresponding to the binocular camera in the left and right perspectives respectively according to the first camera posture and the camera parameters of the binocular camera; and perform Gaussian sputtering on the target Gaussian points according to the left posture information and the right posture information to obtain the left-perspective two-dimensional image and the right-perspective two-dimensional image corresponding to the three-dimensional object.

[0107] In an optional embodiment, the processor 55 is further used to: determine a changed second camera pose and / or a second display pose in response to a change in a first camera pose in a target three-dimensional scene and / or a change in a first display pose corresponding to a three-dimensional object; reselect a target Gaussian point from the multiple first Gaussian points based on the second camera pose and the first display pose, or the second camera pose and the second display pose, or the first camera pose and the second display pose; and perform Gaussian sputtering on the reselected target Gaussian point based on the first display pose or the second display pose to re-render the two-dimensional image corresponding to the three-dimensional object.

[0108] In an optional embodiment, the processor 55 is also used to: obtain multiple sample data, any sample data includes: sample camera pose, sample display pose and reference two-dimensional image; the three-dimensional object corresponds to an initial three-dimensional Gaussian sputtering model, and the initial three-dimensional Gaussian sputtering model includes multiple sample first Gaussian points for describing the three-dimensional object; according to the distance information reflected by the sample camera pose and the sample display pose, select a sample target Gaussian point from the multiple sample first Gaussian points; perform Gaussian sputtering on the sample target Gaussian point according to the sample display pose to obtain a training two-dimensional image corresponding to the three-dimensional object; with the goal of the training two-dimensional image and the reference two-dimensional image meeting the set pixel difference threshold, perform model training on the initial three-dimensional Gaussian sputtering model to obtain a target three-dimensional Gaussian sputtering model.

[0109] Regarding the embodiments of the present application Figure 5 The detailed implementation and beneficial effects of each step in the illustrated device have been described in detail in the aforementioned embodiments and will not be elaborated on here.

[0110] Further, if Figure 5 As shown, the electronic device also includes: a communication component 56, a display 57, a power component 58, an audio component 59 and other components. Figure 5 Only some components are shown schematically, which does not mean that the electronic device only includes Figure 5 In addition, Figure 5The components in the dashed box are optional components, not mandatory components, and the specific components depend on the product form of the electronic device. The electronic device of this embodiment can be implemented as a terminal device such as a desktop computer, a laptop computer, a smart phone, or an IOT device, or a server device such as a conventional server, a cloud server, or a server array. If the electronic device of this embodiment is implemented as a terminal device such as a desktop computer, a laptop computer, a smart phone, etc., it can include Figure 5 If the electronic device of this embodiment is implemented as a server device such as a conventional server, a cloud server or a server array, it may not include Figure 5 Components within the dashed box.

[0111] The detailed implementation and beneficial effects of the electronic device provided in the embodiments of the present application have been described in detail in the aforementioned embodiments and will not be elaborated here.

[0112] Accordingly, the embodiment of the present application further provides a computer-readable storage medium storing a computer program, which can achieve the above-mentioned Figure 1 The steps in the method embodiment shown may be executed by an electronic device.

[0113] Accordingly, the embodiment of the present application also provides a computer program product, including a computer program / instruction, which, when executed by a processor, causes the processor to implement the above Figure 1 The steps in the method embodiment shown may be executed by an electronic device.

[0114] The above-mentioned memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), erasable programmable read only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0115] The above-mentioned communication component is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G / LTE, 5G and other mobile communication networks, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wide Band (UWB) technology, Bluetooth (BT) technology and other technologies.

[0116] The above-mentioned display includes a screen, and the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0117] The power supply assembly provides power to various components of the device where the power supply assembly is located. The power supply assembly may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device where the power supply assembly is located.

[0118] The above-mentioned audio component can be configured to output and / or input audio signals. For example, the audio component includes a microphone (Microphone, MIC), and when the device where the audio component is located is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in a memory or sent via a communication component. In some embodiments, the audio component also includes a speaker for outputting an audio signal.

[0119] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-readable storage media (including but not limited to disk storage, compact disc read-only memory (Compact Disc Read-Only Memory, CD-ROM), optical storage, etc.) containing computer-usable program code.

[0120] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0121] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0122] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0123] In a typical configuration, a computing device includes one or more processors (Central Processing Unit, CPU), input / output interface, network interface and memory.

[0124] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0125] Computer readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0126] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0127] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A three-dimensional rendering method, characterized in that: The method comprises: Determine a first camera pose in a target three-dimensional scene and a first display pose corresponding to a three-dimensional object to be rendered in the target three-dimensional scene; wherein the three-dimensional object corresponds to a target three-dimensional Gaussian sputtering model, and the target three-dimensional Gaussian sputtering model includes a plurality of first Gaussian points for describing the three-dimensional object; Selecting a target Gaussian point from the plurality of first Gaussian points according to the first camera pose and the distance information reflected by the first display pose, wherein the number of the target Gaussian points is negatively correlated with the distance information; Gaussian sputtering is performed on the target Gaussian points according to the first display posture to obtain a two-dimensional image corresponding to the three-dimensional object.

2. The method according to claim 1, characterized in that Selecting a target Gaussian point from the plurality of first Gaussian points according to the distance information reflected by the first camera pose and the first display pose includes: Generate, according to the first display posture, posture information of a polygonal surface included in a target mesh model, wherein the target mesh model is a mesh model corresponding to the three-dimensional object, and the polygonal surface corresponds to at least two first Gaussian points; Determine, according to the posture information of the polygonal surface and the first camera posture, a projection area of ​​the polygonal surface on the screen, wherein the size of the projection area reflects the distance information from the polygonal surface to the screen and is in a negative correlation relationship; According to the size of the projection area, a target Gaussian point is selected from at least two first Gaussian points corresponding to the polygonal surface, and the size of the projection area is positively correlated with the number of the target Gaussian points.

3. The method according to claim 2, characterized in that The polygonal surface corresponds to a plurality of display levels, and the higher the display level, the higher the display detail, and the first Gaussian point corresponds to a display level range; According to the size of the projection area, selecting a target Gauss point from at least two first Gauss points corresponding to the polygonal surface comprises: Determining a target display level from a plurality of display levels corresponding to the polygonal surface according to the size of the projection area; From the at least two first Gauss points corresponding to the polygonal surface, a first Gauss point in the display level range including the target display level is selected as the target Gauss point.

4. The method according to claim 3, characterized in that The method further comprises: For any polygonal face, in descending order of detail of the display levels, a plurality of candidate Gaussian points corresponding to any display level are determined in sequence, wherein the plurality of candidate Gaussian points corresponding to any display level include a plurality of first Gaussian points that can appear in a previous display level, or a plurality of pre-set second Gaussian points; Merging at least two candidate Gaussian points in any display level whose scales are lower than a scale threshold of any display level into at least one new Gaussian point, where different display levels correspond to different scale thresholds; The at least one new Gaussian point and other Gaussian points in any one of the display levels are used as a plurality of first Gaussian points that can appear in any one of the display levels; wherein the other Gaussian points are candidate Gaussian points in any one of the display levels whose scales are higher than or equal to a scale threshold of the any one of the display levels; According to the first Gauss points that can appear in the plurality of display levels, a display level range of each first Gauss point is determined.

5. The method according to claim 4, characterized in that Merging at least two candidate Gaussian points in any display level whose initial scale is lower than a scale threshold of any display level into at least one new Gaussian point, comprising: Dividing a plurality of polygonal meshes on any polygonal surface according to the preset number of meshes corresponding to any display level, wherein the higher the detail level of any display level, the smaller the area of ​​the polygonal meshes and the greater the number of the polygonal meshes; At least two candidate Gaussian points in any polygonal mesh whose initial scale is lower than any display level scale threshold are merged into at least one new Gaussian point.

6. The method according to any one of claims 1 to 5, characterized in that: Gaussian sputtering is performed on the target Gaussian point according to the first display posture to obtain a two-dimensional image corresponding to the three-dimensional object, including: In a case where the first camera posture corresponds to a binocular camera, determining left posture information and right posture information corresponding to the binocular camera under left and right viewing angles, respectively, according to the first camera posture and in combination with camera parameters of the binocular camera; According to the left posture information and the right posture information, Gaussian sputtering is performed on the target Gaussian points respectively to obtain a left-view two-dimensional image and a right-view two-dimensional image corresponding to the three-dimensional object.

7. The method according to any one of claims 1 to 5, characterized in that: Also includes: In response to a change in the first camera pose in the target three-dimensional scene and / or a change in the first display pose corresponding to the three-dimensional object, determining a changed second camera pose and / or a second display pose; Reselecting a target Gaussian point from the plurality of first Gaussian points according to the second camera pose and the first display pose, or the second camera pose and the second display pose, or the first camera pose and the second display pose; Gaussian sputtering is performed on the reselected target Gaussian points according to the first display pose or the second display pose to re-render the two-dimensional image corresponding to the three-dimensional object.

8. The method according to any one of claims 1 to 5, characterized in that: Also includes: Acquire a plurality of sample data, the sample data comprising: a sample camera pose, a sample display pose, and a reference two-dimensional image of the three-dimensional object; wherein, The three-dimensional object corresponds to an initial three-dimensional Gaussian sputtering model, and the initial three-dimensional Gaussian sputtering model includes a plurality of sample first Gaussian points for describing the three-dimensional object; according to the distance information reflected by the sample camera pose and the sample display pose, a sample target Gaussian point is selected from the plurality of sample first Gaussian points; Performing Gaussian sputtering on the Gaussian points of the sample target according to the sample display posture to obtain a training two-dimensional image corresponding to the three-dimensional object; With the training two-dimensional image and the reference two-dimensional image meeting a set pixel difference threshold as a goal, the initial three-dimensional Gaussian sputtering model is trained to obtain a target three-dimensional Gaussian sputtering model.

9. An electronic device, characterized in that: include: Memory and processor; The memory is used to store a computer program; the processor is coupled to the memory and is used to execute the computer program to implement the steps in the method according to any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the processor is caused to implement the steps in the method according to any one of claims 1 to 8.

11. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the processor is caused to implement the steps in any one of the methods of claims 1-8.

Images