Rendering method, readable storage medium, program product and terminal device
By using a first processor and a second processor to share the rendering tasks in the terminal device, the problem of unbalanced hardware load in the terminal device is solved, and rendering efficiency and sorting efficiency are improved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHANGHAI SHIZHUANG INFORMATION TECHNOLOGY CO LTD
- Filing Date
- 2026-04-09
- Publication Date
- 2026-07-14
AI Technical Summary
After the terminal device was adapted to 3D Gaussian sputtering rendering technology, the hardware load was unbalanced, resulting in low rendering efficiency.
The rendering task is shared by a first processor and a second processor. The first processor generates sorting information and sends it to the second processor for rendering when the sorting conditions are met, thereby reducing the load on a single processor.
By offloading the workload to individual processors, the rendering load on a single processor is reduced, thereby improving rendering and sorting efficiency.
Smart Images
Figure CN122391458A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of terminal technology, and in particular to a rendering method, a readable storage medium, a program product, and a terminal device. Background Technology
[0002] 3D Gaussian splatting (3DGS) is a highly efficient method for representing and rendering 3D scenes. It represents a scene as a collection of numerous 3D Gaussian ellipsoids. By projecting these ellipsoids onto the image plane, sorting them by depth, and then blending them with transparency, it achieves high-quality, real-time rendering from a new perspective. Unlike traditional rendering methods based on volumetric ray path integrals, such as neural radiance fields (NeRF), 3DGS employs a rasterization-like rendering pipeline, significantly improving rendering speed and meeting the needs of real-time interactive applications.
[0003] However, when the current 3DGS rendering technology is ported to terminal devices (such as mobile phones, tablets, etc.), there are problems such as uneven hardware load, resulting in low rendering efficiency on terminal devices. Summary of the Invention
[0004] This application provides a rendering method, a readable storage medium, a program product, and a terminal device.
[0005] In a first aspect, embodiments of this application provide a rendering method applied to a terminal device. The terminal device includes a first processor and a second processor. The method includes: the first processor acquiring data to be rendered and a first rendering viewpoint corresponding to the data to be rendered; the first processor generating first sorting information of the data to be rendered based on the first rendering viewpoint when it determines that the first rendering viewpoint meets sorting conditions, and sending the first sorting information to the second processor; and the second processor rendering the data to be rendered based on the first sorting information under the first rendering viewpoint to obtain a first target rendered image.
[0006] The rendering method of this application allows different processors to process the data to be rendered when rendering images on a terminal device, thereby reducing the load on a single processor and improving rendering efficiency. Furthermore, the first processor only performs sorting to generate first sorting information if the first rendering viewpoint meets the sorting conditions. In other words, the first processor does not need to sort for every rendering viewpoint, thus improving the sorting efficiency of the terminal device.
[0007] In one possible implementation of the first aspect above, the sorting condition includes an angular deviation between the first rendering viewpoint and the second rendering viewpoint corresponding to the second sorting information previously generated by the first processor being greater than a preset angle.
[0008] In this possible implementation, if the angular deviation between the first rendering view and the second rendering view is greater than a preset angle, it can be determined that the depth values of each Gaussian ellipsoid corresponding to the first rendering view and the second rendering view change significantly. In this case, the first processor needs to redetermine the sorting information of the first rendering view in order to ensure rendering accuracy.
[0009] In one possible implementation of the first aspect described above, the method further includes: when the first rendering viewpoint does not meet the sorting conditions, the first processor sends first information to the second processor, the first information indicating that the sorting information has not changed. After detecting the first information, the second processor renders the data to be rendered based on the second sorting information under the first rendering viewpoint to generate a second target rendered image.
[0010] In this implementation, when the angular deviation between the first and second rendering views is less than a preset angle, it can be determined that the depth values of the Gaussian ellipsoids corresponding to the first and second rendering views change little. Therefore, the first processor does not need to re-sort the images; instead, it sends the first information to the second processor. After detecting the first information, the second processor can still render the image from the first view using the sorting order corresponding to the second rendering view. Consequently, the terminal device does not need to sort the images, reducing its computing power and improving rendering efficiency.
[0011] In one possible implementation of the first aspect described above, the data to be rendered includes multiple renderables and attribute data corresponding to each renderable, the attribute data including the transparency of the corresponding renderable. The second processor renders the data to be rendered based on the first sorting information from a first rendering perspective to obtain a first target rendered image, including: the second processor removing a first type of renderable from the multiple renderables to obtain multiple renderables, wherein the transparency of each renderable in the first type is lower than a preset transparency; and the second processor rendering the multiple renderables to generate the first target rendered image.
[0012] In this implementation, the lower the transparency of the rendered object, the smaller its impact on the first target rendered image. Therefore, during the rendering of the first target rendered image, the first type of rendered objects with low transparency can be filtered out, thereby improving rendering efficiency.
[0013] In one possible implementation of the first aspect above, the data to be rendered includes multiple rendering bodies. The second processor renders the data to be rendered based on the first sorting information under the first rendering perspective to obtain the first target rendering image. This includes: the second processor adjusting the projection size of each rendering body onto the first rendering plane based on preset size adjustment parameters; and the second processor projecting the adjusted rendering bodies onto the first rendering plane to generate the first target rendering image.
[0014] In one possible implementation of the first aspect above, the data to be rendered includes multiple renderables. The second processor renders the data to be rendered based on the first sorting information under the first rendering perspective to obtain a first target rendered image. This includes: the second processor downsampling the multiple renderables based on a first downsampling coefficient, wherein the first downsampling coefficient is determined based on the rate of change of the rendering perspective; and the second processor projects the downsampled multiple renderables onto the first rendering plane to generate the first target rendered image.
[0015] In this implementation, when some rendered objects are projected onto the first rendering plane, abnormal situations may occur (for example, they may degenerate into a line). Therefore, in some embodiments of this application, the projection size of the rendered objects onto the first rendering plane can be adjusted by adjusting the size adjustment parameters to ensure that each projection always has a minimum visible range, so as to ensure the rendering effect.
[0016] In one possible implementation of the first aspect described above, the rate of change of the rendering perspective is negatively correlated with the first downsampling coefficient.
[0017] In this possible implementation, a larger or faster change in rendering perspective indicates that the user is adjusting the viewpoint of the 3D scene, and therefore, the user may not be paying attention to details. In this case, the first downsampling factor can be reduced to improve rendering efficiency. Conversely, a smaller or slower change in rendering perspective indicates that the user has likely already adjusted the viewpoint of the 3D scene, and the user may be more focused on details. Therefore, rendering accuracy needs to be maintained during rendering, thereby increasing the first downsampling factor.
[0018] In one possible implementation of the first aspect above, the first sorting information of the data to be rendered based on the first rendering perspective includes: the first processor determining the depth value information of the data to be rendered under the first rendering perspective based on the vector dot product, and the first processor determining the first sorting information based on the depth value information of the data to be rendered.
[0019] In this possible implementation, the vector dot product method can focus only on the component of the data to be rendered along the depth direction of the current camera view, without having to calculate the position matrix of the data to be rendered in the three directions, thereby effectively reducing the amount of computation and improving rendering efficiency.
[0020] Secondly, this application provides a terminal device, including: a memory for storing instructions; and at least one processor for executing the instructions to cause the device to implement the method provided in the first aspect and any possible implementation of the first aspect. The beneficial effects achievable in the second aspect can be referred to the beneficial effects of the method provided in any embodiment of the first aspect, and will not be repeated here.
[0021] Thirdly, this application provides a computer-readable storage medium storing instructions that, when executed by a device, cause a computer to implement the methods provided in the first aspect and any possible implementation of the first aspect. The beneficial effects achievable in this third aspect can be referenced to the beneficial effects of the methods provided in any embodiment of the first aspect, and will not be repeated here.
[0022] Fourthly, this application provides a computer program product that stores instructions that, when executed on a device, cause the device to implement the methods provided in the first aspect and any possible implementation of the first aspect. The beneficial effects achievable in the fourth aspect can be found in the beneficial effects of the methods provided in any embodiment of the first aspect, and will not be repeated here. Attached Figure Description
[0023] Figure 1 This demonstrates a process for sputtering a 3D Gaussian scene onto a 2D image scene;
[0024] Figure 2 A schematic diagram of a terminal device is shown according to some embodiments of this application;
[0025] Figure 3A According to some embodiments of this application, an interactive flowchart of a rendering method is shown;
[0026] Figure 3B According to some embodiments of this application, the rendering process of a first rendering perspective when the sorting conditions are not met is shown;
[0027] Figure 4 A schematic diagram of a terminal device is shown according to some embodiments of this application. Detailed Implementation
[0028] The illustrative embodiments of this application include, but are not limited to, rendering methods, readable storage media, program products, and terminal devices.
[0029] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions in the embodiments of this application will be described in detail below with reference to the accompanying drawings and specific implementation methods.
[0030] As shown in the background, when 3DGS rendering technology is ported to terminal devices (such as mobile phones, tablets, etc.), there are problems such as uneven hardware load, resulting in low rendering efficiency of the terminal devices.
[0031] The following describes the rendering process of a 3DGS technology.
[0032] First, the terminal device can obtain the data to be rendered for the 3D spatial scene. The data to be rendered can be a set of 3D Gaussian ellipsoids used to represent the 3D spatial scene, as well as the attribute data of each 3D Gaussian ellipsoid.
[0033] In some embodiments, the attribute data for each 3D Gaussian ellipsoid may include the following parameters:
[0034] The position (mean) is used to indicate the center of the 3D Gaussian ellipsoid. That is, at the initial moment, the position of each 3D Gaussian ellipsoid is the position of the corresponding 3D point cloud.
[0035] The covariance matrix is used to determine the size (scaling) and orientation (rotation) of a 3D Gaussian ellipsoid in three directions.
[0036] Opacity (Alpha) is used to control the "density" of a 3D Gaussian ellipsoid.
[0037] Spherical harmonic coefficients (color) are a compact mathematical representation used to describe the color of a 3D Gaussian ellipsoid under different viewing angles and lighting conditions. This is the key to why it can be viewed as having different colors from different angles.
[0038] After the terminal device obtains the 3D Gaussian ellipsoid, it can perform differentiable rasterization on each 3D Gaussian ellipsoid when rendering the 3D scene, which corresponds to "splatting".
[0039] It's understandable that when rendering from a certain perspective (the perspective of a 3D scene can be adjusted by the user), all 3D Gaussian ellipsoids are projected onto the 2D image plane of that perspective, becoming 2D spots. A 2D spot is an elliptical region formed by the projection of a 3D Gaussian ellipsoid onto the 2D image plane. The terminal device can sort and blend the 2D spots projected from each 3D Gaussian ellipsoid according to depth (from back to front). Furthermore, the final color of a pixel on the 2D image plane is the result of blending the colors of all the 2D spots projected onto that pixel location according to their opacity. After sorting and blending the 2D spots, the terminal device can render an image of the 3D spatial scene from the current perspective.
[0040] In the embodiments of this application, the data to be rendered for the 3D spatial scene can be pre-trained by the server of the corresponding application. That is, the terminal device only needs to obtain the data to be rendered for the 3D spatial scene from the server, that is, it only needs to obtain the set of pre-trained 3D Gaussian ellipsoids that can represent the 3D spatial scene from the server.
[0041] The following describes the process of training a set of 3D Gaussian ellipsoids corresponding to a 3D spatial scene on the server.
[0042] For example, a server can acquire point cloud data in 3D space, which can be generated by the server based on an image. For instance, the server can generate a sparse 3D point cloud from an input image using the Structure from Motion (SfM) method. The positions of each point in the 3D point cloud are the initial centers of a 3D Gaussian ellipsoid; that is, the initial properties of the 3D Gaussian ellipsoid are determined first. Then, the server renders the 3D scene from a specific rendering angle, projecting the initial 3D Gaussian ellipsoid onto the rendering plane corresponding to that angle.
[0043] In the sputtering rendering process described above, the blending of multiple 2D specks is differentiable. Therefore, the server can calculate the error (loss) between the final rendered image and the actual input image, and backpropagate this error to automatically adjust all attributes (position, size, color, opacity) of each Gaussian ellipsoid, thereby training the server's sputtering rendering process. During training, the server can also dynamically add or remove 3D Gaussian ellipsoids.
[0044] For example, in areas lacking detail (where errors are large), a large 3D Gaussian ellipsoid can be split into several smaller 3D Gaussian ellipsoids. In areas of redundancy or where the 3D Gaussian ellipsoid is large, unnecessary 3D Gaussian ellipsoids can be deleted or merged.
[0045] The following describes a workflow for training a 3D Gaussian ellipsoid for a 3D scene on a server.
[0046] First, the server can acquire a set of static scene images taken from different perspectives, including camera poses.
[0047] Then, the server processes the set of images using SfM to obtain point cloud data and creates an isotropic 3D Gaussian ellipsoid for each point in the point cloud.
[0048] Next, we can optimize and iterate on each 3D Gaussian ellipsoid to train the properties of each 3D Gaussian ellipsoid that conform to the current 3D Gaussian scene.
[0049] For example, a server can "splash" all 3D Gaussian ellipsoids into a 2D image from a certain training perspective. Then, it compares the error between this 2D image and the real image. The server can backpropagate the error to update the properties (position, scale, rotation, color, opacity, etc.) of all 3D Gaussian ellipsoids. In some embodiments, the server can also adaptively adjust, cloning or deleting 3D Gaussian ellipsoids based on the optimization state at regular iterations. After a certain number of optimization iterations on each 3D Gaussian ellipsoid of the 3D scene, or after the error is less than a preset error range, the terminal device can output a trained 3D Gaussian scene representation, for example, containing approximately 500,000 to millions of 3D Gaussian ellipsoids with optimized properties.
[0050] It's understandable that once the server trains the 3D Gaussian scene representation, the properties of each 3D Gaussian ellipsoid in the scene are determined. The server can then send the trained 3D Gaussian ellipsoids and their corresponding attribute data to the terminal device. The terminal device can then directly render the 3D spatial scene based on these data. In other words, during the subsequent rendering process, the terminal device only needs to project each 3D Gaussian ellipsoid onto the corresponding 2D plane, perform depth sorting, and render the scene. There's no need for the neural network to re-infer, making the process extremely fast.
[0051] As an example, Figure 1 This demonstrates a process for sputtering a 3D Gaussian scene onto a 2D image scene.
[0052] Reference Figure 1 A 3D Gaussian scene is a 3D space composed of the X, Y, and Z directions. For example, the 3D Gaussian scene trained by the terminal device includes four 3D Gaussian ellipsoids, namely G1, G2, G3, and G4, where G1 is located at (X1, Y1, Z1), G2 at (X2, Y2, Z2), G3 at (X3, Y3, Z3), and G4 at (X4, Y4, Z4).
[0053] During sputtering rendering, the terminal device can determine the viewpoint to be rendered, for example, the current viewpoint to be rendered is the Z-axis viewpoint. The terminal device can then project four 3D Gaussian ellipsoids onto the 2D plane corresponding to the Z-axis viewpoint, i.e., the XOY plane. During sputtering, the terminal device needs to sort the 3D Gaussian ellipsoids projected to the same pixel by depth. For example, when the projection viewpoint is the Z-axis, the depth is the height of each 3D Gaussian ellipsoid in the Z-axis.
[0054] Reference Figure 1After G1 is projected onto the XOY plane, the corresponding 2D spot is B1, located in a single pixel region. After G2 is projected onto the XOY plane, the corresponding 2D spot is B2; after G3 is projected onto the XOY plane, the corresponding 2D spot is B3; and after G4 is projected onto the XOY plane, the corresponding 2D spot is B4. Furthermore, B2, B3, and B4 are located in the same pixel region. When B2, B3, and B4 are projected onto the XOY plane, B3 and B4 overlap. Therefore, the terminal device needs to first sort B3 and B4 based on their depths, and then mix the colors of B3 and B4 to generate the final color of the overlapping B3 and B4 within the pixel region.
[0055] For example, G4 has the lowest Z-direction value, G3 has the highest Z-direction value, and G2 is in the middle. Therefore, the depth order of B2, B3, and B4 is B4, B2, and B3. After depth ordering B2, B3, and B4, the terminal device can blend the overlapping parts of B2, B3, and B4 (e.g., B3 and B4) based on the opacity (Alpha) and spherical harmonic coefficients (color) of G2, G3, and G4 to render that pixel area and generate a 2D image from the current viewpoint.
[0056] However, the aforementioned sputtering rendering process (e.g., the blending and sorting of individual 2D specks and rendering) is all executed within the same processor of the terminal device. For example, taking the terminal device's graphics processing unit (GPU) as an example, when rendering each frame of an image, the terminal device needs to first calculate the sorting depth of each 3D Gaussian ellipsoid and perform depth sorting. After sorting, the terminal device then renders the image using the GPU. Thus, the GPU load on the terminal device is high. If the performance of the terminal device's processor is poor, the rendering process will be lengthy, resulting in poor rendering efficiency for the terminal device.
[0057] To address the aforementioned problems, this application provides a rendering method. The terminal device includes a first processor (e.g., a central processing unit (CPU)) and a second processor (e.g., a GPU). The first processor acquires data to be rendered and a corresponding first rendering viewpoint. Furthermore, if the first rendering viewpoint satisfies a sorting condition, the first processor generates first sorting information for the data to be rendered based on the first rendering viewpoint and sends this first sorting information to the second processor. The second processor renders the data to be rendered based on the first sorting information from the first rendering viewpoint to generate a first target rendered image.
[0058] If the first processor determines that the first rendering view does not meet the sorting conditions, the second processor renders the data to be rendered under the first rendering view based on the second sorting information to generate the second target rendering image.
[0059] By employing the above approach, different processors can handle the data to be rendered when the terminal device renders images, thereby reducing the load on a single processor and improving rendering efficiency. Furthermore, the first processor only performs sorting to generate the first sorting information if the first rendering viewpoint meets the sorting conditions. In other words, the first processor does not need to sort for every rendering viewpoint, thus improving the sorting efficiency of the terminal device.
[0060] In some embodiments of this application, the sorting condition includes that the angular deviation between the first rendering viewpoint and the second rendering viewpoint corresponding to the second sorting information previously generated by the first processor is greater than a preset angle, wherein the preset angle can be any value from 0° to 5°. It can be understood that when the angular deviation between the first and second rendering viewpoints is less than the preset angle, it can be determined that the depth values of the various Gaussian ellipsoids corresponding to the first and second rendering viewpoints change relatively little. Thus, the image of the first viewpoint can still be rendered using the sorting order corresponding to the second rendering viewpoint, thereby eliminating the need for sorting by the terminal device, reducing the computing power of the terminal device, and improving the rendering efficiency of the image.
[0061] The terminal device in the embodiments of this application is described below.
[0062] The method provided in this application can be applied to any terminal device capable of 3DGS rendering, including but not limited to mobile stations (MS) and mobile terminals (MT). For example, the terminal device can be a virtual reality (VR) device, an augmented reality (AR) device, a terminal in industrial control, a terminal in self-driving, a terminal in remote medical surgery, a terminal in a smart grid, a terminal in transportation safety, a terminal in a smart city, a terminal in a smart home, and so on. This application does not limit the specific form of the terminal device.
[0063] For example, Figure 2 According to some embodiments of this application, a schematic diagram of a terminal device is shown.
[0064] Reference Figure 2 The terminal device 100 includes a first processor and a second processor. The first processor includes a first storage space, and the second processor includes a second storage space. In embodiments of this application, the first processor is exemplified by a CPU, and the second processor is exemplified by a GPU. The first storage space can be system memory (e.g., random access memory (RAM)) in the CPU. The second storage space can be video memory (e.g., video random access memory (VRAM)) in the GPU.
[0065] In some embodiments of this application, the terminal device detects that a user has opened a first application and displays a 3D scene on the current display interface. As mentioned above, when rendering the 3D scene, the terminal device can obtain the attribute data of the corresponding 3D Gaussian ellipsoid. It can be understood that the 3D Gaussian ellipsoid and its attribute data can be used as the rendering data in the embodiments of this application, and the training process of the 3D Gaussian ellipsoid can be executed by the server. After the terminal device obtains the attribute data of the 3D Gaussian ellipsoid from the server, it can store the attribute data of the 3D Gaussian ellipsoid in the second storage space of the second processor, so that when the viewpoint of the 3D scene changes (e.g., the user rotates the viewpoint of the 3D scene), the second processor can retrieve the attribute data of the 3D Gaussian ellipsoid from the second storage space to render the image of the corresponding viewpoint. In this way, the second processor does not need to re-infer the attribute data of the 3D Gaussian ellipsoid, thereby improving the rendering efficiency of the 3D scene.
[0066] The following describes the process by which the terminal device renders images of 3D scenes from various perspectives after obtaining the attribute data of the 3D Gaussian ellipsoid.
[0067] As an example, Figure 3A According to some embodiments of this application, an interactive flowchart of a rendering method is shown.
[0068] like Figure 3A As shown, the process includes:
[0069] S301, the first processor acquires the data to be rendered and the first rendering perspective corresponding to the data to be rendered.
[0070] In some embodiments of this application, a user can operate on the display interface of a first application to adjust the 3D scene. For example, the first application of the terminal device can detect user control events on the display interface of the first application, wherein the control events may include touch events, voice control events, gesture control events, or motion control events, etc. The first application can respond to the control events to adjust the viewpoint of the 3D scene or zoom the 3D scene, etc.
[0071] In some embodiments, the first processor may obtain the data to be rendered from the second processor. The data to be rendered may be the attribute data of a 3D Gaussian ellipsoid corresponding to a 3D scene.
[0072] For example, after the first application detects that a user has opened a 3D scene, it can determine the identifier of the 3D scene to be displayed based on the user's control events. Then, the first application can transmit the 3D scene identifier to the rendering engine of the terminal device through the rendering interface. After obtaining the scene identifier, the rendering engine can parse the data to be rendered for the 3D scene, and then the rendering engine can schedule the second processor to load the data to be rendered for the 3D scene through its own scheduler. For example, the second processor loads the attribute data of the 3D Gaussian ellipsoid corresponding to the 3D scene (as an example of the data to be rendered) into the second memory. Furthermore, the second processor can also synchronize the attribute data of the 3D Gaussian ellipsoid to the first processor. For example, the second processor can send the attribute data of the 3D Gaussian ellipsoid to the first processor through the system bus. It can be understood that the first processor only needs to obtain the attribute data of the 3D Gaussian ellipsoid from the second processor once. After obtaining the attribute data of the 3D Gaussian ellipsoid, the first processor can store the attribute data of the 3D Gaussian ellipsoid into the first storage space for sorting the 3D Gaussian ellipsoid. After the first processor completes the sorting, it sends the initial sorting information to the second processor. The second processor can then perform sputter rendering on the 3D Gaussian ellipsoid based on the initial sorting information, thereby rendering the initial image corresponding to the 3D scene.
[0073] After detecting a change in the viewpoint of the current 3D scene, the first application can call the rendering engine through the rendering interface and pass the raw operation data of the user detected by the application (e.g., changes in rotation angle, amount of movement, scaling ratio, etc.) to the rendering engine. Then, the rendering engine calls the first processor to calculate new camera parameters (e.g., camera position, rotation, scaling) based on the raw operation data, and then determines the first rendering viewpoint. It can be understood that the first rendering viewpoint can be the direction that the camera is facing from its current position.
[0074] S302, the first processor generates first sorting information of the data to be rendered based on the first rendering perspective, provided that the first rendering view meets the sorting conditions.
[0075] In some embodiments of this application, after determining the first rendering viewpoint, the first processor can determine whether the first rendering viewpoint meets the sorting conditions. The sorting conditions may include the angular deviation between the first rendering viewpoint and the second rendering viewpoint corresponding to the second sorting information previously generated by the first processor being greater than a preset angle. In some embodiments of this application, the preset angle can be any value between 0° and 5°.
[0076] In some embodiments of this application, when the first processor determines that the first rendering view meets the sorting conditions, it can generate first sorting information of the data to be rendered based on the first rendering view.
[0077] It is understandable that if the sorting conditions are met in the first rendering view, it means that the sorting of the 3D Gaussian ellipsoids corresponding to the first rendering view is significantly different from the sorting of the 3D Gaussian ellipsoids corresponding to the second rendering view. Therefore, the first processor needs to re-sort the 3D Gaussian ellipsoids.
[0078] In some embodiments of this application, the first processor may determine the depth information of the data to be rendered from the first rendering viewpoint based on the vector dot product, and then the first processor may determine the first sorting information based on the depth information of the data to be rendered.
[0079] It can be understood that the first rendering viewpoint can be the camera viewpoint of the current 3D scene. The first processor can determine the depth information of the data to be rendered based on the vector dot product method according to the camera viewpoint of the current 3D scene. For example, the first processor can determine the depth information of the corresponding 3D Gaussian ellipsoid relative to the current camera viewpoint based on the position in the attribute information of each 3D Gaussian ellipsoid. Then, the first processor sorts the depth information of each 3D Gaussian ellipsoid to determine the first sorting information.
[0080] It is understandable that by using vector dot product, we can focus only on the depth component of each 3D Gaussian ellipsoid along the current camera viewpoint, without having to calculate the position matrix of the 3D Gaussian ellipsoid in the three directions, thus effectively reducing the amount of computation and improving the efficiency of sorting calculation.
[0081] S303, the first processor sends the first sorting information to the second processor.
[0082] After determining the first sorting information for the first rendering viewpoint, the first processor can send the first sorting information to the second processor via the system bus. Furthermore, the first processor can also send information indicating sorting changes to the second processor, so that the second processor can determine how to render the image based on the first sorting information sent by the first processor.
[0083] In some embodiments of this application, the information indicating a change in sorting can be a corresponding flag bit. For example, a flag bit of "0" indicates that the sorting information has not changed, and a flag bit of "1" indicates that the sorting information has changed. Furthermore, after the first processor sends the flag bit to the second processor, the second processor can determine whether the sorting information has changed based on the flag bit. In other embodiments, the information indicating a change in sorting can also be in other forms; this application does not limit the information indicating a change in sorting.
[0084] Understandably, the first processor also needs to send the first rendering view to the second processor so that the second processor can sputter each 3D Gaussian ellipsoid according to the first rendering view.
[0085] S304, the second processor renders the data to be rendered from the first rendering perspective based on the first sorting information to obtain the first target rendering image.
[0086] In some embodiments of this application, the second processor can determine whether the sorting has changed based on a flag bit. For example, if the second processor detects that the flag bit is "1", it can determine that the first processor has generated first sorting information according to the first rendering view. Then the second processor can update the previously stored sorting information (e.g., the second sorting information corresponding to the second rendering view) with the first sorting information, and render the data to be rendered under the first rendering view according to the first sorting information, thereby obtaining the first target rendered image.
[0087] In some embodiments of this application, if the first processor determines that the first rendering perspective does not meet the sorting conditions, then no sorting is required, and the second processor can directly render the data to be rendered based on the first rendering perspective.
[0088] For example, Figure 3B According to some embodiments of this application, the rendering process of a first rendering perspective when sorting conditions are not met is shown.
[0089] Reference Figure 3B The process includes:
[0090] S305, the first processor determines that the first rendering view does not meet the sorting conditions.
[0091] It is understood that the sorting criteria may include an angular deviation between the first rendering viewpoint and the second rendering viewpoint that is greater than a preset angle. Therefore, in some embodiments of this application, if the angular deviation between the first rendering viewpoint and the second rendering viewpoint is less than or equal to the preset angle, the first processor can determine that the first rendering viewpoint does not meet the sorting criteria.
[0092] S306, the first processor sends first information to the second processor, the first information being used to indicate that the sorting information has not changed.
[0093] In some embodiments of this application, if the first processor detects that the first rendering view does not meet the sorting conditions, it may not be necessary to sort the data to be rendered, thereby improving the sorting efficiency of the terminal device.
[0094] It is understandable that if the first rendering view does not meet the sorting conditions, and the angular deviation between the first and second rendering views is less than or equal to a preset angle, then the sorting information of the 3D Gaussian ellipsoids corresponding to the first rendering view and the sorting information of the 3D Gaussian ellipsoids corresponding to the second rendering view are relatively similar. Thus, the terminal can still use the sorting order corresponding to the second rendering view to render the image from the first view, thereby eliminating the need for sorting by the terminal device, reducing the computing power of the terminal device, and improving image rendering efficiency.
[0095] At this point, the first processor can send first information to the second processor. For example, the first processor can set a flag bit (as an example of first information) to "0" and send it to the second processor to indicate that the sorting information has not changed.
[0096] S307, after the second processor detects the first information, it renders the data to be rendered from the first rendering perspective based on the second sorting information to generate the second target rendering image.
[0097] In some embodiments of this application, when the second processor detects a flag bit of "0", it can determine that the sorting information has not changed. The second processor can then render the data to be rendered from the first rendering perspective based on the second sorting information previously obtained from the first processor, thereby obtaining the second target image. Thus, a sorting process is unnecessary during image rendering from the first rendering perspective, thereby improving rendering efficiency.
[0098] It is understandable that, during the rendering of the image from the first rendering perspective, the sorting process is configured to be executed on the first processor, thereby reducing the load on the second processor and achieving a load balance between the first and second processors. Even with lower performance on the second processor, the rendering effect of the 3D scene can still be guaranteed. Furthermore, if the first rendering perspective of the 3D scene does not meet the sorting conditions, the first processor does not need to sort the data to be rendered, and the second processor can reuse the second sorting information from rendering the image from the second rendering perspective, thereby improving rendering efficiency.
[0099] The following describes the process by which the second processor in this embodiment renders the data to be rendered.
[0100] In some embodiments of this application, the second processor may downsample the data to be rendered before rendering it, thereby reducing the amount of data to be rendered and improving rendering efficiency. For example, the second processor downsamples multiple rendering objects based on a first downsampling coefficient to obtain multiple rendering objects to be projected, wherein the first downsampling coefficient is determined based on the rate of change of the rendering viewpoint.
[0101] In some embodiments of this application, the second processor can perform texture downsampling on the data to be rendered. For example, the second processor can encode the attribute data of a 3D Gaussian ellipsoid into a 2D image (hereinafter referred to as the encoded image). Exemplarily, the attribute data of each 3D Gaussian ellipsoid can occupy a 4×4 pixel area, and the pixel values of these pixels are no longer ordinary colors, but encoded Gaussian attributes. Furthermore, the second processor can perform downsampling processing on the encoded image. In some embodiments of this application, the downsampling coefficient of the second processor for texture downsampling of the data to be rendered can be determined based on the rate of change of the rendering viewpoint. For example, when the angle between the first rendering viewpoint and the second rendering viewpoint is greater than 30°, the first downsampling coefficient of texture downsampling is adjusted to 0.5 to 0.6. That is, the encoded image is adjusted to 0.5 to 0.6 times its original size. When the angle between the first rendering viewpoint and the second rendering viewpoint is greater than, less than, or equal to 30°, the first downsampling coefficient of texture downsampling is adjusted to 0.7 to 0.9. That is, the encoded image is adjusted to 0.7 to 0.9 times its original size. In other words, the rate of change of the rendering viewpoint is negatively correlated with the first downsampling coefficient. It's understandable that a larger or faster change in rendering perspective indicates the user is adjusting their viewpoint within the 3D scene, and therefore may not be focused on details. In this case, the downsampling factor can be reduced to improve rendering efficiency. Conversely, a smaller or slower change in rendering perspective suggests the user has likely already adjusted their viewpoint and is more focused on details. Therefore, rendering accuracy must be maintained; that is, the downsampling factor should not be too small to ensure the quality of the image rendered on the terminal device, thus providing a better visual experience for the user.
[0102] In other embodiments, the first downsampling coefficient may be set to other values depending on the rate or magnitude of change of the rendering viewpoint. The embodiments of this application do not limit the size of the first downsampling coefficient.
[0103] In some embodiments of this application, the attribute data of the 3D Gaussian ellipsoid also includes size adjustment parameters. Before the second processor renders the 3D Gaussian ellipsoid to generate the first target rendering image, it can also adjust the projection size of each 3D Gaussian ellipsoid onto the first rendering plane based on the size adjustment parameters to generate the target rendering image.
[0104] It is understood that the projection size of a 3D Gaussian ellipsoid onto the first rendering plane is related to the size of the corresponding 3D Gaussian ellipsoid and the projection angle. However, when some 3D Gaussian ellipsoids are projected onto the first rendering plane, abnormal situations may occur (for example, they may degenerate into a line). Therefore, in some embodiments of this application, the projection size of each 3D Gaussian ellipsoid onto the first rendering plane can be adjusted by size adjustment parameters to ensure that each projection always has a minimum visible range, thereby guaranteeing the rendering effect. In the embodiments of this application, the size adjustment parameters can be preset parameters, which can be set to balance rendering efficiency and rendering effect.
[0105] In some embodiments of this application, the second processor can also perform vertex region downsampling on the data to be rendered to reduce the number of 3D Gaussian ellipsoids, thereby improving rendering efficiency. Vertex region downsampling involves dividing the 3D scene space into a mesh and then merging multiple 3D Gaussian ellipsoids within the same mesh. The mesh size can be adjusted based on the viewing distance. For example, at close range, a small mesh can be used for downsampling to preserve detail; at medium range, a medium mesh can be used; and at long range, a large mesh can be used, thus improving rendering efficiency. This allows the terminal device to maintain rendering accuracy while improving rendering efficiency.
[0106] In some embodiments of this application, vertex region downsampling can also be achieved by dynamically adjusting the projection scaling factor pScale parameter of each 3D Gaussian ellipsoid. The pScale parameter affects the size of the projection of the corresponding 3D Gaussian ellipsoid onto the first rendering plane. It can be understood that the larger the pScale parameter, the larger the projection of the 3D Gaussian ellipsoid onto the first rendering plane, and the more computing power the second processor needs to consume when rendering this projection.
[0107] In the embodiments of this application, the pScale parameter can be determined based on the camera projection scaling factor and the distance scaling factor. For example, pScale = projScale × distanceScale, where projScale is the camera projection scaling factor, affecting the scaling of the global 3D Gaussian ellipsoid. distanceScale is the distance scaling factor; the farther the 3D Gaussian ellipsoid is from the first rendering plane, the smaller distanceScale becomes. distanceScale can be dynamically adjusted in the range of 0.7 to 1.0 using the smoothstep function (an interpolation function) based on the view distance. In other words, the pScale parameter of the 3D Gaussian ellipsoid is related to the distance from the 3D Gaussian ellipsoid to the first rendering plane. The farther the 3D Gaussian ellipsoid is from the first rendering plane, the smaller the pScale parameter, and the smaller the projected size of the 3D Gaussian ellipsoid onto the first rendering plane. It is understandable that the greater the distance between the 3D Gaussian ellipsoid and the first rendering plane, the smaller the impact of the 3D Gaussian ellipsoid on the rendered image. Therefore, the size of the 3D Gaussian ellipsoid that is far from the first rendering plane can be reduced to improve rendering efficiency.
[0108] In some embodiments of this application, after the image to be rendered by the second processor is downsampled, it can also remove 3D Gaussian ellipsoids outside the first rendering plane projected onto the first rendering viewpoint based on the view frustum culling technique, thereby reducing invalid rendering.
[0109] For example, the second processor of the terminal device can determine the current position of the camera in the 3D space scene (eye), the position of the target point observed by the current camera in the 3D space scene (center) (or the current camera's line of sight dir = center - eye), and the upward direction of the current 3D space scene (up) based on the first rendering viewpoint. Then, the second processor can determine the view transformation matrix based on eye, center, and up, and transform the current 3D space scene into camera space using the view transformation matrix.
[0110] Then, the second processor converts the camera space into clip space using projection transformation parameters. These parameters include the field of view (fov) (determining the range of the 3D scene visible from the camera space), the aspect ratio (the ratio of the height to the width of the first rendering plane, determined by the screen resolution), the near plane distance (corresponding to the closest visible area in camera space, also the position of the first rendering plane), and the far plane distance (corresponding to the farthest visible distance in camera space). The second processor determines the projection matrix based on fov, aspect, near, and far, thereby converting the camera space into clip space.
[0111] After determining the clipping space, the second processor can transform the clipping space to the normalized device coordinates (NDC) space using perspective division. Perspective division involves dividing the x, y, and z components of the clipping space by the depth component w. In other words, the clipping space is normalized using the depth component as the standard. For example, the coordinates of the clipping space point (x1, y1, z1, w1) after transformation to NDC space using perspective division become (x1 / w1, y1 / w1, z1 / w1). Furthermore, in some embodiments of this application, the NDC space is defined as [-1, 1] along the X direction, [-1, 1] along the Y direction, and [0, 1] along the Z direction. That is, the NDC space is a space based on the depth value w; only when the x, y, and z coordinates of a point in the clipping space are all less than the depth coordinate w can it be projected into the NDC space. In the embodiments of this application, a 3D Gaussian ellipsoid that can be projected into the NDC space can be mapped onto the first rendering plane. Furthermore, the second processor retains 3D Gaussian ellipsoids with coordinates within the range of [-1.1, 1.1] × [-1.1, 1.1] × [0.25, +∞) (hereinafter referred to as the view frustum range), where 0.25 is the depth value. That is, the second processor can cull 3D Gaussian ellipsoids outside the view frustum range to reduce invalid rendering. It can be understood that the space of the view frustum range for retaining the 3D Gaussian ellipsoid is slightly larger than the NDC space in the X and Y directions, thus improving anti-aliasing and eliminating jagged edges.
[0112] In some embodiments of this application, after the second processor completes the frustum culling, it can project a 3D Gaussian ellipsoid located in the range of [-1.1,1.1]×[-1.1,1.1]×[0.25,+∞) onto the first rendering plane, and then perform the subsequent rendering process.
[0113] It is understood that in the embodiments of this application, each 3D Gaussian ellipsoid (as an instance of a rendering body) includes attribute data, which includes the transparency of the corresponding 3D Gaussian ellipsoid. During the rendering process of the data to be rendered, the second processor can extract multiple 3D Gaussian ellipsoids (as an example of a rendering body) from multiple rendering areas to obtain multiple 3D Gaussian ellipsoids to be rendered (as an example of a rendering body), wherein the transparency of each 3D Gaussian ellipsoid in the first type of 3D Gaussian ellipsoid is lower than a preset transparency. Then, the second processor renders the multiple 3D Gaussian ellipsoids to be rendered to generate a first target rendering image.
[0114] It is understood that during the projection process of the second processor, the multiple rendered volumes can be 3D Gaussian ellipsoids within the view frustum. Before the second processor projects the 3D Gaussian ellipsoids within the view frustum onto the first rendering plane, the 3D Gaussian ellipsoids can be filtered based on their transparency. For example, in embodiments of this application, the second processor can remove 3D Gaussian ellipsoids with transparency lower than 1 / 128 (as an example of a preset transparency) within the view frustum (as an example of a first type of 3D Gaussian ellipsoid) to obtain multiple 3D Gaussian ellipsoids to be rendered. Then, the second processor renders the multiple 3D Gaussian ellipsoids to be rendered to generate a first target rendered image. It is understood that the lower the transparency of the 3D Gaussian ellipsoid, the smaller the impact of the corresponding 3D Gaussian ellipsoid on the first target rendered image can be determined, and 3D Gaussian ellipsoids with low transparency can be filtered out, thereby improving rendering efficiency. In other embodiments, the preset transparency can also be other values; embodiments of this application do not limit the size of the preset transparency. It is understandable that after downsampling, frustum culling, and transparency filtering, the 3D Gaussian ellipsoid in the data to be rendered can reduce the amount of data significantly. This improves the efficiency of the second processor in rendering the first target image. Furthermore, the performance requirements of the second processor can be reduced, thus ensuring good rendering results when porting 3DGS rendering technology to terminal devices (such as mobile phones, tablets, and other mobile terminals).
[0115] In some embodiments of this application, different terminal device operating systems may be configured with different rendering engines. Therefore, when porting the 3DGS rendering technology in the embodiments of this application to terminal devices, it is necessary to consider the differences in rendering engines of different terminal devices in order to ensure that the rendering method in the embodiments of this application can maintain good rendering effect on terminal devices with different operating systems.
[0116] As an example, embodiments of this application are based on Android. TM System and iOS TM Taking the system as an example, this paper describes the process of efficient real-time rendering of the rendering method in different systems according to the embodiments of this application. Among them, Android TM The system's rendering engine is based on the OpenGL ES engine, for example. (iOS) TM The system's rendering engine takes the Metal engine as an example. In the embodiments of this application, some logic of the OpenGL ES engine can be converted into Metal engine-compatible syntax, thereby maintaining the consistency of the core algorithm and ensuring the uniformity of the visual effects of 3D scenes rendered by different systems.
[0117] It is understandable that the rendering interface of a terminal device can shield system differences, thereby providing a unified calling interface for applications on the terminal device. The rendering engines configured on terminal devices of different systems can be bridged with the corresponding functions on the rendering interface through their respective engine's bridging functions, thereby enabling the first application to call the rendering engine through the rendering interface.
[0118] For example, in the case of Android terminal devices TM In the case of the system, Android TM The system's OpenGLES rendering engine's application programming interface (API) can be bridged with corresponding functions in the rendering interface through bridging functions, allowing the first application to call the OpenGLES rendering engine to render new images via the rendering interface. Similarly, on the terminal device iOS... TM In the case of the system, the first application can call the Metal API through the rendering interface.
[0119] For example, in the process described above of adjusting the size of the projection of a 3D Gaussian ellipsoid onto the first rendering plane using size adjustment parameters, Android TM The system's terminal devices can use the rendering interface to call the OpenGL ES vertex shader for adjustments, in iOS. TM The system's terminal devices can be adjusted via the rendering interface by calling the vertex shader through Metal.
[0120] Alternatively, during the process of cone removal, Android TM The system's terminal devices can implement view frustum culling by calling the geometry shader of OpenGL ES through the rendering interface, in iOS. TM The system's terminal devices can use the rendering interface to call Metal's geometry shader to perform view frustum culling.
[0121] Alternatively, during the aforementioned transparency filtering process, Android... TM The system's terminal devices can implement transparency filtering by calling OpenGL ES fragment shaders through the rendering interface. (iOS) TM The system's terminal devices can use the rendering interface to call Metal's fragment shader to achieve transparency filtering.
[0122] In some embodiments of this application, different rendering engines can use a unified depth calculation method to determine the depth of a 3D Gaussian ellipsoid when scheduling the CPU to calculate the depth data of the 3D Gaussian ellipsoid through a scheduler. For example, the depth value of each 3D Gaussian ellipsoid can be calculated uniformly using the vector dot product method. It is understood that since the vector dot product method can extract the components of the 3D Gaussian ellipsoid along the depth direction, it is not necessary to calculate the position matrix of the 3D Gaussian ellipsoid in the three directions of 3D space when calculating the depth value of each 3D Gaussian ellipsoid, thereby effectively reducing the computational load.
[0123] When calculating depth values, if the terminal device is Android... TM The system can then incorporate a single-instruction multiple-data (SMD) instruction set and thread pool to process 3D Gaussian ellipsoid data in parallel, processing four vectors at a time to improve computational efficiency. It is understood that the register width is limited to 128 bits, and the data width for each 3D Gaussian ellipsoid is limited to 32 bits, theoretically allowing for a maximum of four vectors to be processed. In other embodiments, a different number of vectors can be processed at a time; however, the embodiments of this application do not limit the number of vectors processed at one time.
[0124] In the embodiments of this application, after determining the depth values of each 3D Gaussian ellipsoid, the 3D Gaussian ellipsoids can be sorted based on their depth values. In the embodiments of this application, to improve sorting efficiency... (Android) TM The system's terminal devices can use threading building blocks (TBB) (a threading tool library) to implement depth-descending sorting through tbb::parallel_sort (a sorting function in TBB) and extract the position data of the index and 3D Gaussian ellipsoid in parallel through tbb::parallel_for (a loop function in TBB), thereby shortening the sorting time.
[0125] If the terminal device is iOS TM The system can then adopt iOS based on the Metal performance optimization principle. TM The system-recommended vector processing interface, combined with iOS TM The system hardware characteristics optimize the computational process. Specifically, iOS... TM The system uses std::sort (a sorting function) to implement depth-descending sorting, in conjunction with iOS. TM The system thread scheduling feature optimizes the index extraction logic (used to extract position data of different 3D Gaussian ellipsoids) to ensure sorting efficiency and system compatibility.
[0126] It is understood that the rendering method in this application embodiment supports operation on different systems, thereby improving the adaptability of 3DGS technology on different terminal device systems. Furthermore, the rendering method of this application can reasonably allocate CPU and GPU utilization, avoid overloading of a single hardware component, reduce device heat generation, and adapt to the hardware characteristics of mobile terminals.
[0127] The terminal device in the embodiments of this application is described below.
[0128] For example, Figure 4 A schematic diagram of a terminal device is shown according to some embodiments of this application.
[0129] The terminal device 100 can be used to implement the rendering methods in the foregoing embodiments.
[0130] like Figure 4 As shown, the terminal device 100 includes one or more processors 101, system memory 102, non-volatile memory (NVM) 103, communication interface 104, input / output device 105, and system control logic unit 106 for coupling the processor 101, system memory 102, non-volatile memory 103, communication interface 104, and input / output (I / O) device 105. Wherein:
[0131] Processor 101 may include one or more processing units, such as processing modules or circuits that include CPUs, GPUs, digital signal processors (DSPs), microprocessors (MCUs), artificial intelligence (AI) processors, field programmable gate arrays (FPGAs), neural network processing units (NPUs), etc., and may include one or more single-core or multi-core processors. In the embodiments of this application, the sorting process in the foregoing embodiments can be executed by the CPU, and the rendering process can be executed by the GPU.
[0132] System memory 102 is volatile memory, such as random-access memory (RAM), double data rate synchronous dynamic random access memory (DDR SDRAM), etc. System memory is used for temporary storage of data and / or instructions; for example, in some embodiments, system memory 102 may be used to store the aforementioned data to be rendered.
[0133] The non-volatile memory 103 may include one or more tangible, non-transitory computer-readable media for storing data and / or instructions. In some embodiments, the non-volatile memory 103 may include any suitable non-volatile memory and / or any suitable non-volatile storage device, such as a hard disk drive (HDD), compact disc (CD), digital versatile disc (DVD), solid-state drive (SSD), etc. In some embodiments, the non-volatile memory 103 may also be a removable storage medium, such as a secure digital (SD) memory card. In other embodiments, the non-volatile memory 103 may be used to store instructions for the rendering methods provided in the foregoing embodiments.
[0134] Specifically, system memory 102 and non-volatile memory 103 may each include a temporary copy and a permanent copy of instruction 107. Instruction 107 may include, when executed by at least one of processors 101, causing terminal device 100 to implement, for example, the rendering methods provided in the embodiments of this application.
[0135] The communication interface 104 may include a transceiver for providing a wired or wireless communication interface for the terminal device 100, thereby enabling communication with any other suitable device via one or more networks. In some embodiments, the communication interface 104 may be integrated into other components of the terminal device 100, for example, the communication interface 104 may be integrated into the processor 101. In some embodiments, the terminal device 100 may communicate with other devices through the communication interface 104.
[0136] The input / output (I / O) device 105 may include input devices and output devices, and users can interact with the terminal device 100 through the input / output (I / O) device 105.
[0137] The system control logic unit 106 may include any suitable interface controller to provide any suitable interface to other modules of the terminal device 100. For example, in some embodiments, the system control logic unit 106 may include one or more memory controllers to provide an interface to the system memory 102 and the non-volatile memory 103.
[0138] In some embodiments, at least one of the processors 101 may be packaged together with the logic of one or more controllers for the system control logic unit 106 to form a system in package (SiP). In other embodiments, at least one of the processors 101 may also be integrated on the same chip with the logic of one or more controllers for the system control logic unit 106 to form a system-on-chip (SoC).
[0139] Understandable. Figure 4 The structure of the terminal device 100 shown is merely an example. In other embodiments, the terminal device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.
[0140] It is understood that the terminal device 100 may include, but is not limited to, mobile phones, in-vehicle systems, smartwatches, netbooks, etc., and this application embodiment does not limit it.
[0141] This application also provides a program product that, when executed on a terminal device, enables the terminal device to implement the methods provided in the foregoing embodiments.
[0142] This application also provides a readable storage medium storing one or more programs, which, when executed by a terminal device, enable the terminal device to implement the methods provided in the foregoing embodiments.
[0143] Various embodiments of the mechanisms disclosed in this application can be implemented in hardware, software, firmware, or combinations of these implementation methods. Embodiments of this application can be implemented as computer programs or program code executable on a programmable system, the programmable system including at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device.
[0144] Program code can be applied to input instructions to execute the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, the processing system includes any system having a processor such as, for example, a digital signal processor, a microcontroller, an application-specific integrated circuit, or a microprocessor.
[0145] The program code can be implemented using a high-level procedural language or an object-oriented programming language to communicate with the processing system. Assembly language or machine language can also be used when needed. In fact, the mechanisms described in this application are not limited to any particular programming language. In either case, the language can be a compiled language or an interpreted language.
[0146] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried on or stored thereon by one or more transient or non-transitory machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, the instructions may be distributed via a network or through other computer-readable media. Therefore, machine-readable media can include any mechanism for storing or transmitting information in a machine-readable (e.g., computer-readable) form, including but not limited to floppy disks, optical disks, CD-ROMs, compact disc-read-only memory (CD-ROMs), magneto-optical disks, read-only memory (ROM), random-access memory (RAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic cards or optical cards, flash memory, or tangible machine-readable storage for transmitting information (e.g., carrier waves, infrared signals, digital signals, etc.) using the Internet in the form of electrical, optical, acoustic, or other forms of propagation signals. Therefore, machine-readable media includes any type of machine-readable medium suitable for storing or transmitting electronic instructions or information in a machine-readable (e.g., computer-readable) form.
[0147] In the accompanying drawings, some structural or methodological features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be necessary. Rather, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. Furthermore, the inclusion of structural or methodological features in a particular figure does not imply that such features are required in all embodiments, and in some embodiments, these features may be omitted or may be combined with other features.
[0148] It should be noted that all units / modules mentioned in the device embodiments of this application are logical units / modules. Physically, a logical unit / module can be a physical unit / module, a part of a physical unit / module, or a combination of multiple physical units / modules. The physical implementation of these logical units / modules themselves is not the most important factor; the combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed in this application. Furthermore, to highlight the innovative aspects of this application, the above-described device embodiments of this application have not introduced units / modules that are not closely related to solving the technical problems proposed in this application. This does not mean that the above-described device embodiments do not contain other units / modules.
[0149] It should be noted that in the examples and description of this patent, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0150] Although this application has been illustrated and described with reference to certain preferred embodiments thereof, those skilled in the art will understand that various changes in form and detail may be made thereto without departing from the scope of this application.
Claims
1. A rendering method, characterized in that, Applied to a terminal device, the terminal device including a first processor and a second processor, the method includes: The first processor acquires the data to be rendered and the first rendering perspective corresponding to the data to be rendered; When the first processor determines that the first rendering view meets the sorting conditions, it generates first sorting information of the data to be rendered based on the first rendering view and sends the first sorting information to the second processor. The second processor renders the data to be rendered from the first rendering perspective based on the first sorting information to obtain the first target rendering image.
2. The method according to claim 1, characterized in that, The sorting condition includes that the angular deviation between the first rendering viewpoint and the second rendering viewpoint corresponding to the second sorting information previously generated by the first processor is greater than a preset angle.
3. The method according to claim 2, characterized in that, The method further includes: If the first processor determines that the first rendering view does not meet the sorting conditions, it sends first information to the second processor. The first information is used to indicate that the sorting information has not changed. After detecting the first information, the second processor renders the data to be rendered in the first rendering view based on the second sorting information to generate a second target rendering image.
4. The method according to any one of claims 1 to 3, characterized in that, The data to be rendered includes multiple renderable objects and attribute data corresponding to each renderable object, and the attribute data includes the transparency of the corresponding renderable object; The second processor renders the data to be rendered based on the first sorting information under the first rendering perspective to obtain a first target rendered image, including: The second processor removes the first type of renderer from the plurality of renderers to obtain a plurality of renderers to be rendered, wherein the transparency of each renderer in the first type of renderer is lower than a preset transparency. The second processor renders the plurality of objects to be rendered to generate the first target rendered image.
5. The method according to any one of claims 1 to 3, characterized in that, The data to be rendered includes multiple rendering bodies; The second processor renders the data to be rendered based on the first sorting information under the first rendering perspective to obtain a first target rendered image, including: The second processor adjusts the projection size of each rendered object onto the first rendering plane based on preset size adjustment parameters; The second processor projects the adjusted render objects onto the first rendering plane to generate the first target render image.
6. The method according to any one of claims 1 to 3, characterized in that, The data to be rendered includes multiple rendering bodies; The second processor renders the data to be rendered based on the first sorting information under the first rendering perspective to obtain a first target rendered image, including: The second processor performs downsampling processing on the plurality of rendered objects based on a first downsampling coefficient, wherein the first downsampling coefficient is determined based on the rate of change of the rendering viewpoint; The second processor projects the downsampled plurality of render bodies onto the first rendering plane to generate the first target render image.
7. The method according to claim 6, characterized in that, The rate of change of the rendering perspective is negatively correlated with the first downsampling coefficient.
8. The method according to any one of claims 1 to 3, characterized in that, The generation of the first sorting information of the data to be rendered based on the first rendering perspective includes: The first processor determines the depth information of the data to be rendered from the first rendering perspective based on the vector dot product method; The first processor determines the first sorting information based on the depth value information of the data to be rendered.
9. A terminal device, characterized in that, Includes memory for storing instructions; At least one processor is configured to execute the instructions to cause the terminal device to implement the method of any one of claims 1 to 8.
10. A computer-readable storage medium, characterized in that, The readable storage medium stores instructions that, when executed on a computer, cause the computer to perform the method of any one of claims 1 to 8.
11. A computer program product, characterized in that, The computer program product stores instructions, which, when executed on the device, cause the device to perform the method of any one of claims 1 to 8.