Edge-cloud collaborative rendering method and related apparatus

Abstract

The application provides an end-cloud cooperative rendering method and related device. In the method, a terminal device can request a cloud to assist in performing pre-computation of high-order rendering. Thus, the terminal device can realize high-order rendering of the terminal device with the aid of high GPU computing power of the cloud, thereby improving the rendering effect of the terminal device. In addition, when requesting the cloud to perform pre-computation, the terminal device can send data of part of a region in a scene. Thus, the cloud can calculate pre-processing data of the part of the region, the terminal device can perform basic rendering on the part of the region, and perform high-order rendering in combination with the pre-processing data of the part of the region from the cloud. The terminal device can quickly load a picture of the part of the region, the loading speed is fast, the display of the picture is not affected, and the experience of a user is not affected.

Description

[0001] This application is a divisional application. The original application has the application number 202410799098.5 and the original application date is June 19, 2024. The entire contents of the original application are incorporated herein by reference. Technical Field

[0002] This application relates to the field of electronic technology, and in particular to a rendering method and related apparatus for edge-cloud collaboration. Background Technology

[0003] With the increasing prevalence of three-dimensional (3D) applications on mobile devices and other terminal devices, providing rendering capabilities for these 3D applications is the core function of these devices. However, due to limitations in size and power consumption, the computing power of the graphics processing unit (GPU) in these terminal devices is relatively small, lagging significantly behind that of GPUs in personal computers (PCs).

[0004] Cloud-based GPUs possess significant computing power, offering a substantial advantage over GPUs on edge devices. With the development of cloud computing technology, transferring computing power from the edge to the cloud and collaboratively handling edge-side tasks is gradually becoming a technological evolution direction. Currently, how to achieve efficient rendering processing in collaboration with the cloud to achieve better rendering results on the edge remains to be studied. Summary of the Invention

[0005] This application provides a cloud-edge collaborative rendering method and related apparatus. The terminal device can request the cloud to assist in pre-computing high-level rendering of a certain area in the terminal scene. On the one hand, by leveraging the high GPU computing power of the cloud, the rendering effect of the terminal device can be improved. On the other hand, the terminal can perform basic and high-level rendering on a certain area in the terminal scene, which can improve the loading speed of the image in that area and improve the user experience.

[0006] Firstly, this application provides a cloud-edge collaborative rendering method. The executing entity of this method can be a first terminal device or a chip within the first terminal device. The following description uses a first terminal device as an example. In this method, the first terminal device can perform basic rendering on a target area in a terminal-side scene to obtain first rendering data. The terminal-side scene includes multiple areas, and the target area is contained within these multiple areas. The first terminal device can send a first request to the cloud. The first request includes information about the terminal-side scene, an identifier of the target area, and status data. The first request instructs the cloud to obtain preprocessed data for high-level rendering of the target area in the terminal-side scene based on the status data. It should be understood that there is no sequential distinction between the first terminal device sending the first request to the cloud and the first terminal device performing basic rendering on the target area in the terminal-side scene.

[0007] In response to the first request, the cloud can obtain preprocessed data for high-level rendering of the target area in the terminal scene. The cloud processing procedure can refer to the cloud execution steps in the second or third aspect. In response to the preprocessed data from the cloud, the first terminal device can perform high-level rendering on the target area based on the preprocessed data to obtain second rendering data. The first terminal device can then obtain an image of the target area based on the first rendering data and the second rendering data, and display the image.

[0008] In this embodiment, the terminal device can request the cloud to assist in performing pre-computation for high-level rendering. Leveraging the high GPU computing power of the cloud, the rendering effect of the terminal device can be improved. Furthermore, when requesting the cloud to perform pre-computation, the terminal device can send data for a portion of the scene. In this way, the cloud can calculate the pre-processed data for that portion of the scene, and the terminal device can perform basic rendering on that portion. Combined with the pre-processed data from the cloud, high-level rendering can be performed, allowing the terminal device to quickly load the image of that portion of the scene. This fast loading speed does not affect the display of the image or the user experience.

[0009] In one possible implementation, the state data includes lighting information used for higher-order rendering of global lighting types, the lighting information including time, and / or, lighting information in the edge scene triggered by user operation.

[0010] For example, the higher-order rendering is a global illumination type rendering, which includes any of the following: Dynamic Diffuse Global Illumination (DDGI) or global illumination based on spherical harmonics.

[0011] In this example, the lighting information can be pre-configured for the edge scene, and / or, the lighting information can also be triggered by user operation. In this implementation method, the first terminal device can report the time, and / or the lighting information in the edge scene triggered by the user operation, to the cloud. Correspondingly, the cloud can pre-calculate global illumination for the target area in the edge scene based on the time, and / or the lighting information in the edge scene triggered by the user operation, obtaining pre-processed data, which can improve the completeness and accuracy of the calculation. Accordingly, when the first terminal device implements global illumination rendering, it can display not only the lighting pre-configured for the edge scene, but also the lighting in the edge scene triggered by the user operation, which can improve the lighting and shadow effects of the image.

[0012] In one possible implementation, the information of the terminal-side scene includes: an identifier of the terminal-side scene, and / or information about objects in the terminal-side scene, wherein the objects are pre-configured in the terminal-side scene. When the information of the terminal-side scene includes information about objects in the terminal-side scene, the user can perform operations on the first terminal device, and in response to the user's operation on the object, the first terminal device can obtain the information about the object.

[0013] In this implementation, edge-cloud collaborative rendering is applicable not only to scenarios with pre-configured edge scenes (such as open worlds) but also to user-generated content (UGC) scenarios. In UGC scenarios, users can adjust objects in the edge scene themselves or construct the edge scene themselves. Accordingly, in this scenario, the first terminal device needs to report information about objects in the edge scene to the cloud so that the cloud can construct the edge scene based on this information, thereby pre-computing the target area of ​​the edge scene and obtaining pre-processed data.

[0014] In one possible implementation, the terminal-side scenario is a scenario within an application, and the identifier of the terminal-side scenario includes: the identifier of the application and the identifier of the scenario; or, the identifier of the terminal-side scenario includes: the identifier of the application, the identifier of the application copy, and the identifier of the scenario.

[0015] In one possible implementation, the user focuses on the scene surrounding a first character, the target area being related to the position of the first character in the terminal scene, the first character corresponding to the first terminal device.

[0016] In one possible implementation, the user not only focuses on objects within the target area, but also specifically on objects within the character's field of view within that target area. In this implementation, the first request further includes a field of view, specifically used to instruct the cloud to obtain preprocessed data for high-order rendering of the field of view in the target area based on the state data.

[0017] Correspondingly, the terminal device can perform high-level rendering on the field of view, which can result in a more detailed high-level rendering area. The terminal device can perform high-level rendering on the field of view, but not on the non-field of view areas in the target area, which can reduce the high-level rendering workload of the terminal device.

[0018] In one possible implementation, the state data further includes at least one of the following: role information or scene update information.

[0019] Secondly, this application provides a cloud-edge collaborative rendering method. The executing entity of this method can be the cloud or a chip in the cloud. The following description uses the cloud as an example. In this method, the cloud can receive a first request from a first terminal device. The first request includes information about the edge-side scene, an identifier of a target region, and status data. The edge-side scene includes multiple regions, and the target region is contained within these multiple regions. Based on the information about the edge-side scene and the identifier of the target region, the cloud obtains the target region in the edge-side scene. After obtaining the target region, the cloud can, based on the status data, obtain preprocessing data for high-level rendering of the target region in the edge-side scene and send the preprocessing data to the first terminal device.

[0020] In one possible implementation, the state data includes lighting information used for higher-order rendering of global lighting types, the lighting information including time, and / or, lighting information in the edge scene triggered by user operation.

[0021] In one possible implementation, the information of the edge scene includes: an identifier of the edge scene, and / or information about objects in the edge scene, wherein the objects are objects pre-configured in the edge scene.

[0022] In one possible implementation, the terminal-side scenario is a scenario within an application, and the identifier of the terminal-side scenario includes: the identifier of the application and the identifier of the scenario; or, the identifier of the terminal-side scenario includes: the identifier of the application, the identifier of the application copy, and the identifier of the scenario.

[0023] In one possible implementation, when the information of the edge scene includes information about objects in the edge scene, obtaining the target region in the edge scene includes: the cloud constructing the edge scene based on the information about objects in the edge scene, and obtaining the target region in the edge scene based on the identifier of the target region.

[0024] In this implementation, since the edge scene is triggered or generated by the user, the cloud needs to first obtain the edge scene in order to facilitate high-level rendering pre-computation of the edge scene by the cloud. Therefore, the information of the edge scene can include information about the objects in the edge scene. Accordingly, the cloud can construct the edge scene based on the information of the objects in the edge scene, so that it can subsequently perform high-level rendering pre-computation of the target area in the edge scene.

[0025] In one possible implementation, the cloud includes: pre-processed data corresponding to the edge scenario at different times, wherein the pre-processed data is obtained offline by the cloud, or the pre-processed data is pre-configured in the cloud.

[0026] When the lighting information includes time, the step of obtaining preprocessed data for high-order rendering of the target region in the edge scene based on the state data includes: the cloud can query the first preprocessed data corresponding to the edge scene at the time in the cloud according to the time. The cloud can obtain the second preprocessed data corresponding to the target region at the time from the first preprocessed data, and use the second preprocessed data as the preprocessed data for high-order rendering of the target region.

[0027] In this implementation, the cloud can pre-configure the pre-processed data of the edge scene at different times, or obtain the pre-processed data of the edge scene at different times when offline. In this way, in response to the first request, the cloud can directly query the pre-processed data without performing complex pre-calculations, which can improve the speed and efficiency of collaborative rendering.

[0028] In one possible implementation, the first request further includes: a field of view. After obtaining the second preprocessed data corresponding to the time for the target region, the request further includes: the cloud obtaining the preprocessed data under the field of view from the second preprocessed data, and using the preprocessed data under the field of view as preprocessed data for high-order rendering of the target region.

[0029] In this implementation, the cloud can obtain preprocessed data under the field of view in the target area, which makes it easier for the first terminal device to render the scene under that field of view without having to process other areas in the target area that are not under that field of view, and can also improve the loading speed of the first terminal device.

[0030] In one possible implementation, when the lighting information includes lighting information in the edge scene triggered by the user operation, the step of obtaining preprocessed data for high-order rendering of the target area in the edge scene based on the state data includes: obtaining preprocessed data for high-order rendering of the target area in the cloud based on the pre-configured lighting at the current time and the lighting information in the edge scene triggered by the user operation.

[0031] In one possible implementation, the first request further includes: a field of view, and after obtaining the preprocessed data for high-order rendering of the target region, it further includes: obtaining preprocessed data for the field of view from the preprocessed data for high-order rendering of the target region.

[0032] In one possible implementation, the state data further includes at least one of the following: role information or scene update information.

[0033] In one possible implementation, the terminal-side scenario is a scenario in an application, and the method further includes: sending the preprocessed data to a second terminal device, wherein the application copy running on the second terminal device is the same as the application copy running on the first terminal device.

[0034] In this implementation, the cloud can send the calculation results (preprocessed data) to the terminal devices of other users in the same application copy. In this way, for the same scenario of the same application, the cloud can avoid duplicate calculations, which can save cloud resources and improve the experience of other users.

[0035] In one possible implementation, the higher-order rendering is a global illumination type rendering, which includes any of the following: Dynamic Diffuse Global Illumination (DDGI) or global illumination based on spherical harmonics.

[0036] In one possible implementation, the target region corresponds to a target space, in which multiple DDGI probes are deployed. The preprocessed data includes multiple DDGI probe data, and each DDGI probe data includes lighting information at and around the DDGI probe. When the higher-order rendering is DDGI, obtaining preprocessed data for higher-order rendering of the target region in the edge scene based on the state data includes: the cloud determining the target space corresponding to the target region and the multiple DDGI probes in the target space, and obtaining the multiple DDGI probe data according to the lighting information. In this embodiment, to reduce the bandwidth occupied by data transmission, the cloud can encode the multiple DDGI probe data and send the encoded DDGI probe data to the first terminal device.

[0037] Thirdly, embodiments of this application provide an edge-cloud collaborative rendering method that can be applied to an edge-cloud collaborative rendering system, which may include the first terminal device and the cloud as described above.

[0038] In this method, a first terminal device can send a first request to the cloud. The first request includes information about the terminal-side scene, an identifier of the target region, and status data. The terminal-side scene includes multiple regions, and the target region is contained within these regions. In response to the first request, the cloud can obtain the target region in the terminal-side scene based on the information about the terminal-side scene and the identifier of the target region, and based on the status data, obtain preprocessed data for high-level rendering of the target region in the terminal-side scene. The cloud can then send the preprocessed data to the first terminal device.

[0039] In this method, the first terminal device can perform basic rendering on a target area in the scene on the device side to obtain first rendering data. After receiving preprocessed data from the cloud, the first terminal device can perform high-level rendering on the target area based on the preprocessed data to obtain second rendering data. The first terminal device can then obtain an image of the target area based on the first rendering data and the second rendering data, and display the image.

[0040] In one possible implementation, the state data includes lighting information used for higher-order rendering of global lighting types, the lighting information including time, and / or, lighting information in the edge scene triggered by user operation.

[0041] In one possible implementation, the information of the edge scene includes: an identifier of the edge scene, and / or information about objects in the edge scene, wherein the objects are objects pre-configured in the edge scene.

[0042] For example, during the use of the first terminal device, the user obtains information about the object in response to the user's operation on the object.

[0043] In one possible implementation, the terminal-side scenario is a scenario within an application, and the identifier of the terminal-side scenario includes: the identifier of the application and the identifier of the scenario; or, the identifier of the terminal-side scenario includes: the identifier of the application, the identifier of the application copy, and the identifier of the scenario.

[0044] In one possible implementation, the target area is related to the position of a first character in the edge-side scene, and the first character corresponds to the first terminal device.

[0045] In one possible implementation, the state data further includes at least one of the following: role information or scene update information.

[0046] In one possible implementation, the higher-order rendering is a global illumination type rendering, which includes any of the following: Dynamic Diffuse Global Illumination (DDGI) or global illumination based on spherical harmonics.

[0047] In one possible implementation, the target region corresponds to a target space, in which multiple DDGI probes are deployed. The preprocessed data includes multiple DDGI probe data, and each DDGI probe data includes illumination information at and around the DDGI probe.

[0048] When the higher-order rendering is DDGI, the cloud, based on the state data, obtains preprocessed data for higher-order rendering of the target region in the terminal scene, including: the cloud determining the target space corresponding to the target region, and the plurality of DDGI probes in the target space. The cloud obtains the plurality of DDGI probe data according to the lighting information and encodes the plurality of DDGI probe data. Correspondingly, the cloud sends the preprocessed data to the first terminal device, including: sending the encoded DDGI probe data to the first terminal device.

[0049] In one possible implementation, when the information of the edge scene includes information about objects in the edge scene, the cloud obtains the target region in the edge scene, including: the cloud constructs the edge scene based on the information about objects in the edge scene, and obtains the target region in the edge scene based on the identifier of the target region.

[0050] In one possible implementation, the cloud includes: pre-processed data corresponding to the edge scenario at different times, wherein the pre-processed data is obtained offline by the cloud, or the pre-processed data is pre-configured in the cloud.

[0051] When the lighting information includes time, the cloud, based on the state data, obtains preprocessed data for high-level rendering of the target area in the edge scene, including: the cloud queries the cloud for first preprocessed data corresponding to the edge scene at that time. The cloud can obtain second preprocessed data corresponding to the target area at that time from the first preprocessed data, and use the second preprocessed data as preprocessed data for high-level rendering of the target area.

[0052] In one possible implementation, the first request further includes: a field of view. After the cloud obtains the second preprocessed data corresponding to the time for the target region, the request further includes: the cloud obtains the preprocessed data under the field of view from the second preprocessed data, and uses the preprocessed data under the field of view as preprocessed data for high-order rendering of the target region.

[0053] In one possible implementation, when the lighting information includes lighting information in the edge scene triggered by the user operation, the cloud obtains preprocessed data for high-order rendering of the target area in the edge scene based on the state data, including: the cloud obtains preprocessed data for high-order rendering of the target area based on the pre-configured lighting at the current time and the lighting information in the edge scene triggered by the user operation.

[0054] In one possible implementation, the first request further includes: a field of view. After the cloud obtains the preprocessed data for high-order rendering of the target region, the request further includes: obtaining preprocessed data for the field of view from the preprocessed data for high-order rendering of the target region.

[0055] In one possible implementation, the rendering system may further include a second terminal device, the second terminal device running an application copy identical to the application copy running on the first terminal device. Here, the terminal-side scene is a scene within the application, and the method further includes: the cloud can send the preprocessed data to the second terminal device.

[0056] Fourthly, embodiments of this application provide an electronic device, including a processor and a memory, wherein the memory is used to store code instructions and the processor is used to execute the code instructions to perform the methods described in the foregoing aspects and any possible implementation.

[0057] In some embodiments, the electronic device may be a first terminal device, which may perform the methods described in the first aspect above and in any possible implementation.

[0058] In some embodiments, the electronic device may be in the cloud, and the cloud may execute the methods described in the second aspect above and in any possible implementation.

[0059] Fifthly, embodiments of this application provide a computer-readable storage medium storing a computer program or instructions that, when executed on a computer, cause the computer to perform the methods described in the foregoing aspects and any possible implementation.

[0060] Sixthly, embodiments of this application provide a computer program product including a computer program, which, when run on a computer, causes the computer to perform the methods described in the above aspects and any possible implementation.

[0061] Seventhly, this application provides a chip or chip system including at least one processor and a communication interface. The communication interface and the at least one processor are interconnected via a circuit. The at least one processor is used to run computer programs or instructions to perform the methods described in the foregoing aspects and any possible implementations. The communication interface in the chip can be an input / output interface, pins, or circuits, etc.

[0062] In one possible implementation, the chip or chip system described above in this application further includes at least one memory storing instructions. The memory can be an internal storage unit of the chip, such as a register or cache, or it can be a storage unit of the chip itself (e.g., read-only memory, random access memory, etc.).

[0063] It should be understood that the second to seventh aspects of this application correspond to the technical solutions of the first aspect of this application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation are similar, and will not be repeated here. Attached Figure Description

[0064] Figure 1 A schematic diagram of a system architecture applicable to the edge-cloud collaborative rendering method provided in the embodiments of this application;

[0065] Figure 2 This is a schematic diagram illustrating the interaction between the terminal device and the cloud in an existing edge-cloud collaborative rendering method.

[0066] Figure 3A A schematic diagram of the game screen provided in the embodiments of this application without high-level rendering;

[0067] Figure 3B A schematic diagram of the game screen provided in the embodiments of this application when high-level rendering is used;

[0068] Figure 4 This is a schematic diagram of mesh division in a scenario provided by an embodiment of this application;

[0069] Figure 5 A flowchart illustrating one embodiment of the edge-cloud collaborative rendering method provided in this application.

[0070] Figure 6 A scenario diagram provided for an embodiment of this application;

[0071] Figure 7 This is a schematic diagram of the distribution of DDGI probes provided in an embodiment of this application;

[0072] Figure 8 A flowchart illustrating the process of obtaining DDGI probe data;

[0073] Figure 9 A flowchart illustrating another embodiment of the edge-cloud collaborative rendering method provided in this application.

[0074] Figure 10 This is a schematic diagram illustrating a scenario where multiple terminal devices share preprocessed data, as provided in an embodiment of this application.

[0075] Figure 11 This is a schematic diagram illustrating the interaction between a terminal device and the cloud, as provided in an embodiment of this application.

[0076] Figure 12 This is a schematic diagram illustrating the interaction between a terminal device and the cloud, as provided in an embodiment of this application.

[0077] Figure 13 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0078] Three-dimensional (3D) rendering technology is the process of projecting objects in a constructed 3D scene into a two-dimensional digital image according to a pre-defined viewpoint, lighting, and material information. With the continuous innovation of applications (APPs), the number of 3D apps installed on terminal devices is increasing. 3D apps can include, but are not limited to, 3D game apps, 3D modeling apps, 3D navigation apps, and 3D home decoration apps. In 3D apps, terminal devices can use 3D rendering technology to render 3D visuals, enhancing the user experience.

[0079] For example, taking a 3D game app as an example, objects in a 3D scene can include, but are not limited to, characters, buildings, plants, animals, rivers, etc. When users play the game, the terminal device can use 3D rendering technology to render 3D game graphics, enhancing the user's immersion.

[0080] It should be understood that the objects in a 3D scene can differ between different 3D apps, or even within the same app, depending on the specific scene. For example, objects in a 3D navigation app's scene may include, but are not limited to, vehicles, roads, buildings, and pedestrians. Similarly, objects in a 3D home decoration app's scene may include, but are not limited to, furniture, appliances, users, and pets.

[0081] In traditional 3D rendering techniques, such as rasterization, terminal devices can segment objects in a 3D scene using polygons. Through geometric transformations, the 3D coordinates of the polygon vertices are converted into 2D coordinates on the image. Finally, textures are filled into the polygons on the image, thus mapping the 3D object model onto a 2D screen to achieve image rendering. In some embodiments, the polygons can be triangles. Traditional 3D rendering techniques struggle to realistically reproduce light reflection, object shadows, and refraction effects in 3D scenes. Therefore, images rendered using traditional 3D rendering techniques often fail to deliver a lifelike 3D visual experience.

[0082] Compared to traditional 3D rendering techniques, ray tracing technology can provide a more realistic 3D visual experience. Ray tracing technology simulates the propagation of light in a 3D scene through reflection, refraction, shadows, and scattering, calculates the color and brightness values ​​at each location in the 3D scene, and renders a 2D image whose lighting effects conform to real-world physical laws, thus presenting a more realistic 3D virtual scene on the terminal device.

[0083] Ray tracing technology requires simulating a vast number of light paths to achieve visual effects that closely resemble the real world. This means that the technology demands high computing power from the graphics processing unit (GPU) of the terminal device. When the terminal device is a mobile terminal such as a smartphone or smartwatch, the GPU deployed in the mobile terminal has low computing power due to limitations in size and power consumption, making it unable to support ray tracing technology. This results in the mobile terminal being unable to use ray tracing technology to render scenes in 3D apps, leading to a poor visual experience for users. Alternatively, when the terminal device is a virtual reality (VR) device, augmented reality (AR) device, or a central control screen in a smart home, high-performance GPUs are not deployed in order to save costs. Therefore, these terminal devices also cannot use ray tracing technology to render scenes in 3D apps.

[0084] GPUs deployed in the cloud have a significant computing power advantage over those deployed in the aforementioned terminal devices. The high computing power of cloud-deployed GPUs can support high-computation-demand rendering techniques such as ray tracing. Therefore, with the development of cloud computing technology, transferring the computing power of GPUs from terminal devices (edge-side) to the cloud (cloud-side) to collaboratively process terminal-side business is gradually becoming a direction of technological evolution.

[0085] It is understood that in the following embodiments, "terminal device" refers to a terminal device with low GPU computing power, and "server" refers to a cloud with high GPU computing power. Here, "low GPU computing power" and "high GPU computing power" are relative concepts. In some embodiments, "low GPU computing power" means insufficient to support the related calculations in ray tracing technology, while "high GPU computing power" means supporting the related calculations in ray tracing technology.

[0086] It should be noted that the edge-cloud collaborative rendering method provided in this application is not only applicable to ray tracing technology, but also to other rendering technologies that require high GPU computing power. The following embodiments use ray tracing technology as an example for illustration.

[0087] In some embodiments, the terminal device may be equipped with a GPU with first computing power, while the cloud may be equipped with a GPU with second computing power. The second computing power is greater than the first computing power. In some embodiments, the second computing power is greater than or equal to a preset computing power, and the first computing power is less than the preset computing power. In this application embodiment, the first computing power is insufficient to support the relevant calculations in ray tracing technology, while the second computing power can support the relevant calculations in ray tracing technology.

[0088] In some embodiments, it can also be said that a PC-level GPU is deployed in the cloud.

[0089] The following describes the system architecture to which the edge-cloud collaborative rendering method provided in the embodiments of this application is applicable:

[0090] Figure 1 An exemplary system architecture to which embodiments of this application apply is illustrated. For example... Figure 1 As shown, the system architecture may include terminal device 100 and the cloud.

[0091] In some embodiments, the terminal device 100 may be a mobile phone, tablet computer, desktop computer, laptop computer, handheld computer, notebook computer, ultra-mobile personal computer (UMPC), netbook, or a cellular phone, personal digital assistant (PDA), augmented reality (AR) device, virtual reality (VR) device, artificial intelligence (AI) device, wearable device (e.g., smart bracelet), in-vehicle device, smart home device (e.g., smart TV, smart screen, large-screen device, etc.), and / or smart city device. This application does not impose special limitations on the specific type of the terminal device 100; the terminal device 100 may also be referred to as an edge device or electronic device. The following embodiments use a first terminal device and a second terminal device as examples.

[0092] In some embodiments, the terminal device 100 may be a mobile device or a non-mobile device.

[0093] In some embodiments, the cloud may include server 200. Server 200 may be a single server, a server cluster consisting of multiple servers, or a cloud-side computing center. The server 200 involved in the embodiments of this application may also be referred to as a cloud server, cloud-side, or cloud-based. It is not limited to server 200; the cloud may also include many other devices, which are not specifically limited here.

[0094] Terminal device 100 can communicate with the cloud via a communication network, and the cloud can provide rendering services for 3D apps to terminal device 100.

[0095] In some embodiments, the communication network may include, but is not limited to, local area networks (LANs), wide area networks (WANs), etc.

[0096] It should be understood that Figure 1The architecture shown does not constitute a specific limitation on the system architecture, which may include more or fewer devices than illustrated. For example, the system architecture may also include wireless repeater devices and wireless backhaul devices. Figure 1 (not shown in the text), which is not limited here.

[0097] The following is combined Figure 1 The system architecture shown illustrates the edge-cloud collaborative rendering method:

[0098] Taking 3D games as an example, in some embodiments, all rendering work in the game app on the terminal device can be transferred to the cloud, utilizing the GPU computing power of the cloud to improve the rendering effect on the terminal device. For example, such as... Figure 2 As shown, the terminal device has a game app client installed, which may include an operation command processing module and a video decoding module. The game app server can run in the cloud, and the server may include a rendering module, a logic module, and a virtual host / container. All rendering work on the server can be performed in the virtual host / container in the cloud.

[0099] During gameplay on a user's device, the device's operation command processing module collects the user's actions and uploads these actions to the game app's server. These actions can include, but are not limited to, left / right movement, perspective switching, and clicking. The game app's server sends the received operation commands to the logic module, which updates the game state based on the commands and triggers the rendering module to render the game scene in the latest state. The game app's server then transmits the rendered game footage as a video stream to the game app's client. The game app's client's video decoding module decodes the video stream, captures the game footage, and triggers the device to display the game.

[0100] In this example, relying on the powerful computing power of PC-level GPUs in the cloud, the cloud-based rendering module can perform basic rendering of the game scene, as well as high-level rendering of the game scene using ray tracing technology and other techniques.

[0101] Basic rendering and advanced rendering are relative concepts. Basic rendering requires relatively low GPU computing power, which is sufficient for the GPU of the terminal device. The game scene visuals obtained based on basic rendering can ensure the normal operation of the game app and the normal interaction of user operations. In some embodiments, the cloud can use rasterization technology to perform basic rendering of the game scene; this application does not specifically limit the basic rendering technology.

[0102] High-level rendering requires relatively high GPU computing power, and the GPU computing power of terminal devices cannot support the calculations related to high-level rendering. However, the GPU computing power in the cloud can independently complete high-level rendering. High-level rendering is usually easy to decouple from basic rendering. In the embodiments of this application, high-level rendering can be, for example, rendering performed using ray tracing technology.

[0103] In some embodiments, high-level rendering can be a lighting-related rendering task. For example, rendering of lighting properties such as global illumination (GI), ambient occlusion (AO), soft shadows, reflection, refraction, and caustics, and may also include rendering of other properties that are computationally intensive in the cloud.

[0104] In some embodiments, high-order rendering can be various forms of global illumination (GI). Examples include Dynamic Diffuse Global Illumination (DDGI), spherical harmonics-based global illumination, voxel-based global illumination, or point-based global illumination. The following section will combine... Figure 3A and Figure 3B Using GI as an example, this section introduces the rendering effects with and without advanced rendering:

[0105] For example, Figure 3A and Figure 3B This demonstrates the same game screen from a game app, showing the rendering effects with and without high-level rendering (such as GI). For example... Figure 3A As shown, the game can still run normally without GI, but the visuals are dark and lack the details of indirect lighting, such as the light and shadow details created by reflections from walls onto objects. Figure 3B As shown, with GI enabled, the visual effects are brighter, more realistic, and more vibrant, and the game screen will also display various lighting and shadow details.

[0106] Implementing the aforementioned edge-cloud collaborative rendering method leverages the high-performance GPUs in the cloud for all rendering tasks of the game scene. This enhances the visual effects when displaying the game scene on the terminal device, while reducing the GPU's computational requirements and load, thus broadening its applicability. However, this rendering method still has the following problems:

[0107] (1) All rendering work (basic rendering and advanced rendering) is performed in the cloud. The cloud computing cost is high, and the terminal device is only used as a video player, which wastes the GPU resources of the terminal device.

[0108] (2) Poor experience in weak network conditions. Users frequently encounter situations with poor network conditions. When the network connection to the terminal device is poor, because basic rendering is also performed in the cloud, the terminal device cannot display basic game graphics, affecting the normal operation of the game app. In addition, the user's operation response latency is very high, affecting the game operation experience.

[0109] (3) High transmission bandwidth cost. The cloud uses video streaming to transmit the rendered game screen to the terminal device. At 1080P@60fps, the transmission bandwidth is about 10Mbps, which results in high costs for both the cloud's outbound bandwidth and the terminal device's downlink bandwidth.

[0110] (4) The rendering results in the cloud cannot be shared among multiple users. For users participating in the same game, the cloud needs to repeatedly calculate all the rendering work of the game scene, which further increases the computing cost of the cloud.

[0111] In summary, how to achieve efficient rendering processing in collaboration with the cloud to achieve better rendering results on the edge (terminal device) still needs further research.

[0112] Currently, the computing power of GPUs in terminal devices can support basic rendering, but not high-level rendering. This is because high-level rendering requires calculations based on multiple data points in the scene before rendering. These calculations can be called pre-computation, and the results can be called pre-computation results, pre-computation data, or pre-processed data. Pre-computation requires high GPU computing power, but the GPUs deployed in terminal devices have low computing power and cannot support it. However, the GPU computing power required for rendering based on the calculation results is low, and the GPUs deployed in terminal devices can support this part of the rendering work. Therefore, this application provides another edge-cloud collaborative rendering method. In this method, the terminal device can perform basic rendering, and it can send the scene data required for pre-computation to the cloud. Leveraging the high GPU computing power of the cloud, the cloud can perform pre-computation based on multiple data points in the scene. The cloud can then send the calculation results to the terminal device, which will then perform high-level rendering based on the results.

[0113] In this way, the terminal device can perform basic rendering. When the network connection of the terminal device is poor, the terminal device can display basic game graphics, and the game app can run normally. In addition, the cloud can perform pre-computation, which can assist the terminal device in achieving high-level rendering and improve the display effect of the game graphics. Furthermore, because the terminal device can perform basic rendering and high-level rendering based on the calculation results of the cloud, the GPU resources of the terminal device will not be wasted, thus achieving efficient rendering processing. In summary, this edge-cloud collaborative rendering method can solve the above problems (1) and (2).

[0114] In addition, since the cloud no longer performs basic and advanced rendering, but instead performs calculations before advanced rendering, the cloud will not transmit the rendered game screen via video streaming, but will instead send the calculation results with low bandwidth consumption. Thus, this edge-cloud collaborative rendering method can solve the above problems (3).

[0115] In addition, regarding the above problem (4), in the end-to-cloud collaborative rendering method provided in this application embodiment, the cloud can send the calculation results to the terminal devices of other players participating in the game APP. In this way, for the same scene of the game APP, the cloud can avoid repeated calculations, and while saving cloud resources, it can also improve the game experience of other players.

[0116] In some embodiments, 3D apps, such as games, include open-world scenes. In open-world scenes, users can freely interact with a virtual world, contrasting with the linear and structured game world. Open-world scenes are typically large, for example, 10km x 10km. In such scenarios, due to the limitations of the terminal device's GPU computing power, the terminal device cannot load the entire scene at once. Consequently, when requesting pre-computation from the cloud, the terminal device does not send the entire scene's data, but only a portion of it. Otherwise, if the cloud obtains the computational results for the entire scene, the terminal device might not be able to load the entire scene, or the loading time might be too long, resulting in prolonged game screen updates, which would negatively impact the user's gaming experience.

[0117] To address this technical problem, and in accordance with the aforementioned technical concept, in this embodiment of the application, when the terminal device requests pre-computation from the cloud, the terminal device can send data for a portion of the scene. This allows the terminal device to perform basic rendering on that portion of the scene and, combined with the calculation results from the cloud, perform higher-level rendering. The terminal device can quickly load the image of that portion, resulting in fast loading speed without affecting the game's display or the user's gaming experience.

[0118] Based on the above technical concept, the edge-cloud collaborative rendering method provided in the embodiments of this application is described in detail below. To facilitate understanding of the edge-cloud collaborative rendering method provided in the embodiments of this application, the terminology involved in the edge-cloud collaborative rendering method is first introduced:

[0119] 1) Device-side scene: This can be understood as the scene in a 3D app on a terminal device. For example, taking a game app, the device-side scene can be different game scenes. In this embodiment, the device-side scene is a 3D scene. A 3D scene can be seen as a virtual three-dimensional environment (e.g., a city, park, forest, street, mountains), providing a multimedia virtual world. Users can control characters in the virtual scene by operating the terminal device, observing objects, animals, people, scenery, etc., from the character's perspective. Furthermore, users can switch perspectives by operating the terminal device, such as observing objects in the virtual scene from a God's-eye view or a third-person perspective.

[0120] In some embodiments, taking a game app as an example, the client-side scene can be the scene of any game level, or the scene from any perspective in any game level.

[0121] 2) Regions within the terminal scene: Due to the large scale of the scene, the terminal device cannot load the entire scene at once. In this case, the entire scene can be divided into multiple regions, each with an equal or unequal size and shape. These regions can be 3D or planar areas within the scene.

[0122] In some embodiments, for ease of calculation, the entire scene can be divided into a fine-grained grid. In this example, regions within the edge scene can be grids. For example, a scene of 10km × 10km can be divided into grids with a granularity of 100m × 100m. For example, a scene of such scale... Figure 4 The rectangle shown illustrates how a terminal device can divide a scene into an n×n grid, where each grid has the same size. Here, n is an integer greater than or equal to 2. For example... Figure 4 As shown, n is 5.

[0123] It should be understood that Figure 4 Different grids are distinguished by letters and numbers, as illustrated in the example.

[0124] 3) Basic rendering and advanced rendering: Basic rendering and advanced rendering are relative concepts. Please refer to the descriptions in the above embodiments for a more detailed understanding of basic rendering and advanced rendering.

[0125] In this embodiment, the GPU computing power required for basic rendering is relatively low, and the terminal device can complete basic rendering independently. The game scene generated by the terminal device's basic rendering ensures the normal operation of the game app and normal user interaction. When the communication network between the terminal device and the cloud is poor or disconnected, and the terminal device does not receive the calculation results of the higher-level rendering from the cloud, the game app on the terminal device can directly display the game screen after basic rendering to ensure normal user operation and viewing of the game app.

[0126] High-level rendering requires relatively high GPU computing power, and terminal devices cannot complete high-level rendering independently. In the embodiments of this application, high-level rendering is estimated to be completed independently by the GPU computing power in the cloud, and high-level rendering tasks are usually easily decoupled from basic rendering tasks.

[0127] In some embodiments, the terminal device can also determine the current high-level rendering type that can be performed for edge-cloud collaborative rendering based on real-time network conditions. In some embodiments, developers can divide the application (e.g., game app) into basic rendering and high-level rendering based on actual application needs and the GPU capabilities of the terminal device; no specific limitations are made here.

[0128] 4) App Development Phase: Also known as the production phase, this is where developers create the app's logic code and scenarios. After development is complete, developers can package the developed files into an app application package that can run on terminal devices, such as an Android application package (APK).

[0129] 5) APP runtime state: The terminal device can download the APP and run the APP application package on the terminal device. When the APP is in runtime state, the terminal device can display the APP interface and interact with users, the cloud, etc., based on the logic code and scenarios in the application package.

[0130] 6) Application copy: An application may include at least one application copy. Taking a game app as an example, a game copy refers to a scene or area within a scene that allows multiple users (such as teammates) to operate without interference from other users (non-teammates), and other users cannot enter that scene or area within the scene.

[0131] The rendering method for edge-cloud collaboration provided in this application will be described below with reference to specific embodiments. These embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0132] Figure 5This is a flowchart illustrating one embodiment of the edge-cloud collaborative rendering method provided in this application. (Refer to...) Figure 5 The edge-cloud collaborative rendering method provided in this application embodiment may include:

[0133] S501, the first terminal device performs basic rendering on the target area in the terminal scene and obtains the first rendering data. The terminal scene includes multiple areas, and the target area is contained in multiple areas.

[0134] The terminology used to describe the edge scene and the regions within it can be understood using the terms explained above. The edge scene comprises multiple regions, and the target region is contained within these regions. In other words, the first terminal device can perform basic rendering on a portion of the edge scene; this portion can be called the target region. By performing basic rendering on the target region within the edge scene, the first terminal device can obtain first rendering data.

[0135] In some embodiments, the first rendering data may be a base-rendered image, or intermediate data used in the process of generating the base-rendered image (e.g., irradiance of each pixel obtained through base rendering), or image data of the base-rendered image. The GPU computing power required by the rendering technology used for base rendering is provided by the terminal device, and this application embodiment does not specifically limit the rendering technology. In some embodiments, the terminal device may use a rasterization method to perform base rendering on the target area in the terminal scene to obtain the first rendering data.

[0136] Taking a game app as an example, the game scene includes a first character, which can be a character created by the user within the game app. The first character corresponds to a first terminal device. This correspondence can be understood as: the first character is the character the user plays when logging into the game app on the first terminal device, or the first character is the character the user plays while playing the game on the first terminal device.

[0137] While a user is playing a game, the user focuses on the scene around the first character. In some embodiments, the target area is related to the position of the first character in the scene on the device.

[0138] For example, the target area could be the area where the first character is located. Figure 4 For example, when the first role is in a G(2,2) grid, the target grid can be a G(2,2) grid.

[0139] For example, the target area can be the area where the first character is located, and the area surrounding the first character. Figure 4For example, when the first character is in grid G(2,2), the target grid can be the four surrounding grids, including grid G(2,2). For instance, the target grids could be G(2,2), G(2,3), G(3,2), and G(3,3). Alternatively, when the first character is in grid G(2,2), the target grids can be nine grids centered on grid G(2,2). For instance, the target grids could be: G(1,1), G(1,2), G(1,3), G(2,1), G(2,2), G(2,3), G(3,1), G(3,2), and G(3,3).

[0140] In some embodiments, the target area can be a user-specified area. For example, in a device-side scenario, if a user needs to view objects within a certain area or several areas, the user can click on the corresponding area on the terminal device; this one or more areas can be considered the target area. In this example, the target areas can be adjacent or non-adjacent.

[0141] In some embodiments, the number of target grids may be related to the GPU computing power of the terminal device. The higher the GPU computing power of the terminal device, the more target grids the terminal device can support loading. This application embodiment does not limit the number of target grids.

[0142] In one possible scenario, where the target area is a user-specified area, for example, if the terminal device's GPU computing power supports three areas, when the user clicks to select the target area, and when the user has selected three areas and is selecting the fourth, the terminal device can prompt the user that the area selection has been completed, and it cannot load any more.

[0143] S502, the first terminal device sends a first request to the cloud. The first request includes information about the terminal scene, the identifier of the target area, and status data.

[0144] In some embodiments, S501 and S502 are not distinguished by order and can be executed simultaneously. In this embodiment, the first terminal device can perform basic rendering on the target area, while pre-computation requiring high GPU computing power can be performed using the high GPU computing power of the cloud.

[0145] The first terminal device can send a first request to the cloud, which instructs the cloud to perform pre-calculation before high-level rendering. The first request may include information about the terminal scene, the identifier of the target area, and state data. Specifically, the first request instructs the cloud to obtain pre-processed data for high-level rendering of the target area in the terminal scene based on the state data. It should be understood that the cloud performs pre-calculation before high-level rendering based on the state data, and can obtain the calculation results (which can be called pre-processed data) for high-level rendering of the target area in the terminal scene.

[0146] The following sections will introduce the information of the edge scenario, the identification of the target area, and the status data.

[0147] First, information from the edge scenario.

[0148] The information of the device-side scene may include: the identifier of the device-side scene, and / or, information about objects in the device-side scene. The objects in the device-side scene are pre-configured within the device-side scene.

[0149] 1) In some embodiments, the scene on the device side can be determined in the development state of the app and remain unchanged in the runtime state of the app. For example, the scene in a chase game app is determined in the development state of the app, and the user cannot modify the scene in the game app while using the game app.

[0150] In this embodiment, the client-side application of the first terminal device is pre-configured with an identifier for the terminal-side scene, and the server-side application of the cloud can also be pre-configured with an identifier for the terminal-side scene. The identifier for the terminal-side scene is used to distinguish different terminal-side scenes; for example, number 1 represents a city scene, number 2 represents a mountain scene, or...

[0151] The client-side app on the first terminal device is pre-configured with an identifier for the edge-side scenario, while the server-side app on the cloud is pre-configured with an identifier for the cloud-side scenario. Edge-side and cloud-side scenarios correspond to each other, and their identifiers can be the same or different. It's understandable that the edge-side and cloud-side scenario identifiers can correspond to the same scenario. The purpose of this setup is that the edge and cloud sides have different data storage methods, therefore the identifiers for storing the same data can be different. This setup adapts to the current storage capabilities of the edge and cloud sides without requiring modification to their storage logic. For example, the edge can use numbers 1, 2, 3, etc., to distinguish different edge-side scenarios, and the cloud can use numbers 1-1, 1-2, 1-3, etc., to distinguish different cloud-side scenarios. For instance, edge-side scenario 1 can correspond to cloud-side scenario 1-1, and edge-side scenario 1 and cloud-side scenario 1-1 represent the same city scenario. Similarly, edge-side scenario 2 can correspond to cloud-side scenario 1-2, and edge-side scenario 2 and cloud-side scenario 1-2 represent the same mountain and river scenario.

[0152] In some embodiments, the information of the device-side scene may include an identifier for the device-side scene. The device-side scene can be a scene within an application, and the identifier for the device-side scene may include both the application's identifier and the identifier of the scene within the application. For applications containing application copies, the same device-side scene may differ in different application copies; therefore, the identifier for the device-side scene may also include the identifier of the application copy. In some embodiments, taking a game app as an example, the game app contains many game levels, and the scenes in different game levels are also different; therefore, the identifier for the device-side scene may also include the identifier of the level.

[0153] It should be understood that an identifier can be viewed as information used to uniquely distinguish certain content (such as a client-side scene, application, application copy, level, etc.). For example, an identifier can be information such as a number or name.

[0154] In this embodiment, when the cloud is configured with an identifier for the edge-side scenario, the cloud can determine the identifier for the edge-side scenario in response to the first request. When the cloud is configured with an identifier for the cloud-side scenario, the cloud can convert the identifier for the edge-side scenario into an identifier for the cloud-side scenario in response to the first request.

[0155] 2) In some embodiments, the client-side scene can be determined in the development state of the app, and can be changed in the runtime state of the app. For example, in a user-generated content (UGC) scene, the client-side scene can be changed by the user. For instance, taking a home decoration app as an example, the objects in the scene of the home decoration app can include, but are not limited to, televisions, sofas, bookshelves, etc., and these objects can be determined in the development state of the app. During the user's use of the home decoration app, the user can freely place objects and / or adjust the size of objects, etc., and the positions of the objects in the scene, and / or, change, i.e., the scene changes.

[0156] In this embodiment, the information of the device-side scene may include: the identifier of the device-side scene, and the information of objects in the device-side scene. Objects are objects pre-configured in the device-side scene. For example, in a home decoration app scenario, objects may be a house, and items that can be placed in the house, such as a television, sofa, and bookshelves. The information of objects in the device-side scene may include, but is not limited to: object identifiers, style descriptions, quantity, location, size, and color.

[0157] 3) In some embodiments, for apps with a single device-side scene, such as a room-decorating game app for younger users, the app may include a scene, for example, a room scene. In this example, the device-side scene can be determined in the app's development state and can be changed in the runtime state. For example, the user can change the position of objects in the room, and the scene will also change. In this scene, the information of objects in the device-side scene may include: the size, layout, and distribution of the room, as well as information such as the position of people, desks, beds, etc. in the room.

[0158] However, since this app only contains one scene, the scene information on the client side may not include the scene identifier, allowing the cloud to determine the app's scene. In this example, the client-side scene information may include information about objects within the client-side scene.

[0159] For example, a design app's scene might include objects such as spheres, cones, and cubes (basic objects). Users in a design app can use these objects to arrange the scene. For example, refer to... Figure 6 The scene can include one sphere, three cones, and three cubes. Correspondingly, the information in the end-side scene can include: the quantity, size, color, and position of each sphere, cone, and cube. (See reference...) Figure 6 The first request sent by the terminal device to the cloud includes information about the objects in the scene on the terminal side.

[0160] Secondly, the identification of the target area.

[0161] In this embodiment, the identifier of the target area can be the target area's number, name, location in the scene, or location relative to the scene center, etc., and this embodiment does not impose any limitations on this. For example, taking a grid as the target area, the identifier of the grid can be the grid's number, such as G(2,2).

[0162] Third, state data

[0163] State data is the data used by the cloud to perform pre-computation. In other words, state data is used by the cloud to obtain pre-processed data, which is then used for high-level rendering of the target area in the edge scene. In other words, the terminal device can perform high-level rendering of the target area in the edge scene based on the pre-processed data.

[0164] Different types of high-order rendering require different state data for pre-computation in the cloud. For example, in a lighting-related high-order rendering task, the state data can be lighting information. Lighting information can include, but is not limited to, the type, position, number, direction, intensity, and color of light sources. Similarly, in a character-related high-order rendering task, the state data can be character information. Character information can include, but is not limited to, the character's age, clothing, skin condition, and occupation.

[0165] Taking high-level rendering, which is a rendering task related to lighting, as an example, the state data can include lighting information.

[0166] 1) In some embodiments, the lighting in the scene is determined in the development state of the app and remains unchanged in the running state. For example, the lighting in the scene can change dynamically according to a certain time pattern. For example, the intensity, color, and direction of the lighting in the scene are pre-configured based on time changes. For example, in the morning, the intensity of the lighting in the scene is intensity 1, the direction of the light is from the east, and the color of the light is neutral light. In the evening, the intensity of the lighting in the scene is intensity 2, the direction of the light is from the west, and the color of the light is warm light. For example, the color temperature of neutral light can be 4000K, and the color temperature of warm light can be 3000K, etc. This application embodiment does not limit this.

[0167] In this embodiment, because the lighting in the scene changes dynamically according to a certain time pattern, the lighting in the scene is time-related. In some embodiments, the lighting information may include time.

[0168] In some embodiments, when the lighting in a scene changes dynamically according to a certain time pattern, the lighting information can be empty. In this embodiment, in response to the lighting information being empty, the cloud can determine that the lighting information is the current time.

[0169] 2) In some embodiments, the lighting in the scene is determined in the development state of the app, and the lighting can be changed in the running state of the app. For example, taking a game app as an example, actions such as releasing skills, opening and closing doors, and lighting torches can all change the lighting in the scene. Releasing skills, opening and closing doors, and lighting torches are often triggered by the user. For example, a user pressing a skill button can trigger the release of a skill. In this embodiment, in addition to pre-configured lighting, the scene may also include user-triggered lighting, which can affect the lighting in the scene. User-triggered lighting can also be understood as: lighting in the scene on the device side triggered by user actions.

[0170] In this embodiment, the state data may include time and user-triggered data. The user-triggered data is used to influence the lighting in the edge-side scene. For example, the user-triggered data may include user action data such as clicking a button on the interface. Based on this action data, the cloud can determine the lighting information triggered by the action data. The lighting information may include, but is not limited to, the type, intensity, color, direction, and location of the light. The type of lighting can be categorized as sunlight, lamplight, torchlight, and skill-based lighting, etc.

[0171] For example, in response to a user clicking a button on the interface, the terminal device (e.g., a game app) can determine the lighting information triggered by the operation data. In this example, the user-triggered data may include: lighting information in the edge scene triggered by the user operation.

[0172] In this embodiment, the state data may not include time. Accordingly, in response to illumination information, the cloud can determine the time as the current time.

[0173] S503: The cloud obtains the target area in the edge scene based on the information of the edge scene and the identifier of the target area.

[0174] In this embodiment of the application, in response to the first request, the cloud can first obtain the target region in the terminal scene based on the information of the terminal scene and the identifier of the target region, and then pre-calculate the target region in the terminal scene based on the state data. In some embodiments, the process of the cloud obtaining the target region in the terminal scene can also be referred to as the process of the cloud converting the terminal scene into a cloud scene.

[0175] The following section, using information from the edge-side scenario in S502's "Part One," describes methods for obtaining target regions from edge-side scenarios via the cloud:

[0176] 1) In some embodiments, the terminal scenario can be determined in the development state of the APP and remain unchanged in the running state of the APP.

[0177] In this example, the information of the device-side scene may include an identifier for the device-side scene. In some embodiments, the cloud can query the database for a target region within the device-side scene based on the identifier of the device-side scene and the identifier of the target region. The database can be local to the cloud or set up independently. The database can store various device-side scenes from different apps.

[0178] In some embodiments, the cloud can convert the identifier of the edge scene into an identifier of the cloud scene, and obtain the cloud scene based on the identifier of the cloud scene. The cloud can obtain the target area in the cloud scene based on the identifier of the target area.

[0179] In some embodiments, the cloud or database may store information about objects corresponding to identifiers in either the endpoint scene or the cloud scene. In this embodiment, the cloud can determine the information of objects corresponding to the endpoint scene identifiers based on the endpoint scene identifiers, i.e., the information of objects contained in the endpoint scene. The cloud can construct the endpoint scene based on the information of objects contained in the endpoint scene, and then obtain the target region in the endpoint scene based on the identifier of the target region. Alternatively, in this embodiment, the cloud can first convert the endpoint scene identifier into a cloud scene identifier, and then use the information of objects corresponding to the cloud scene identifiers, i.e., the information of objects contained in the cloud scene. The cloud can construct a cloud scene (corresponding to the endpoint scene) based on the information of objects contained in the cloud scene, and then obtain the target region in the cloud scene based on the identifier of the target region. The target region in the cloud scene can be considered as the target region in the endpoint scene.

[0180] 2) In some embodiments, the client-side scenario can be determined in the development state of the APP, and the client-side scenario can be changed in the running state of the APP.

[0181] For example, when the information of the edge scene includes the identifier of the edge scene and the information of the objects in the edge scene, the cloud can determine the edge scene based on the identifier of the edge scene, as described in 1). The cloud can place the objects in the determined edge scene based on the information of the objects in the edge scene and update the edge scene. The cloud can obtain the target region in the updated edge scene based on the identifier of the target region.

[0182] For example, when the information of the edge scene includes information about objects in the edge scene, the cloud can first construct the edge scene based on the information about objects in the edge scene, and then obtain the target area in the edge scene based on the identifier of the target area.

[0183] S504, based on state data, obtains preprocessed data for high-level rendering of target areas in the edge scene.

[0184] In response to the first request, the cloud can obtain preprocessed data for high-level rendering of the target region in the edge scene based on the state data. In other words, the cloud can perform pre-computation based on the state data to obtain preprocessed data. This preprocessed data is used by the terminal device to perform high-level rendering of the target region in the edge scene.

[0185] After acquiring the target region in the edge scene, the cloud can pre-calculate the target region based on the lighting information in the state data, obtaining pre-processed data for high-order rendering of the target region in the edge scene. The following describes the process of the cloud acquiring pre-processed data for high-order rendering of the target region in the edge scene, based on different scenarios:

[0186] 1) In some embodiments, the lighting in the scene is determined in the development state of the APP and remains unchanged in the running state of the APP.

[0187] Scenario 1: In this embodiment, pre-processed data for each edge scenario can be pre-configured in the cloud, or pre-processed data for the corresponding cloud-side scenario can be pre-configured for each edge scenario. The following explanation uses the example of pre-configured pre-processed data for each edge scenario in the cloud:

[0188] In scenario 1, the lighting in the edge scene can dynamically change according to a certain time pattern, and pre-processed data for each edge scene at each time can be pre-configured in the cloud. In this embodiment, the first request may include time, or the first request may be empty. Additionally, the first request also includes information about the edge scene and an identifier for the target area.

[0189] In response to the first request, the cloud can determine the edge-side scene and the target region within the edge-side scene. In some embodiments, the cloud can query the pre-processed data of the edge-side scene from pre-configured pre-processed data. When the first request may include time, the cloud can obtain the first pre-processed data corresponding to the edge-side scene at that time from the pre-processed data of the edge-side scene. When the first request is empty, the cloud can obtain the first pre-processed data corresponding to the edge-side scene at the current time from the pre-processed data of the edge-side scene.

[0190] It should be understood that the division of the target region in the edge-side scene is pre-configured, and both the edge-side and cloud-side data are known in advance or can be queried. After obtaining the first pre-processed data, the cloud can also obtain the pre-processed data corresponding to the target region from the first pre-processed data based on the target region's identifier; this can be referred to as the second pre-processed data. In this embodiment, the second pre-processed data can be used as "pre-processed data for high-level rendering of the target region in the edge-side scene".

[0191] Understandably, the cloud can also obtain the preprocessed data corresponding to the target area of ​​the edge scene from the preprocessed data of the edge scene based on the information of the edge scene and the identifier of the target area. In this preprocessed data, the cloud can further obtain the preprocessed data of the target area of ​​the edge scene corresponding to that time based on the time, which is "preprocessed data for high-level rendering of the target area in the edge scene".

[0192] Similar to pre-configuring pre-processed data for each edge scene, the cloud can pre-configure pre-processed data for the corresponding cloud scene for each edge scene. In this example, in response to the first request, the cloud can first convert the edge scene into a cloud scene, and then obtain "pre-processed data for high-level rendering of the target region in the edge scene" from the pre-processed data of the cloud scene.

[0193] Scenario 2: The cloud can calculate and store the preprocessed data for each edge scenario while offline. Alternatively, the cloud can retrieve and store the preprocessed data for the corresponding cloud scenario for each edge scenario while offline. The following example illustrates this scenario, where the cloud stores the preprocessed data for each edge scenario:

[0194] In scenario 2, the lighting in the edge scene can dynamically change according to a certain time pattern. The cloud can calculate the preprocessed data for each edge scene at each time point based on the lighting at each time point, even when offline. The cloud calculation process can be referenced... Figures 7-9 The description in the text.

[0195] In this example, the first request may include time, or the first request may be empty. Additionally, the first request may include information about the edge scene and an identifier for the target region. In response to the first request, the cloud can retrieve preprocessed data for high-level rendering of the target region in the edge scene from stored preprocessed data, as described in Case 1.

[0196] Similar to calculating preprocessed data for each edge scene offline, the cloud can calculate preprocessed data for the corresponding cloud scene for each edge scene offline. In this example, in response to the first request, the cloud can first convert the edge scene into a cloud scene, and then obtain the preprocessed data for high-order rendering of the target region in the edge scene from the preprocessed data of the cloud scene, as described in Case 1.

[0197] 2) In some embodiments, the lighting in the scene is determined in the development state of the APP, and the lighting can be changed in the running state of the APP.

[0198] In this example, in response to a first request, the cloud can calculate preprocessed data in real time for high-order rendering of the target region in the edge scene. In this embodiment, the first request may include user-triggered data and time, or the first request may include user-triggered data.

[0199] In response to the first request, the cloud can determine the pre-configured lighting in the edge scene based on time (or the current time), and determine the user-triggered lighting based on user-triggered data. The cloud can then calculate pre-processed data for high-order rendering of the target area in the edge scene based on the pre-configured lighting in the edge scene and the user-triggered lighting. The cloud's calculation process can be referenced... Figures 7-9 The description in the text.

[0200] In some embodiments, high-order rendering is a rendering task related to lighting characteristics, and the state information includes lighting information. Accordingly, preprocessed data is used to perform high-order rendering on the target area in the edge scene to obtain the lighting and shadow effects of high-order rendering.

[0201] Different high-order rendering types can have different preprocessing data, and correspondingly, the first rendering data can also be different. For example, in GI rendering, the preprocessing data includes the irradiance of each pixel after GI rendering, while the first rendering data can include the irradiance of each pixel after basic rendering. Similarly, in reflection rendering, the preprocessing data includes the cubemap captured by the reflection probe, while the first rendering data can include the color of each pixel after basic rendering.

[0202] S505: The cloud sends pre-processed data to the first terminal device.

[0203] S506, the first terminal device performs high-level rendering on the target area based on the preprocessed data to obtain the second rendering data.

[0204] The first terminal device receives the preprocessed data and can perform high-level rendering on the target area based on the preprocessed data to obtain the second rendering data. For example, the first terminal device can use shaders, combined with the preprocessed data of the target area, to color the target area and complete the high-level rendering.

[0205] S507, the first terminal device acquires an image of the target area based on the first rendering data and the second rendering data.

[0206] S508, the first terminal device displays an image of the target area.

[0207] In S507 and S508, the first rendering data and the second rendering data correspond to the target area in the terminal scene. Both the first rendering data and the second rendering data can contain the rendering data of each pixel in the target area. The first terminal device can fuse the first rendering data and the second rendering data according to the pixel correspondence to obtain the image of the target area. The first terminal device can send the image of the target area to the display to show the image of the target area.

[0208] It should be understood that when the first terminal device displays the image of the target area, it also displays images of other areas in the scene on the terminal side. However, the image of the target area is an image updated by the cloud, while the images of other areas have not been updated.

[0209] In this embodiment, the terminal device can request the cloud to assist in performing pre-computation for high-level rendering, and the effect of this technology can be referred to the description in the above embodiments. Furthermore, when requesting the cloud to perform pre-computation, the terminal device can send data for a portion of the scene. In this way, the cloud can calculate the pre-processed data for that portion of the scene, and the terminal device can perform basic rendering on that portion of the scene, and combine it with the pre-processed data from the cloud to perform high-level rendering. This allows the terminal device to quickly load the image of that portion of the scene, resulting in fast loading speed without affecting the display or user experience.

[0210] The following example illustrates the cloud-based pre-computation process, using the calculation of DDGI based on state data (lighting information). DDGI can be viewed as pre-processed data applied to the target area of ​​the edge scene. In some embodiments, when the cloud calculates DDGI based on lighting information, the pre-processed data can exist in the form of DDGI probe data. To facilitate understanding of DDGI probe data, we will first introduce GI probes and GI probe data:

[0211] GI probes are essentially sampling points used to store lighting information at different locations in a scene. One way to place GI probes is to arrange them evenly at a certain density in a 3D scene. Each GI probe can then collect or detect lighting in all directions from its location, recording the lighting information in a cached format to obtain GI probe data. When a terminal device needs to render a shading point, it only needs to find several GI probes around that shading point and interpolate the lighting information stored in these probes to obtain the lighting information at that shading point.

[0212] In some embodiments, the caching format may be, for example, an octahedral mapping format or a spherical harmonics format. For instance, when caching lighting information at the GI probe in an octahedral mapping format, the GI probe data may include octahedral mapping texture data. When caching lighting information at the GI probe in a spherical harmonics format, the GI probe data may include spherical harmonics.

[0213] In summary, when the cloud calculates GI based on lighting information, the resulting preprocessed data can be GI probe data. Similarly, when the cloud calculates DDGI based on lighting information, the resulting preprocessed data is DDGI probe data. DDGI probe data can include the lighting information of the DDGI probe, or it can be referred to as the texture data of the DDGI probe.

[0214] Figure 7 This diagram illustrates one possible distribution of DDGI probes in a scene. (Refer to...) Figure 7 The scene includes the ground and houses. Figure 7 The ground is represented by quadrilaterals, and DDGI probes distributed throughout the scene are represented by black dots. Figure 7 The different sizes of the black dots represent the DDGI probes distributed at different locations in the edge scene.

[0215] In some embodiments, DDGI probes in a scenario (edge-side scenario) can be pre-deployed in the terminal device and the cloud, and both the cloud and the terminal device can determine the information of the DDGI probes in each edge-side scenario. For example, the information of the DDGI probes may include, but is not limited to, the number of DDGI probes, their distribution, etc.

[0216] In some embodiments, DDGI probes in the edge scenario can be pre-deployed in the terminal device, and the terminal device can determine the information of the DDGI probes in each edge scenario. In this example, when the terminal device requests pre-computation from the cloud, it can send the information of the DDGI probes in the edge scenario to the cloud. In this example, for example, the first request may also include the information of the DDGI probes in the edge scenario. Thus, in response to the first request, the cloud can determine the information of the DDGI probes in the edge scenario.

[0217] In some embodiments, refer to Figure 8 The calculation process for DDGI probe data may include the following steps:

[0218] Step 1: Count the Volumes (spaces) that need to be updated.

[0219] Volumes can be understood as the 3D space where DDGI probes are distributed. In some embodiments, regions in the edge scene are mapped to Volumes. For example, one region corresponds to one Volume, or one region corresponds to multiple Volumes, or multiple regions correspond to one Volume, and the mapping relationship between regions and Volumes can be pre-configured.

[0220] In some embodiments, the size of the volume can be related to the GPU computing resources on the cloud side and the edge side.

[0221] In some embodiments, the size of a volume may be constant; for example, the mapping between regions and volumes may be pre-configured. In some embodiments, the size of a volume may vary with the GPU computing resources on the cloud side and / or the edge side, and the mapping between regions and volumes may also change accordingly.

[0222] In some embodiments, in response to a first request, the cloud can determine the target region in the edge scenario that needs to be updated, and based on the mapping relationship between the region and the volume, determine the volumes that need to be updated. These volumes that need to be updated can be referred to as the target volume or the target space.

[0223] Step 2: Update Volumes in a loop to obtain the DDGI probe data in Volumes.

[0224] Updating Volumes once in a loop can be understood as retrieving DDGI probe data from Volumes once.

[0225] The following example illustrates step 2 of the process of obtaining DDGI probe data from a Volume. In some embodiments, updating a Volume may include the following steps:

[0226] Step 2-1: The DDGI probe emits rays to find the intersection and returns the illumination result (RT radiance).

[0227] The DDGI probe within the Volume emits rays, which are reflected when they come into contact with objects in the scene. Thus, light is collected or detected in all directions from the DDGI probe's location, resulting in the illumination at the DDGI probe's position within the Volume. The illumination at the DDGI probe can include values ​​such as Irradiance and Distance. Irradiance represents the irradiance information at the DDGI probe, while Distance provides the DDGI probe offset information for final rendering and shading.

[0228] Step 2-2: Irradiance blending of DDGI probe.

[0229] For each DDGI probe, the irradiance information at the DDGI probe can be mixed with that of the surrounding DDGI probes to obtain the comprehensive irradiance information at that DDGI probe.

[0230] Steps 2-3: DDGI probe distance blending.

[0231] This step involves calculating the visibility (or degree of occlusion) of the illumination at each DDGI probe. In some embodiments, distance blending may also be referred to as visibility blending. For example, for each DDGI probe, the visibility of the illumination at that DDGI probe can be determined based on the distance to surrounding DDGI probes or other factors.

[0232] Steps 2-4: Distance Border Update of DDGI Probe Irradiance Information.

[0233] After determining that the DDGI probe irradiance information is mixed, the irradiance information at the DDGI probe boundary needs to be updated to ensure information consistency.

[0234] Steps 2-5: DDGI probe distance to edge copy (Distance Border Update).

[0235] After determining the visibility of the illumination at the DDGI probe, the visibility at the DDGI probe boundary needs to be updated to ensure information consistency.

[0236] Steps 2-6: Adjust the position of the DDGI probes (RelocateProbes).

[0237] In some embodiments, steps 2-6 are optional.

[0238] In some embodiments, the position of the DDGI probe can be adjusted to better capture the light at the DDGI probe location.

[0239] Steps 2-7: DDGI Probe Status Classification

[0240] In some embodiments, steps 2-7 are optional.

[0241] In some embodiments, a status flag may be used to indicate whether the data at the DDGI probe has been updated or not, so as to update the data of each DDGI probe in a timely manner.

[0242] In some embodiments, step 2 above can also be briefly described as follows:

[0243] The cloud uses the DDGI algorithm to acquire DDGI probe data from Volumes. In some embodiments, DDGI probes can be used to store scene lighting information and dynamically updated using ray tracing, thereby achieving real-time dynamic diffuse global illumination effects. The DDGI algorithm packages a group of DDGI probes into one or more Volumes. Simply drag a Volume into the scene to be rendered, and the Volumes will automatically place DDGI probes within it. The shading points within the Volume will automatically capture lighting information through the surrounding DDGI probes.

[0244] For any shading point within a volume (e.g., shading point 1), the terminal device can acquire 8 DDGI probe data around shading point 1. Based on these 8 DDGI probe data, the terminal device performs interpolation processing on the irradiance to obtain the irradiance of shading point 1. In some embodiments, the preprocessed DDGI data may include the irradiance of each pixel corresponding to a shading point in the target region of the edge scene.

[0245] In some embodiments, the DDGI probe stores spherical information. The DDGI algorithm encodes the spherical data into a two-dimensional texture map using octahedral mapping. The smallest unit of the texture map is a texel, and one texel corresponds to one or more pixels. DDGI probe data may include the irradiance received from the hemisphere along the texel (w) direction, the distance r (w) between the probe and the nearest object seen from the texel direction, and the square of the distance r. 2 (w). Here, irradiance is encoded as a 3D vector texture, r(w) and r... 2 (w) are encoded together as a two-dimensional vector texture (x component stores r(w), y component stores r). 2 (w)).

[0246] In one implementation, probe 1 is any one of the eight DDGI probes mentioned above. The terminal device, for example, interpolates the irradiance based on the data from the eight DDGI probes to obtain the irradiance of the colored point 1. Specifically, this may include: obtaining three weighting coefficients for probe 1, namely the trilinear interpolation coefficient, the orientation coefficient, and the Chebyshev coefficient; using the normalized value of the product of these three coefficients as the weight of probe 1; and weighting the irradiance of the eight DDGI probes based on the weight of each probe to obtain the irradiance of the colored point 1.

[0247] The trilinear interpolation coefficient indicates the distance between probe 1 and shaded point 1; a larger coefficient reduces the weight of probe 1. The direction coefficient indicates the angle between the direction from shaded point 1 to probe 1 and the surface normal of shaded point 1; a larger coefficient reduces the weight of probe 1. The Chebyshev coefficient indicates the probability of an obstruction between probe 1 and shaded point 1; a larger coefficient reduces the weight of probe 1. The Chebyshev coefficient is based on the distance r(w) and the squared distance r... 2 (w) is certain.

[0248] Step 3: Fill the Volumes with a texture (discripter).

[0249] Based on the DDGI probe data in each Volume, a texture can be filled into the entire Volume. Filling the texture can be understood as filling the Volume with a lighting texture.

[0250] Step 4: Render the Volumes (the target areas of the corresponding edge scene).

[0251] Step 4 allows you to configure a lower resolution. For example, to improve rendering efficiency, you can convert the DDGI rendering to a lower resolution version.

[0252] Step 5: Upsampling.

[0253] In some embodiments, since the DDGI rendering can be converted to a low-resolution version in step 4, biosampling can be performed in step 5 to restore the target area of ​​the edge scene to its original resolution, thereby improving the image clarity of the target area of ​​the edge scene.

[0254] It should be understood that the process of obtaining DDGI probe data in this application embodiment is briefly described, and the specific algorithm can be referred to in the detailed description of the DDGI algorithm in the prior art.

[0255] In some embodiments of this application, reference is made to Figure 9 The cloud can execute steps 1 and 2 to obtain DDGI probe data for the target area of ​​the edge scene. After obtaining the DDGI probe data for the target area of ​​the edge scene, the cloud can perform the following steps:

[0256] Step 3A: Encode the DDGI probe data in Volumes in the cloud.

[0257] Step 3A can also be referred to as texture encoding.

[0258] In some embodiments, the cloud may use a preset encoding algorithm to encode the DDGI probe data in Volumes.

[0259] Step 4A: The cloud sends the encoded DDGI probe data to the terminal device.

[0260] The encoded DDGI probe data can be understood as: encoded texture data.

[0261] Step 5A: The terminal device decodes the DDGI probe data.

[0262] In some embodiments, the terminal device may use a decoding algorithm corresponding to a preset encoding algorithm to decode the received preprocessed data (DDGI probe data) to obtain the DDGI probe data of the target area.

[0263] Step 6A: The terminal device performs high-level rendering on the target area based on the DDGI probe data to obtain the second rendering data.

[0264] In some embodiments, the target region of the edge scene includes at least one shading point. High-order rendering of the target region by the terminal device can be understood as follows: the terminal device queries the DDGI probe data for the shading point, and based on the DDGI probe data at that location and the neighboring DDGI probe data, colors the shading point, thus completing the rendering of that shading point. Using the same method, the terminal device can render each shading point in the target region, completing the high-order rendering of the target region.

[0265] Step 7A: The terminal device performs over-resolution processing.

[0266] In some embodiments, step 7A may be an optional step.

[0267] The super-resolution processing in step 7A has the same purpose as the upsampling in step 5: to restore the target area of ​​the scene to its original resolution, thereby improving the image clarity of the target area. Super-resolution (SR) is a low-level image processing task that maps a low-resolution image to a high-resolution image to enhance image details.

[0268] Reference Figure 9 In this embodiment, the terminal device can send the lighting information of the target area of ​​the scene to the cloud. The cloud can perform pre-calculation based on this lighting information to obtain pre-processed data such as DDGI probe data. The cloud can send the pre-processed data to the terminal device so that the terminal device can perform high-level rendering of the target area of ​​the scene based on the pre-processed data. In this embodiment, the calculation of DDGI probe data relies on the high GPU computing power of the cloud so that the terminal device can achieve high-level rendering with global illumination. In addition, the pre-calculation performed by the cloud and the high-level rendering performed by the terminal device based on the pre-processed data can also make comprehensive use of the GPU resources of the cloud and the GPU resources of the terminal device, achieving joint use and avoiding waste of the GPU resources of the terminal device.

[0269] While playing a game, users not only focus on objects within the target area, but also specifically on objects within the character's field of view within that target area. To obtain more refined preprocessed data and reduce the high-level rendering workload of the terminal device, in some embodiments, the first request may also include the field of view (FOV). The field of view, also known as the viewing angle, is used in game apps as an example. During gameplay, users can observe objects in the scene from different perspectives, such as first-person, third-person, and top-down views.

[0270] In this embodiment, the first request may include: information about the edge scene, an identifier of the target region, status data, and a field of view. It should be understood that the target region may cover the field of view so that the cloud can fully acquire the preprocessed data for high-level rendering of that field of view. Accordingly, in response to the first request, the cloud can acquire the preprocessed data for the target region's field of view.

[0271] In some embodiments, referring to the descriptions in 1) and 2) of S504, after the cloud obtains the preprocessed data (such as the second preprocessed data) for high-order rendering of the target region of the edge scene, the cloud can also obtain the preprocessed data under the field of view angle from the preprocessed data (such as the second preprocessed data) for high-order rendering of the target region of the edge scene. For example, the cloud can retain the preprocessed data under the field of view angle in the target region and delete the preprocessed data under the field of view angle in the target region to obtain the preprocessed data under the field of view angle.

[0272] It should be understood that the non-field of view area in the target region can be understood as: the area in the target region other than the field of view.

[0273] In this embodiment, the cloud can send preprocessed data of the target area under that field of view to the terminal device. The terminal device can perform basic rendering on the target area under that field of view of the edge scene to obtain first rendering data. When the terminal receives the preprocessed data of the target area under that field of view from the cloud, it can perform high-level rendering on the target area under that field of view of the edge scene based on the preprocessed data to obtain second rendering data. The terminal device can also obtain an image of the target area under that field of view of the edge scene based on the first rendering data and the second rendering data, and display the image of the target area under that field of view of the edge scene.

[0274] In this embodiment, when the terminal device requests preprocessing data for high-level rendering from the cloud, it can also send the field of view to the cloud so that the cloud can calculate the field of view of the target area of ​​the scene on the terminal side and obtain the preprocessing data. Accordingly, the terminal device can perform high-level rendering on the field of view to obtain a more refined high-level rendering area. The terminal device can perform high-level rendering on the field of view but not on the non-field of view areas in the target area, which can reduce the workload of high-level rendering on the terminal device.

[0275] The above embodiments describe the process of obtaining preprocessed data for high-order rendering of the target area of ​​the terminal scene from the cloud, and preprocessed data for high-order rendering of the target area of ​​the terminal scene from a specific field of view. In some embodiments, after obtaining the above preprocessed data, the cloud can not only send the preprocessed data to the first terminal device, but also send the preprocessed data to the terminal devices of other players using the APP, so as to avoid the cloud performing repeated pre-calculation of the same target area of ​​the same scene and reduce the computational workload of the cloud.

[0276] For example, taking a game app as an example, multiple players can be in the same game instance, and these multiple players can be in the same game scene. If each player's terminal device requests preprocessed data of the game scene from the cloud, the cloud needs to perform multiple pre-calculations. In this embodiment of the application, refer to... Figure 10 Taking multiple players' terminal devices, including a first terminal device and a second terminal device, as an example, in response to the first request from the first terminal device, the cloud can perform a pre-calculation to obtain pre-processed data of the target area (or the field of view of the target area) in the game scene.

[0277] In some embodiments, because the first request includes an identifier for an application copy, the cloud can use this identifier to locate a second terminal device that is in the same game copy as the first terminal device. The cloud can then send the preprocessed data to both the first and second terminal devices.

[0278] In response to the preprocessed data, the first terminal device can perform high-level rendering on the target area (or the field of view of the target area) and display the image of the target area (or the field of view of the target area). For the second terminal device, when the interface displayed by the second terminal device includes the target area (or the field of view of the target area), the second terminal device can also perform high-level rendering on the target area (or the field of view of the target area) based on the preprocessed data, so that the second terminal device can update the image of the target area (or the field of view of the target area).

[0279] It is conceivable that when the interface displayed by the second terminal device does not include the target area (or the field of view of the target area), the second terminal device may not perform high-level rendering of the target area (or the field of view of the target area). As the second character corresponding to the second terminal device moves in the game scene, when the interface displayed by the second terminal device includes the target area (or the field of view of the target area), the second terminal device then performs high-level rendering of the target area (or the field of view of the target area) based on the preprocessed data, so that the second terminal device can update the image of the target area (or the field of view of the target area), which can speed up the image update speed of the second terminal device.

[0280] In this embodiment, when the scene on the client side involves multiple users (or players), the cloud does not need to perform pre-computation for each user; it only needs to perform pre-computation once and share it with other users. This further reduces the consumption of cloud-side GPU resources in the client-cloud collaborative rendering scheme.

[0281] The above embodiments describe the edge-cloud collaborative rendering method provided by this application from the perspective of the cloud and the first terminal device. The following will combine... Figures 11-12 The internal modules of the cloud and the first terminal device are discussed, and the rendering method for end-to-cloud collaboration provided in this application is introduced from the perspective of internal module interaction.

[0282] Figure 11 This is a schematic diagram illustrating the interaction between a terminal device and the cloud, as provided in an embodiment of this application. The terminal device can be a first terminal device or a second terminal device.

[0283] Reference Figure 11 The terminal device may include an application (APP). This APP may be a 3D application. In some embodiments, the APP may include a logic module and a rendering module. The cloud may include a high-order rendering pre-computation module, as well as a virtual host / container. The pre-computation in the cloud can be performed within a virtual host / container.

[0284] The rendering module is used to perform basic rendering on the target area of ​​the edge scene to obtain first rendering data. In some embodiments, the rendering module is also used to perform basic rendering on the field of view of the target area of ​​the edge scene to obtain first rendering data.

[0285] The logic module is used to determine information about the edge scene, the identifier of the target area, and status data in response to user operations. In some embodiments, the logic module is also used to determine the field of view.

[0286] The logic module is also used to send a first request to the higher-order rendering pre-computation module. This first request may include information about the edge scene, an identifier of the target region, and state data. Alternatively, the first request may include: information about the edge scene, an identifier of the target region, state data, and the field of view.

[0287] The advanced rendering pre-computation module is used to respond to the first request, acquire preprocessed data of the target region (or the field of view of the target region) for the edge scene, and send the preprocessed data to the rendering module. The method by which the advanced rendering pre-computation module acquires the preprocessed data can be referred to the description in the above embodiments.

[0288] The rendering module is also used to perform high-level rendering on the target area (or the field of view of the target area) of the terminal scene based on the preprocessed data to obtain the second rendering data. The rendering module is also used to acquire an image of the target area (or the field of view of the target area) based on the first rendering data and the second rendering data and send it for display so that the terminal device can display the image of the target area (or the field of view of the target area).

[0289] The embodiments of this application have the same technical principles and effects as the embodiments described above, and can be referred to the descriptions in the embodiments described above.

[0290] Figure 12 This is another schematic diagram illustrating the interaction between the terminal device and the cloud provided in an embodiment of this application. (Refer to...) Figure 12 The terminal devices include: an app, a device-side 3D engine, and a device-cloud collaboration plugin SDK. The cloud includes: advanced rendering services, a device-cloud collaboration framework, and a cloud-side 3D engine.

[0291] An app can include logic code. Logic code is the code used to implement the logic of the app.

[0292] The device-side 3D engine comprises key modules such as the engine framework, scene management, and rendering pipeline. The engine framework provides the device-side app with capabilities for logic parsing, model parsing, and animation parsing. Scene management manages the scene information of one or more scenes (i.e., device-side scenes) built by the app. The rendering pipeline converts the 3D scene model into a 2D space in screen pixels and outputs the scene image. Specifically, the rendering pipeline includes the following functions: converting the 3D coordinates of objects into 2D coordinates in screen pixel space and coloring each pixel on the screen.

[0293] The edge-side 3D engine operates in two modes: development and runtime. In development mode, it provides developers with an integrated development environment (IDE) to create the app's logic code and render 3D scenes. After development, developers can package the relevant files into an application package for a game app that can run on terminal devices, such as an Android application package. In runtime mode, it provides the app with a runtime environment, which typically has cross-platform capabilities, supporting the app's operation on mobile devices running different operating systems such as HarmonyOS, Android, and iOS. In this embodiment, during runtime, the app can call the aforementioned rendering pipeline to perform basic rendering of the edge-side scene to be rendered, generating first rendering data. Additionally, during runtime, the app can also call the edge-cloud collaboration plugin SDK to request pre-computation of high-level rendering from the cloud. Furthermore, it can fuse the first and second rendering data, obtaining a displayable high-level rendered image based on the fused data.

[0294] Edge-Cloud Collaboration Plugin SDK: This SDK encapsulates the key capabilities of edge-cloud collaborative rendering and, based on the plugin mechanism provided by the edge 3D engine, enables third-party edge 3D engines to possess edge-cloud collaborative rendering capabilities. The Edge-Cloud Collaboration Plugin SDK includes some or all of the following sub-functions: state synchronization, transmission communication, decoding, and pipeline adaptation. In this embodiment, edge-cloud collaborative rendering is achieved by providing an Edge-Cloud Collaboration Plugin SDK to third-party 3D engines, avoiding intrusive modifications to the third-party 3D engines.

[0295] End-side state synchronization: Used to synchronize state data to the cloud side. For example, state data may include some or all of the following: lighting information (e.g., type, position, posture, brightness, number of light sources, etc.), character information (e.g., position, posture, animation, skills, number of virtual characters, etc.), camera information (e.g., camera position, posture, field of view (FOV), etc.), and scene update information (e.g., information related to object movement, object animation, destruction, etc.).

[0296] End-side transmission communication: used to upload end-side state synchronization data to the cloud side, and to receive high-level rendering preprocessing data sent from the cloud side.

[0297] Decoding on the client side: This is used to decode the compressed and encoded preprocessed data sent from the cloud side into preprocessed data that can be consumed by the rendering pipeline on the client side.

[0298] End-side pipeline adaptation: Based on the extension mechanism of the end-side 3D engine rendering pipeline, it adds the ability to consume and render decoded preprocessed data on the basis of the original end-side rendering pipeline, that is, the ability to fuse the first rendering data and the second rendering data to obtain higher-order rendering effects.

[0299] In some embodiments, the high-order rendering service may include a pre-computation module. The pre-computation module is configured to perform pre-computation based on state data in response to a first request, thereby obtaining pre-processed data.

[0300] Cloud-based 3D engine: Includes key functional modules such as scene management, rendering pipeline, and engine framework, providing a capability foundation for edge-cloud collaborative frameworks and high-level rendering services.

[0301] End-to-cloud collaboration framework: includes some or all of the following sub-functions: state synchronization, transmission communication, decoding, and session management.

[0302] Cloud-side state synchronization: This is used to receive the first request from the terminal side and, based on the terminal side scene information in the first request, update the scene information of the corresponding cloud-side scene in the scene management module of the cloud-side 3D engine.

[0303] Cloud-side encoding: Used to compress and encode preprocessed data for high-level rendering services to reduce the amount of data transmitted.

[0304] Cloud-side transmission communication: used to receive status data transmitted from the end side, and to send compressed and encoded preprocessed data to the end side.

[0305] Session management: This is used to assign clients accessing the platform to corresponding high-level rendering services. For example, it can assign services based on information uploaded by the client, such as the game ID, level / scene ID, or instance ID. In essence, the cloud can establish sessions with multiple client-side clients and their corresponding high-level rendering services, thus providing high-level rendering services to multiple clients (terminal devices) simultaneously.

[0306] Based on a cloud-based 3D engine and an edge-cloud collaborative framework, it provides high-level rendering services, namely high-level rendering pre-computation, such as GI pre-computation, AO pre-computation, reflection pre-computation, soft shadow pre-computation, and other high-level rendering services.

[0307] In some embodiments, to save cloud-side GPU computing power and facilitate development and debugging, during the APP development phase, Figure 12 The terminal device and the cloud module shown can both be simulated for development and testing on the same device. That is, the cloud-side 3D engine and the terminal-side 3D engine in development state can also be set on the same device, which can be a terminal device or the cloud, without specific limitations.

[0308] The embodiments of this application have the same technical principles and effects as the embodiments described above, and can be referred to the descriptions in the embodiments described above.

[0309] In the following embodiments of this application, the term "user interface (UI)" or "interface" refers to the medium interface through which an application or operating system interacts and exchanges information with the user. It realizes the conversion between the internal form of information and the form that the user can accept. The user interface is source code written in a specific computer language such as Java or Extensible Markup Language (XML). The interface source code is parsed and rendered on the electronic device, ultimately presenting content that the user can recognize. A common form of user interface is the graphical user interface (GUI), which refers to a user interface related to computer operation displayed graphically. It can be visible interface elements such as text, icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, and widgets displayed on the screen of an electronic device.

[0310] It should be noted that the data involved in this application (including but not limited to data used for analysis, data stored, data displayed, etc.) are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0311] In one embodiment, this application also provides an electronic device, which can be the terminal device or cloud computing described in the above embodiments. (See also...) Figure 13 The electronic device may include a processor 1301 (e.g., CPU) and a memory 1302. The memory 1302 may include high-speed random-access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device. The memory 1302 may store various instructions for performing various processing functions and implementing the method steps of this application.

[0312] Optionally, the electronic device involved in this application may further include: a power supply 1303, a communication bus 1304, and a communication port 1305. The aforementioned communication port 1305 is used to enable communication between the electronic device and other peripherals. In this embodiment, the memory 1302 is used to store computer-executable program code, which includes instructions; when the processor 1301 executes the instructions, the instructions cause the processor 1301 of the electronic device to perform the actions described in the above method embodiment. The implementation principle and technical effects are similar and will not be repeated here.

[0313] Optionally, the electronic device involved in this application may further include: a display screen 1306. The display screen 1306 is used to display the interface of the electronic device.

[0314] It should be noted that the modules or components described in the above embodiments can be one or more integrated circuits configured to implement the above methods, such as one or more application-specific integrated circuits (ASICs), one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs), etc. Furthermore, when a module is implemented through processing element scheduler code, the processing element can be a general-purpose processor, such as a central processing unit (CPU) or other processors capable of calling program code, such as a controller. Additionally, these modules can be integrated together to implement a system-on-a-chip (SOC).

[0315] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., a solid-state disk (SSD)).

[0316] The term "multiple" in this document refers to two or more. The term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Furthermore, the character " / " in this document generally indicates an "or" relationship between the preceding and following related objects; in formulas, " / " indicates a "division" relationship. Additionally, it should be understood that in the description of this application, words such as "first" and "second" are used only for descriptive purposes and should not be construed as indicating or implying relative importance or order.

[0317] It is understood that the various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application.

[0318] It is understood that, in the embodiments of this application, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

Claims

1. A method for end-cloud collaborative rendering, the method comprising: Applied to a first terminal device, the method includes: A first rendering is performed on a target area in the edge scene to obtain first rendering data. The edge scene includes multiple areas, and the target area is contained within the multiple areas. Send a first request to the cloud. The first request includes information about the edge scene, the identifier of the target area, and status data. The status data includes lighting information, which is used for the second rendering of the global lighting type. The lighting information includes time and / or lighting information in the edge scene triggered by user operation. The first request is used to instruct the cloud to perform calculations before the second rendering based on the state data, and to obtain preprocessed data for the second rendering of the target area in the terminal scene; wherein the cloud includes: preprocessed data of the terminal scene at different times, the preprocessed data being obtained offline by the cloud, or the preprocessed data being pre-configured in the cloud; when the lighting information includes time, the calculations before the second rendering based on the state data to obtain preprocessed data for the second rendering of the target area in the terminal scene specifically involve: querying the first preprocessed data of the terminal scene at the time in the cloud, obtaining the second preprocessed data of the target area at the time from the first preprocessed data, and using the second preprocessed data as the preprocessed data for the second rendering of the target area; Receive the preprocessed data from the cloud; Based on the preprocessed data, the target area is rendered a second time to obtain second rendering data; Based on the first rendering data and the second rendering data, obtain the image of the target area; The image is displayed.

2. The method of claim 1, wherein, The information of the endpoint scene includes: the identifier of the endpoint scene, and / or, the information of objects in the endpoint scene, wherein the objects are objects pre-configured in the endpoint scene.

3. The method according to claim 2, characterized in that, The method further includes: In response to user actions on the object, information about the object is obtained.

4. The method according to claim 2 or 3, characterized in that, The terminal-side scenario refers to a scenario within an application, and the identifier of the terminal-side scenario includes: the identifier of the application and the identifier of the scenario; or, The identifiers of the terminal-side scenario include: the identifier of the application, the identifier of the application copy, and the identifier of the scenario.

5. The method according to any one of claims 1-3, characterized in that, The target area is related to the position of the first character in the terminal scene, and the first character corresponds to the first terminal device.

6. The method according to any one of claims 1-3, characterized in that, The first request also includes: field of view, specifically used to instruct the cloud to obtain preprocessed data for the second rendering of the field of view in the target area based on the state data.

7. The method according to any one of claims 1-3, characterized in that, The status data also includes at least one of the following: character information or scene update information.

8. The method according to any one of claims 1-3, characterized in that, The second rendering is a global illumination type rendering, which includes any one of the following: Dynamic Diffuse Global Illumination (DDGI) or global illumination based on spherical harmonics.

9. A rendering method for edge-cloud collaboration, characterized in that, Applied to the cloud, the method includes: A first request is received from a first terminal device. The first request includes information about the terminal scene, an identifier of the target area, and status data. The terminal scene includes multiple areas, and the target area is contained within the multiple areas. The status data includes lighting information, which is used for second rendering of the global lighting type. The lighting information includes time and / or lighting information in the terminal scene triggered by user operation. Based on the information of the edge scene and the identifier of the target area, the target area in the edge scene is obtained; Based on the state data, perform calculations before the second rendering to obtain preprocessed data for the second rendering of the target area in the terminal scene; Send the preprocessed data to the first terminal device; The cloud includes: preprocessed data of the edge scene at different times, wherein the preprocessed data is acquired offline by the cloud, or the preprocessed data is pre-configured in the cloud; when the lighting information includes time, the calculation based on the state data before the second rendering to obtain preprocessed data for the second rendering of the target area in the edge scene includes: Based on the time, query the cloud for the first preprocessed data corresponding to the time of the edge scene; In the first preprocessed data, the second preprocessed data corresponding to the target region at the time is obtained; The second preprocessed data is used as the preprocessed data for the second rendering of the target region.

10. The method according to claim 9, characterized in that, The information of the endpoint scene includes: the identifier of the endpoint scene, and / or, the information of objects in the endpoint scene, wherein the objects are objects pre-configured in the endpoint scene.

11. The method according to claim 10, characterized in that, The terminal-side scenario refers to a scenario within an application, and the identifier of the terminal-side scenario includes: the identifier of the application and the identifier of the scenario; or, The identifiers of the terminal-side scenario include: the identifier of the application, the identifier of the application copy, and the identifier of the scenario.

12. The method according to claim 10 or 11, characterized in that, When the information of the endpoint scene includes information about objects in the endpoint scene, obtaining the target region in the endpoint scene includes: The terminal scene is constructed based on the information of the objects in the terminal scene; Based on the identifier of the target area, obtain the target area in the edge scene.

13. The method according to claim 9, characterized in that, The first request further includes: field of view; after acquiring the second preprocessed data of the target region corresponding to the time, it further includes: In the second preprocessed data, the preprocessed data under the field of view is obtained; The step of using the second preprocessed data as preprocessed data for the second rendering of the target region in the edge scene includes: The preprocessed data under the field of view is used as the preprocessed data for the second rendering of the target area.

14. The method according to claim 9, characterized in that, When the lighting information includes lighting information in the edge scene triggered by the user operation, the step of obtaining preprocessed data for the second rendering of the target area in the edge scene based on the state data includes: Based on the pre-configured lighting at the current time and the lighting information in the edge scene triggered by the user operation, pre-processed data for the second rendering of the target area is obtained.

15. The method according to claim 14, characterized in that, The first request further includes: a field of view, and after obtaining the preprocessed data for the second rendering of the target region, it further includes: In the preprocessed data for the second rendering of the target region, preprocessed data for the field of view is obtained.

16. The method according to any one of claims 9-11 and 13-15, characterized in that, The status data also includes at least one of the following: character information or scene update information.

17. The method according to any one of claims 9-11 and 13-15, characterized in that, The endpoint scenario refers to a scenario in an application, and the method further includes: The preprocessed data is sent to a second terminal device, the application running on the second terminal device being the same as the application running on the first terminal device.

18. The method according to claim 9, characterized in that, The second rendering is a global illumination type rendering, which includes any one of the following: Dynamic Diffuse Global Illumination (DDGI) or global illumination based on spherical harmonics.

19. The method according to claim 18, characterized in that, The target area corresponds to the target space, and multiple DDGI probes are deployed in the target space. The preprocessed data includes multiple DDGI probe data, and each DDGI probe data includes illumination information at the DDGI probe and around the DDGI probe. When the second rendering is DDGI, the step of obtaining preprocessed data for the second rendering of the target region in the terminal scene based on the state data includes: Determine the target space corresponding to the target region, and the plurality of DDGI probes in the target space; Based on the illumination information, the data of the plurality of DDGI probes are obtained; The data from the multiple DDGI probes are encoded; Sending the preprocessed data to the first terminal device includes: The encoded DDGI probe data is sent to the first terminal device.

20. An electronic device, characterized in that, The electronic device includes: one or more processors and memory; The memory is coupled to the one or more processors, the memory being used to store computer program code, the computer program code including computer instructions, the one or more processors invoking the computer instructions to cause the electronic device to perform the method as described in any one of claims 1-19.

21. A chip system, characterized in that, The chip system is applied to an electronic device, the chip system including one or more processors, the one or more processors being used to invoke computer instructions to cause the electronic device to perform the method as described in any one of claims 1-19.

22. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes computer instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of claims 1-19.

23. A computer program product, characterized in that, The computer program product includes computer program code that, when run on an electronic device, causes the electronic device to perform the method as described in any one of claims 1-19.

Images