End-cloud collaborative rendering method and related device

Abstract

The invention provides an end-cloud collaborative rendering method and a related device, in the method, a terminal device can request a cloud to assist in executing pre-calculation of high-order rendering, so that the terminal device realizes high-order rendering of the terminal device by means of high GPU computing power of a cloud side, and the rendering effect of the terminal device is further improved. In addition, when the terminal equipment requests the cloud to execute pre-calculation, the terminal equipment can send data of partial areas in the scene. Therefore, the cloud end can calculate to obtain the preprocessed data of the partial region, the terminal equipment can execute basic rendering on the partial region and execute high-order rendering in combination with the preprocessed data of the partial region from the cloud end, the terminal equipment can rapidly load the picture of the partial region, the loading speed is high, picture display is not affected, and the user experience is improved. And the user experience is improved.

Description

[0001] This application is a divisional application. The application number of the original application is 202410799098.5, and the original application date is June 19, 2024. The entire contents of the original application are incorporated into this application by reference. Technical Field

[0002] The embodiments of the present application relate to the field of electronic technology, and in particular to a terminal-cloud collaborative rendering method and related devices. Background Art

[0003] There are more and more three-dimensional (3D) applications on mobile phones and other terminal devices. Providing rendering capabilities for the above 3D applications is the most core function of terminal devices to support the operation of 3D applications. However, due to the limitations of size and power consumption, the computing power of the graphics processing unit (GPU) in terminal devices is small, which is far behind that of personal computer (PC)-level GPU.

[0004] The computing power of GPUs in the cloud is large, which is a huge advantage over GPUs in terminal devices. With the development of cloud computing technology, transferring computing power from the terminal side to the cloud side and coordinating with the cloud side to process the services on the terminal side has gradually become a direction of technological evolution. At present, how to coordinate with the cloud side to achieve efficient rendering processing and thus achieve better rendering effects on the terminal side remains to be studied. Summary of the invention

[0005] The embodiments of the present application provide a rendering method and related devices for end-cloud collaboration. The terminal device can request the cloud to assist in pre-calculating high-level rendering of some areas in the end-side scene. On the one hand, with the help of the high GPU computing power of the cloud side, the rendering effect of the terminal device can be improved. On the other hand, the terminal can perform basic rendering and high-level rendering on some areas in the end-side scene, which can increase the speed of loading the picture of the part of the area and improve the user experience.

[0006] In the first aspect, an embodiment of the present application provides a rendering method for end-cloud collaboration, and the execution subject of the method may be a first terminal device or a chip in the first terminal device. The first terminal device is taken as an example for explanation below. In this method, the first terminal device may perform basic rendering on a target area in an end-side scene and obtain first rendering data, wherein the end-side scene includes multiple areas, and the target area is included in the multiple areas. The first terminal device may send a first request to the cloud, wherein the first request includes information about the end-side scene, an identifier of the target area, and status data, and the first request is used to instruct the cloud to obtain pre-processed data for high-order rendering of the target area in the end-side scene based on the status data. It should be understood that there is no order of precedence between the first terminal device sending the first request to the cloud and the first terminal device performing basic rendering on the target area in the end-side scene.

[0007] In response to the first request, the cloud can obtain preprocessing data for high-level rendering of the target area in the terminal scene. The processing process of the cloud can refer to the second aspect, or the execution steps of the cloud in the third aspect. In response to the preprocessing data from the cloud, the first terminal device can perform high-level rendering on the target area according to the preprocessing data to obtain second rendering data. The first terminal device can obtain an image of the target area according to the first rendering data and the second rendering data, and display the image.

[0008] In an embodiment of the present application, the terminal device can request the cloud to assist in performing pre-calculation of high-level rendering. With the help of the high GPU computing power on the cloud side, the rendering effect of the terminal device can be improved. In addition, when the terminal device requests the cloud to perform pre-calculation, the terminal device can send data of a partial area in the scene. In this way, the cloud can calculate the pre-processed data of the partial area, the terminal device can perform basic rendering on the partial area, and perform high-level rendering in combination with the pre-processed data of the partial area from the cloud. The terminal device can quickly load the screen of the partial area, and the loading speed is fast, which does not affect the display of the screen and the user experience.

[0009] In a possible implementation, the state data includes lighting information, the lighting information is used for high-level rendering of a global illumination type, and the lighting information includes time and / or lighting information in the terminal-side scene triggered by a user operation.

[0010] Exemplarily, the high-level rendering is a global illumination type of rendering, and the global illumination type of rendering includes any of the following: dynamic diffuse global illumination DDGI or global illumination based on spherical harmonics.

[0011] In this example, the lighting information may be preconfigured for the end-side scene, and / or the lighting information may also be triggered by a user operation. In this implementation method, the first terminal device may report the time, and / or the lighting information in the end-side scene triggered by a user operation to the cloud. Accordingly, the cloud may pre-calculate the global lighting of the target area in the end-side scene based on the time, and / or the lighting information in the end-side scene triggered by a user operation, and obtain pre-processed data, which may improve the calculation completeness and accuracy. Accordingly, when the first terminal device implements global lighting rendering, it may not only display the lighting pre-configured for the end-side scene, but also display the lighting in the end-side scene triggered by a user operation, which may improve the light and shadow effects of the picture.

[0012] In a possible implementation, the information of the terminal side scene includes: an identifier of the terminal side scene, and / or information of an object in the terminal side scene, where the object is an object preconfigured in the terminal side scene. When the information of the terminal side scene includes information of an object in the terminal side scene, a user may perform an operation on the first terminal device, and in response to the user's operation on the object, the first terminal device may obtain the information of the object.

[0013] In this implementation, end-cloud collaborative rendering can be applied not only to pre-configured end-side scenes (such as open worlds), but also to user-generated content (UGC) scenes. In UGC scenes, users can adjust objects in the end-side scenes or build end-side scenes by themselves. Correspondingly, in this scenario, the first terminal device needs to report information about objects in the end-side scene to the cloud, so that the cloud can build the end-side scene based on the information about objects in the end-side scene, so as to pre-calculate the target area of ​​the end-side scene and obtain pre-processed data.

[0014] In a possible implementation, the terminal-side scene is a scene in an application, and the identifier of the terminal-side scene includes: an identifier of the application and an identifier of the scene; or, the identifier of the terminal-side scene includes: an identifier of the application, an identifier of the application copy, and an identifier of the scene.

[0015] In a possible implementation, the user focuses on a picture around a first character, the target area is related to a position of the first character in the terminal-side scene, and the first character corresponds to the first terminal device.

[0016] In a possible implementation, the user not only pays attention to the objects in the target area, but also pays attention to the objects under the field of view of the character in the target area. In this implementation, the first request also includes: field of view, and the first request is specifically used to instruct the cloud to obtain pre-processed data for high-level rendering of the field of view in the target area based on the state data.

[0017] Correspondingly, the terminal device can perform high-order rendering on the field of view angle to obtain a more refined high-order rendering area. The terminal device can perform high-order rendering on the field of view angle and not perform high-order rendering on the non-field of view angle area in the target area, thereby reducing the high-order rendering workload of the terminal device.

[0018] In a possible implementation manner, the status data further includes at least one of the following: role information or scene update information.

[0019] In the second aspect, an embodiment of the present application provides a rendering method for end-cloud collaboration, and the execution subject of the method may be the cloud or a chip in the cloud. The cloud is taken as an example for explanation below. In the method, the cloud may receive a first request from a first terminal device, and the first request includes information about an end-side scene, an identifier of a target area, and status data. The end-side scene includes multiple areas, and the target area is included in the multiple areas. The cloud obtains the target area in the end-side scene based on the information about the end-side scene and the identifier of the target area. After obtaining the target area in the end-side scene, the cloud may obtain pre-processed data for high-order rendering of the target area in the end-side scene based on the status data, and send the pre-processed data to the first terminal device.

[0020] In a possible implementation, the state data includes lighting information, the lighting information is used for high-level rendering of a global illumination type, and the lighting information includes time and / or lighting information in the terminal-side scene triggered by a user operation.

[0021] In a possible implementation manner, the information of the terminal side scene includes: an identifier of the terminal side scene, and / or information of an object in the terminal side scene, where the object is an object preconfigured in the terminal side scene.

[0022] In a possible implementation, the terminal-side scene is a scene in an application, and the identifier of the terminal-side scene includes: an identifier of the application and an identifier of the scene; or, the identifier of the terminal-side scene includes: an identifier of the application, an identifier of the application copy, and an identifier of the scene.

[0023] In a possible implementation, when the information of the terminal side scene includes information of objects in the terminal side scene, obtaining the target area in the terminal side scene includes: the cloud constructs the terminal side scene according to the information of the objects in the terminal side scene, and, according to an identifier of the target area, obtains the target area in the terminal side scene.

[0024] In this implementation, because the end-side scene is triggered or generated by the user, and because the cloud needs to first obtain the end-side scene in order to facilitate the cloud to pre-calculate the high-order rendering of the end-side scene, the information of the end-side scene may include information about objects in the end-side scene. Accordingly, the cloud can construct the end-side scene based on the information about the objects in the end-side scene, so that the target area in the end-side scene can be pre-calculated for high-order rendering later.

[0025] In a possible implementation, the cloud includes: pre-processed data corresponding to the terminal-side scenario at different times, the pre-processed data is acquired offline by the cloud, or the pre-processed data is pre-configured in the cloud.

[0026] When the illumination information includes time, the obtaining of preprocessed data for high-level rendering of the target area in the end-side scene based on the state data includes: the cloud can query the first preprocessed data corresponding to the end-side scene at the time in the cloud according to the time. The cloud can obtain second preprocessed data corresponding to the target area at the time from the first preprocessed data, and use the second preprocessed data as preprocessed data for high-level rendering of the target area.

[0027] In this implementation, the cloud can pre-configure the pre-processed data corresponding to the terminal scene at different times, or obtain the pre-processed data corresponding to the terminal scene at different times when offline. In this way, in response to the first request, the cloud can directly query and obtain the pre-processed data without complex pre-calculation, which can improve the speed and efficiency of collaborative rendering.

[0028] In a possible implementation, the first request also includes: field of view angle, and after obtaining the second preprocessed data of the target area corresponding to the time, the cloud also includes: obtaining the preprocessed data under the field of view angle in the second preprocessed data, and using the preprocessed data under the field of view angle as preprocessed data for high-order rendering of the target area.

[0029] In this implementation, the cloud can obtain pre-processed data under the field of view angle in the target area, so that the first terminal device can render the scene under the field of view angle without processing other areas in the target area that are not under the field of view angle, and can also increase the speed of loading the screen on the first terminal device.

[0030] In one possible implementation, when the lighting information includes the lighting information in the terminal side scene triggered by the user operation, the obtaining of preprocessed data for high-order rendering of the target area in the terminal side scene based on the status data includes: the cloud obtains preprocessed data for high-order rendering of the target area based on the preconfigured lighting at the current time and the lighting information in the terminal side scene triggered by the user operation.

[0031] In a possible implementation, the first request also includes: a field of view angle, and after obtaining the preprocessed data for high-level rendering of the target area, it also includes: obtaining preprocessed data for the field of view angle in the preprocessed data for high-level rendering of the target area.

[0032] In a possible implementation manner, the status data further includes at least one of the following: role information or scene update information.

[0033] In a possible implementation, the terminal-side scenario is a scenario in an application, and the method further includes: sending the preprocessed data to a second terminal device, and the application copy running on the second terminal device is the same as the application copy running on the first terminal device.

[0034] In this implementation, the cloud can send the calculation results (preprocessed data) to the terminal devices of other users of the same application copy. In this way, the cloud can avoid repeated calculations for the same scenario of the same application, and while saving cloud resources, it can also improve the experience of other users.

[0035] In a possible implementation, the high-level rendering is a global illumination type of rendering, and the global illumination type of rendering includes any one of the following: dynamic diffuse global illumination DDGI or global illumination based on spherical harmonics.

[0036] In a possible implementation, the target area corresponds to a target space, a plurality of DDGI probes are deployed in the target space, the preprocessed data includes a plurality of DDGI probe data, and each DDGI probe data includes lighting information at and around the DDGI probe. When the high-level rendering is DDGI, the preprocessed data for high-level rendering of the target area in the end-side scene is obtained based on the state data, including: the cloud determines the target space corresponding to the target area, and the plurality of DDGI probes in the target space, and obtains the plurality of DDGI probe data based on the lighting information. In an embodiment of the present application, in order to reduce the bandwidth occupied by data transmission, the cloud may encode the plurality of DDGI probe data and send the encoded DDGI probe data to the first terminal device.

[0037] In a third aspect, an embodiment of the present application provides a terminal-cloud collaborative rendering method, which can be applied to a terminal-cloud collaborative rendering system, and the rendering system may include the first terminal device and a cloud as described above.

[0038] In this method, the first terminal device may send a first request to the cloud, the first request including information of the end-side scene, an identifier of the target area, and status data, the end-side scene including multiple areas, and the target area being included in the multiple areas. In response to the first request, the cloud may obtain the target area in the end-side scene based on the information of the end-side scene and the identifier of the target area, and obtain pre-processed data for high-level rendering of the target area in the end-side scene based on the status data. The cloud may send the pre-processed data to the first terminal device.

[0039] In this method, the first terminal device can perform basic rendering on the target area in the terminal-side scene to obtain first rendering data. After the first terminal device receives the pre-processed data from the cloud, the first terminal device can perform high-level rendering on the target area according to the pre-processed data to obtain second rendering data. The first terminal device can obtain an image of the target area according to the first rendering data and the second rendering data, and display the image.

[0040] In a possible implementation, the state data includes lighting information, the lighting information is used for high-level rendering of a global illumination type, and the lighting information includes time and / or lighting information in the terminal-side scene triggered by a user operation.

[0041] In a possible implementation manner, the information of the terminal side scene includes: an identifier of the terminal side scene, and / or information of an object in the terminal side scene, where the object is an object preconfigured in the terminal side scene.

[0042] Exemplarily, when the user is using the first terminal device, the information of the object is obtained in response to the user's operation on the object.

[0043] In a possible implementation, the terminal-side scene is a scene in an application, and the identifier of the terminal-side scene includes: an identifier of the application and an identifier of the scene; or, the identifier of the terminal-side scene includes: an identifier of the application, an identifier of the application copy, and an identifier of the scene.

[0044] In a possible implementation manner, the target area is related to a position of a first character in the terminal side scene, and the first character corresponds to the first terminal device.

[0045] In a possible implementation manner, the status data further includes at least one of the following: role information or scene update information.

[0046] In a possible implementation, the high-level rendering is a global illumination type of rendering, and the global illumination type of rendering includes any one of the following: dynamic diffuse global illumination DDGI or global illumination based on spherical harmonics.

[0047] In a possible implementation, the target area corresponds to a target space, a plurality of DDGI probes are deployed in the target space, the preprocessed data includes a plurality of DDGI probe data, each DDGI probe data includes illumination information at the DDGI probe and around the DDGI probe

[0048] When the high-level rendering is DDGI, the cloud obtains pre-processed data for high-level rendering of the target area in the terminal scene based on the status data, including: the cloud determines the target space corresponding to the target area, and the multiple DDGI probes in the target space. The cloud obtains the multiple DDGI probe data based on the lighting information, and encodes the multiple DDGI probe data. Accordingly, the cloud sends the pre-processed data to the first terminal device, including: sending the encoded DDGI probe data to the first terminal device.

[0049] In a possible implementation, when the information of the terminal side scene includes information of objects in the terminal side scene, the cloud obtains a target area in the terminal side scene, including: the cloud constructs the terminal side scene according to the information of the objects in the terminal side scene, and obtains the target area in the terminal side scene according to an identifier of the target area.

[0050] In a possible implementation, the cloud includes: pre-processed data corresponding to the terminal-side scenario at different times, the pre-processed data is acquired offline by the cloud, or the pre-processed data is pre-configured in the cloud.

[0051] When the illumination information includes time, the cloud obtains preprocessed data for high-level rendering of the target area in the client-side scene based on the state data, including: the cloud queries the first preprocessed data corresponding to the client-side scene at the time in the cloud according to the time. The cloud can obtain second preprocessed data corresponding to the target area at the time from the first preprocessed data, and use the second preprocessed data as preprocessed data for high-level rendering of the target area.

[0052] In a possible implementation, the first request also includes: field of view angle, and after the cloud obtains the second preprocessed data of the target area corresponding to the time, it also includes: the cloud obtains the preprocessed data under the field of view angle in the second preprocessed data, and uses the preprocessed data under the field of view angle as preprocessed data for high-order rendering of the target area.

[0053] In one possible implementation, when the lighting information includes the lighting information in the terminal side scene triggered by the user operation, the cloud obtains preprocessed data for high-order rendering of the target area in the terminal side scene based on the status data, including: the cloud obtains preprocessed data for high-order rendering of the target area based on the preconfigured lighting at the current time and the lighting information in the terminal side scene triggered by the user operation.

[0054] In a possible implementation, the first request further includes: a field of view angle. After the cloud obtains preprocessed data for high-level rendering of the target area, the cloud further includes: obtaining preprocessed data for the field of view angle in the preprocessed data for high-level rendering of the target area.

[0055] In a possible implementation, the rendering system may further include a second terminal device, the second terminal device running the same copy of the application as the first terminal device. The terminal side scene is a scene in the application, and the method further includes: the cloud can send the pre-processed data to the second terminal device,

[0056] In a fourth aspect, an embodiment of the present application provides an electronic device, including a processor and a memory, the memory being used to store code instructions, and the processor being used to run the code instructions to execute the methods described in the above aspects and any possible implementation methods.

[0057] In some embodiments, the electronic device may be a first terminal device, and the first terminal device may execute the method described in the first aspect and any possible implementation manner.

[0058] In some embodiments, the electronic device may be in the cloud, and the cloud may execute the method described in the second aspect and any possible implementation manner.

[0059] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program or instruction is stored. When the computer program or instruction is run on a computer, the computer executes the method described in the above aspects and any possible implementation method.

[0060] In a sixth aspect, an embodiment of the present application provides a computer program product including a computer program. When the computer program runs on a computer, the computer executes the methods described in the above aspects and any possible implementation methods.

[0061] In a seventh aspect, the present application provides a chip or a chip system, the chip or chip system comprising at least one processor and a communication interface, the communication interface and the at least one processor are interconnected by a line, and the at least one processor is used to run a computer program or instruction to execute the method described in the above aspects and any possible implementation. The communication interface in the chip can be an input / output interface, a pin or a circuit, etc.

[0062] In a possible implementation, the chip or chip system described above in the present application further includes at least one memory, in which instructions are stored. The memory may be a storage unit inside the chip, such as a register, a cache, etc., or a storage unit of the chip (e.g., a read-only memory, a random access memory, etc.).

[0063] It should be understood that the second to seventh aspects of the present application correspond to the technical solutions of the first aspect of the present application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation methods are similar and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 A schematic diagram of a system architecture applicable to the end-cloud collaborative rendering method provided in an embodiment of the present application;

[0065] Figure 2 A schematic diagram of interaction between a terminal device and a cloud in an existing end-cloud collaborative rendering method;

[0066] Figure 3A A schematic diagram of a game screen provided in an embodiment of the present application without high-level rendering;

[0067] Figure 3B A schematic diagram of a game screen provided in an embodiment of the present application when high-level rendering is performed;

[0068] Figure 4 A schematic diagram of dividing a grid in a scene provided in an embodiment of the present application;

[0069] Figure 5 A schematic diagram of a flow chart of an embodiment of a rendering method for end-cloud collaboration provided in an embodiment of the present application;

[0070] Figure 6 A schematic diagram of a scenario provided for an embodiment of the present application;

[0071] Figure 7 A schematic diagram of the distribution of DDGI probes provided in the embodiments of the present application;

[0072] Figure 8 A schematic diagram of a process for obtaining DDGI probe data;

[0073] Fig. 9 A flowchart of another embodiment of the end-cloud collaborative rendering method provided in an embodiment of the present application;

[0074] Fig.10 A schematic diagram of a scenario in which multiple terminal devices share pre-processed data according to an embodiment of the present application;

[0075] Fig.11 A schematic diagram of interaction between a terminal device and the cloud provided in an embodiment of the present application;

[0076] Fig.12 A schematic diagram of interaction between a terminal device and the cloud provided in an embodiment of the present application;

[0077] Fig.13 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0078] Three-dimensional (3D) rendering technology is the process of projecting objects in a constructed 3D scene into two-dimensional digital images according to the set viewpoint, lighting, material and other information. With the introduction of new applications (APPs), more and more 3D APPs are installed on terminal devices. 3D APPs can include but are not limited to: 3D game APPs, 3D modeling APPs, 3D navigation APPs, 3D home improvement APPs, etc. In 3D APPs, terminal devices can use 3D rendering technology to render 3D effect images to enhance user experience.

[0079] For example, taking a 3D game APP as an example, objects in a 3D scene may include, but are not limited to, characters, buildings in the game environment, plants, animals, rivers, etc. When a user plays a game, the terminal device may use 3D rendering technology to render a 3D effect game screen to enhance the user's sense of immersion.

[0080] It should be understood that different 3D apps or different scenes in the same app may have different objects in the 3D scene. For example, the objects in the scene of a 3D navigation app may include but are not limited to: vehicles, roads, buildings, pedestrians, etc. For example, the objects in the scene of a 3D home improvement app may include but are not limited to: furniture, home appliances, users, pets, etc.

[0081] In traditional 3D rendering technology, such as rasterization technology, the terminal device can segment the objects in the 3D scene with polygons, transform the three-dimensional coordinates of the polygon vertices into two-dimensional coordinates on the image through geometric transformation, and finally fill the polygons on the image with textures, thereby mapping the 3D object model to a two-dimensional screen to achieve image rendering. In some embodiments, the polygons can be triangles. Traditional 3D rendering technology is difficult to truly restore light and shadow effects such as light reflection, object shadows, and refraction in 3D scenes. Therefore, images rendered by traditional 3D rendering technology are difficult to present a realistic 3D visual experience.

[0082] Compared with traditional 3D rendering technology, ray tracing technology can provide a more realistic 3D visual experience. Ray tracing technology simulates the process of light propagation in a 3D scene through reflection, refraction, shadow, scattering, etc., calculates the color and brightness value of each position in the 3D scene, and renders the light and shadow effects of the two-dimensional image in accordance with the laws of real physics, so that a more realistic 3D virtual scene can be presented on the terminal device.

[0083] Ray tracing technology needs to simulate a large number of light paths to obtain visual effects that fit the real world, which means that the technology has high requirements for the computing power of the graphics processing unit (GPU) of the terminal device. When the terminal device is a mobile terminal such as a mobile phone or a watch, due to the limitations of the size and power consumption of the terminal device, the GPU computing power deployed in the mobile terminal is low and cannot support ray tracing technology, which results in the mobile terminal being unable to use ray tracing technology to render scenes in 3D apps, and the user's visual experience is poor. Alternatively, when the terminal device is a virtual reality (VR) device, an augmented reality (AR) device, a central control screen in a smart home, etc., in order to save costs, a high-computing-power GPU will not be deployed in the terminal device, so these terminal devices cannot use ray tracing technology to render scenes in 3D apps.

[0084] Compared with the GPUs deployed in the above terminal devices, the GPUs deployed in the cloud have a huge computing power advantage. The high-computing GPUs deployed in the cloud can support rendering technologies that require high computing power, such as ray tracing. Therefore, with the development of cloud computing technology, transferring the GPU computing power of terminal devices (end side) to the cloud side (cloud side) and coordinating the cloud side to process the business on the end side has gradually become a direction of technological evolution.

[0085] It is understandable that in the following embodiments, the terminal device represents a terminal device with low GPU computing power, and the server represents a cloud with high GPU computing power. Among them, low GPU computing power and high GPU computing power are relative concepts. In some embodiments, low GPU computing power refers to insufficient support for related calculations in ray tracing technology, and high GPU computing power refers to support for related calculations in ray tracing technology.

[0086] It should be noted that the end-cloud collaborative rendering method provided in the embodiment of the present application is not only applicable to ray tracing technology, but also applicable to other rendering technologies that require high GPU computing power. The following embodiment is described by taking ray tracing technology as an example.

[0087] In some embodiments, it can also be said that a GPU with a first computing power is deployed in the terminal device, and a GPU with a second computing power is deployed in the cloud. Among them, the second computing power is greater than the first computing power. In some embodiments, the second computing power is greater than or equal to the preset computing power, and the first computing power is less than the preset computing power. In the embodiment of the present application, the first computing power is not enough to support the relevant calculations in the ray tracing technology, and the second computing power can support the relevant calculations in the ray tracing technology.

[0088] In some embodiments, it can also be said that a PC-level GPU is deployed in the cloud.

[0089] The following is an introduction to the system architecture applicable to the end-cloud collaborative rendering method provided in the embodiment of the present application:

[0090] Figure 1 The system architecture to which the embodiments of the present application are applicable is shown exemplarily. Figure 1 As shown, the system architecture may include a terminal device 100 and a cloud.

[0091] In some embodiments, the terminal device 100 may be a mobile phone, a tablet computer, a desktop computer, a laptop computer, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a cellular phone, a personal digital assistant (PDA), an augmented reality (AR) device, a virtual reality (VR) device, an artificial intelligence (AI) device, a wearable device (such as a smart bracelet), a vehicle-mounted device, a smart home device (such as a smart TV, a smart screen, a large-screen device, etc.) and / or a smart city device. The embodiment of the present application does not impose any special restrictions on the specific type of the terminal device 100, and the terminal device 100 may also be referred to as a terminal side or an electronic device. In the following embodiments, the first terminal device and the second terminal device are taken as examples.

[0092] In some embodiments, the terminal device 100 may be a mobile device or a non-mobile device.

[0093] In some embodiments, the cloud may include a server 200. The server 200 may be a server, or a server cluster consisting of multiple servers, or a cloud computing center. The server 200 involved in the embodiment of the present application may also be referred to as a cloud server, a cloud side, or a cloud. Not limited to the server 200, the cloud may also include more other devices, which are not specifically limited here.

[0094] The terminal device 100 can communicate with the cloud through a communication network, and the cloud can provide rendering services for 3D apps for the terminal device 100.

[0095] In some embodiments, the communication network may include, but is not limited to, local area networks (LAN), wide area networks (WAN), etc.

[0096] It should be understood that Figure 1The architecture shown does not constitute a specific limitation on the system architecture, and the system architecture may include more or fewer devices than shown. For example, the system architecture may also include a wireless relay device and a wireless backhaul device ( Figure 1 ), which is not shown in the figure and is not limited here.

[0097] Combine the following Figure 1 The system architecture shown here introduces the rendering method of end-cloud collaboration:

[0098] Taking a 3D APP as a game APP as an example, in some embodiments, all rendering work in the game APP in the terminal device can be transferred to the cloud, and the GPU computing power of the cloud can be used to improve the rendering effect of the terminal device. Figure 2 As shown, the terminal device is installed with a client of the game APP, which may include an operation instruction processing module and a video decoding module. The server of the game APP can be run in the cloud, and the server may include a rendering module, a logic module, and a virtual host / container. All rendering work on the server can be performed in the virtual host / container in the cloud.

[0099] When a user plays a game on a terminal device, the operation instruction processing module of the terminal device can collect the user's operation and upload the operation instruction triggered by the user's operation to the server of the game APP. Among them, the user's operation may include but is not limited to moving left and right, switching perspectives, clicking, etc. The server of the game APP sends the received operation instruction to the logic module. The logic module can update the game status according to the operation instruction and trigger the rendering module to render the game scene in the latest status. The server of the game APP transmits the rendered game screen to the client of the game APP in the form of a video stream. The video decoding module of the client of the game APP decodes the video stream to obtain the game screen, and triggers the terminal device to display the game screen.

[0100] In this example, relying on the powerful computing power of PC-level GPUs in the cloud, the rendering module in the cloud can achieve basic rendering of game scenes, as well as advanced rendering of game scenes using ray tracing technology.

[0101] Basic rendering and advanced rendering are relative concepts. The GPU computing power required for basic rendering is relatively low, and the GPU computing power of the terminal device can support the completion of basic rendering. The picture of the game scene obtained based on basic rendering can ensure the normal operation of the game APP and the normal interaction of user operations. In some embodiments, the cloud can use rasterization technology to perform basic rendering of the game scene, and the embodiments of this application do not specifically limit the basic rendering technology.

[0102] The GPU computing power required for high-level rendering is relatively high. The GPU computing power of the terminal device cannot support the relevant calculations for completing high-level rendering. The GPU computing power of the cloud can independently complete high-level rendering. High-level rendering is usually easy to decouple from basic rendering. The high-level rendering in the embodiment of the present application can be, for example, rendering performed using ray tracing technology.

[0103] In some embodiments, high-level rendering can be a rendering task related to lighting, such as rendering of lighting characteristics such as global illumination (GI), ambient occlusion (AO), soft shadows (SoftShadow), reflection, refraction, and caustics, and can also include rendering of other characteristics that are relatively intensive in cloud computing.

[0104] In some embodiments, high-level rendering can be various forms of global illumination (GI). For example, dynamic diffuse global illumination (DDGI), spherical harmonics-based global illumination, voxel-based global illumination, or point-based global illumination. Figure 3A and Figure 3B , taking GI as an example, this paper introduces the rendering effects without and with high-level rendering:

[0105] For example, Figure 3A and Figure 3B It shows the same game screen of the game app without and with advanced rendering (such as GI). Figure 3A As shown in the figure, the game screen can run normally without GI, but the picture effect is dim and lacks the light and shadow details of indirect lighting. For example, the game screen lacks the light and shadow details formed by the reflection of objects on the wall. Figure 3B As shown, when GI is used, the picture effect will be brighter, more realistic and fuller, and the game screen will also display various light and shadow details.

[0106] Implementing the above-mentioned end-cloud collaborative rendering method, using the high-computing GPU in the cloud to perform all rendering work of the game scene, can achieve the purpose of enhancing the picture effect when the terminal device displays the game scene, and can reduce the computing power requirements and load of the GPU of the terminal device, and has a wider range of applications. However, the above-mentioned rendering method still has the following problems:

[0107] (1) All rendering work (basic rendering and advanced rendering) is performed in the cloud, which has high computing costs. The terminal device is only used as a video player, wasting the GPU resources of the terminal device.

[0108] (2) Poor network experience. For users, they often encounter poor network conditions. When the network of the terminal device is poor, the basic rendering is also performed by the cloud, and the terminal device cannot display the basic game screen, affecting the normal operation of the game APP. In addition, the user's operation response delay is very high, affecting the game operation experience.

[0109] (3) High transmission bandwidth cost. The cloud uses video streaming to transmit rendered game images to terminal devices. In the case of 1080P@60fps, the transmission bandwidth is about 10Mbps, resulting in high costs for both the cloud egress bandwidth and the terminal device downlink bandwidth.

[0110] (4) The rendering results of the cloud cannot be shared among multiple users. For users participating in the same game, the cloud needs to repeatedly calculate all the rendering work of the game scene, further increasing the computing cost of the cloud.

[0111] In summary, at present, how to collaborate with the cloud to achieve efficient rendering processing, so as to achieve better rendering effects on the end side (terminal device), still needs to be studied.

[0112] At present, the computing power of the GPU in the terminal device can support basic rendering, but cannot support high-order rendering. The reason is that: in the high-order rendering process, it is necessary to perform calculations before high-order rendering based on multiple data in the scene, and then render according to the calculation results. Among them, the calculation before high-order rendering can be called pre-calculation, and the calculation result can be called pre-calculation result, pre-calculation data or pre-processed data. Pre-calculation requires high GPU computing power support, but the computing power of the GPU deployed in the terminal device is low and cannot support pre-calculation. However, the GPU computing power required for rendering based on the calculation results is low, and the GPU deployed in the terminal device can support this part of the rendering work. Accordingly, the embodiment of the present application can provide another end-cloud collaborative rendering method, in which the terminal device can perform basic rendering, and the terminal device can send the data in the scene required for pre-calculation to the cloud, and with the help of the high GPU computing power of the cloud, enable the cloud to perform pre-calculation based on multiple data in the scene. The cloud can send the calculation results to the terminal device, and the terminal device performs high-order rendering according to the calculation results.

[0113] In this way, the terminal device can perform basic rendering. When the network where the terminal device is located is poor, the terminal device can display the basic game screen and the game APP can run normally. In addition, the cloud can perform pre-calculation, which can assist the terminal device to achieve high-level rendering and improve the display effect of the game screen. In addition, because the terminal device can perform basic rendering and high-level rendering based on the calculation results of the cloud, the GPU resources of the terminal device will not be wasted, so efficient rendering processing can be achieved. In summary, the end-cloud collaborative rendering method can solve the above problems (1) and (2).

[0114] In addition, because the cloud no longer performs basic rendering and high-level rendering, but performs calculations before high-level rendering, the cloud will not use video streaming to transmit the rendered game screen, but will send calculation results with low bandwidth usage. In this way, the end-cloud collaborative rendering method can solve the above problem (3).

[0115] In addition, in response to the above question (4), in the end-cloud collaborative rendering method provided in the embodiment of the present application, the cloud can send the calculation results to the terminal devices of other players participating in the game APP. In this way, for the same scene of the game APP, the cloud can avoid repeated calculations, and while saving cloud resources, it can also improve the gaming experience of other players.

[0116] In some embodiments, in a 3D APP, such as a game APP, an open world scene is included. In an open world scene, users can freely handle the virtual world, which is opposed to the linear and structured game world. In an open world scene, the scene scale is relatively large, for example, the scene scale can be 10km×10km. In this scenario, due to the limitation of the GPU computing power of the terminal device, the terminal device cannot load the entire scene at one time. Accordingly, when the terminal device requests the cloud to perform pre-calculation, the terminal device will not send the data of the entire scene to the cloud, but will send partial data. Otherwise, after the cloud obtains the calculation results of the entire scene, the terminal device cannot load the entire scene, or it takes a long time to load the entire scene, and the game screen updates for a long time, which will also affect the user's gaming experience.

[0117] In view of this technical problem, combined with the above technical concept, in the embodiment of the present application, when the terminal device requests the cloud to perform pre-calculation, the terminal device can send the data of a part of the scene. In this way, the terminal device can perform basic rendering on the part of the area, and perform high-level rendering in combination with the calculation results of the part of the area from the cloud. The terminal device can quickly load the screen of the part of the area, with a fast loading speed, without affecting the display of the game screen and the user's gaming experience.

[0118] Based on the above technical concept, the end-cloud collaborative rendering method provided by the embodiment of the present application is described in detail below. In order to facilitate understanding of the end-cloud collaborative rendering method provided by the embodiment of the present application, the terminology involved in the end-cloud collaborative rendering method is first introduced:

[0119] 1) End-side scene: It can be understood as the scene in the 3D APP in the terminal device. Exemplarily, taking the game APP as an example, the end-side scene can be different game scenes. The end-side scene in the embodiment of the present application is a 3D scene. The 3D scene can be regarded as a virtual three-dimensional scene environment (such as a city, park, forest, street, mountain and river). It can provide a multimedia virtual world. Users can control the characters in the virtual scene by operating the terminal device and observe objects, animals, people, scenery and other objects in the virtual scene from the perspective of the characters. In addition, users can also switch perspectives by operating the terminal device, such as observing objects in the virtual scene from a God's perspective or a third-person perspective.

[0120] In some embodiments, taking a game APP as an example, the terminal-side scene may be a scene of any game level, or a scene of any perspective in any game level.

[0121] 2) Regions in the end-side scene: Due to the large scale of the scene, the terminal device cannot load the entire scene at one time. In this case, the entire scene can be divided into multiple regions, and the size and shape of each region can be equal or different. The region can be a 3D region or a plane region in the scene.

[0122] In some embodiments, the entire scene can be divided into fine-grained grids for ease of calculation. In this example, the area in the end-side scene can be a grid. For example, if the scene scale is 10km×10km, it can be divided into 100m×100m granularity. Figure 4 As shown in the rectangle, the terminal device can divide the scene into n×n grids, each grid has the same size. Where n is an integer greater than or equal to 2. Figure 4 As shown, n is 5.

[0123] It should be understood that Figure 4 Different grids are distinguished by letters + numbers, which are examples.

[0124] 3) Basic rendering and advanced rendering: Basic rendering and advanced rendering are relative concepts. For basic rendering and advanced rendering, please refer to the description in the above embodiments.

[0125] In the embodiment of the present application, the GPU computing power required for basic rendering is relatively low, and the terminal device can independently complete the basic rendering. Among them, the terminal device performs basic rendering, and the generated game scene screen can ensure the normal operation of the game APP and the normal interaction of user operations. When the network of the communication network between the terminal device and the cloud is poor or disconnected, and the terminal device does not obtain the calculation results of the high-order rendering fed back by the cloud, the game APP of the terminal device can directly display the game screen after basic rendering to ensure the normal operation and viewing of the game APP by the user.

[0126] The GPU computing power required for high-level rendering is relatively high, and the terminal device cannot independently complete high-level rendering. In the embodiment of the present application, the estimated calculation of high-level rendering can only be completed independently by the GPU computing power of the cloud, and high-level rendering tasks are usually easy to decouple from basic rendering tasks.

[0127] In some embodiments, the terminal device can also determine the type of high-level rendering that can be currently performed for end-cloud collaborative rendering based on real-time network conditions. In some embodiments, developers can divide applications (such as game apps) into basic rendering and high-level rendering based on actual application requirements and the GPU capabilities of the terminal device, which is not specifically limited here.

[0128] 4) APP development state: The development state can also be called the production state. In the APP development state, developers can develop the APP's logic code and scenarios. After the development is completed, the developer can package the developed related files into an APP application package that can be run on the terminal device for release, such as the Android application package (APK).

[0129] 5) APP running state: The terminal device can download the APP and run the APP application package on the terminal device. When the APP is in the running state, the terminal device can display the APP interface based on the logic code, scenes, etc. in the application package, and interact with users, the cloud, etc.

[0130] 6) Application copy: An application can include at least one application copy. Taking a game app as an example, a game copy refers to a scene or area in a scene that allows multiple users (such as teammates) to be free from interference from other users (non-teammates), and other users cannot enter the scene or area in the scene.

[0131] The following is a description of the end-cloud collaborative rendering method provided by the embodiments of the present application in conjunction with specific embodiments. The following embodiments may be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0132] Figure 5A flowchart of an embodiment of a rendering method for end-cloud collaboration provided in an embodiment of the present application. Figure 5 The end-cloud collaborative rendering method provided in the embodiment of the present application may include:

[0133] S501: A first terminal device performs basic rendering on a target area in a terminal-side scene to obtain first rendering data. The terminal-side scene includes multiple areas, and the target area is included in the multiple areas.

[0134] The end-side scene and the area in the end-side scene can refer to the above-mentioned terminology explanation. Among them, the end-side scene includes multiple areas, and the target area is included in the multiple areas. In other words, the first terminal device can perform basic rendering on a partial area in the end-side scene, and the partial area can be called the target area. The first terminal device performs basic rendering on the target area in the end-side scene, and can obtain first rendering data.

[0135] In some embodiments, the first rendering data may be a basic rendering image, or intermediate data used in the process of generating a basic rendering image (for example, the irradiance of each pixel obtained by basic rendering), or image data of a basic rendering image. The GPU computing power required for the rendering technology used in the basic rendering is what the terminal device can provide, and the embodiments of the present application do not specifically limit the rendering technology. In some embodiments, the terminal device may use a rasterization method to perform basic rendering on the target area in the terminal-side scene to obtain the first rendering data.

[0136] Taking a game APP as an example, the game scene includes a first character, which can be a character created by the user in the game APP. The first character corresponds to the first terminal device. The correspondence between the first character and the first terminal device can be understood as: the first character is the character when the user logs into the game APP on the first terminal device, or the first character is the character when the user plays the game on the first terminal device.

[0137] When a user plays a game, the user pays attention to the scene around the first character. In some embodiments, the target area is related to the position of the first character in the client-side scene.

[0138] For example, the target area may be the area where the first character is located. Figure 4 For example, illustratively, when the first character is in a G(2, 2) grid, the target grid may be a G(2, 2) grid.

[0139] For example, the target area may be the area where the first character is located, and the area around the first character. Figure 4For example, illustratively, when the first character is in the G(2, 2) grid, the target grid may be four surrounding grids including the G(2, 2) grid. For example, the target grid may be G(2, 2), G(2, 3), G(3, 2), and G(3, 3). For example, when the first character is in the G(2, 2) grid, the target grid may be nine grids centered on the G(2, 2) grid. For example, the target grid may be: G(1, 1), G(1, 2), G(1, 3), G(2, 1), G(2, 2), G(2, 3), G(3, 1), G(3, 2), and G(3, 3).

[0140] In some embodiments, the target area may be an area specified by the user. For example, in a terminal side scenario, if the user needs to view objects in a certain area or several areas, the user can click the corresponding area on the terminal device, and the one or several areas can be regarded as the target area. In this example, the target areas may be adjacent or non-adjacent.

[0141] In some embodiments, the number of target grids may be related to the GPU computing power of the terminal device. The higher the GPU computing power of the terminal device, the more target grids the terminal device supports loading. The embodiment of the present application does not limit the number of target grids.

[0142] In one possible scenario, when the target area is an area specified by the user, for example, the GPU computing power of the terminal device supports 3 areas. When the user clicks to select the target area, when the user has selected 3 areas and is selecting the 4th area, the terminal device can prompt the user that the area has been selected and no more areas can be loaded.

[0143] S502, the first terminal device sends a first request to the cloud, where the first request includes information about the terminal scene, an identifier of the target area, and status data.

[0144] In some embodiments, there is no order distinction between S501 and S502, and the two can be executed simultaneously. In the embodiment of the present application, the first terminal device can perform basic rendering on the target area, and the pre-calculation requiring high GPU computing power can be performed with the help of high GPU computing power in the cloud.

[0145] The first terminal device may send a first request to the cloud, and the first request is used to instruct the cloud to perform pre-calculation before high-order rendering. The first request may include information about the end-side scene, an identifier of the target area, and status data. Specifically, the first request is used to instruct the cloud to obtain pre-processed data for high-order rendering of the target area in the end-side scene based on the status data. It should be understood that the cloud performs pre-calculation before high-order rendering based on the status data, and can obtain the calculation result (which may be referred to as pre-processed data) for high-order rendering of the target area in the end-side scene.

[0146] The following describes the terminal scene information, the target area identifier, and the status data.

[0147] First, information about the client side scenario

[0148] The information of the terminal side scene may include: an identifier of the terminal side scene, and / or information of objects in the terminal side scene, wherein the objects in the terminal side scene are pre-configured in the terminal side scene.

[0149] 1) In some embodiments, the end-side scene can be determined in the development state of the APP and remain unchanged in the running state of the APP. For example, the scene in a pursuit game APP is determined in the development state of the APP, and the user cannot modify the scene in the game APP during the user's use of the game APP.

[0150] In this embodiment, the client of the APP of the first terminal device is pre-configured with the identifier of the terminal scene, and the server of the APP in the cloud can also be pre-configured with the identifier of the terminal scene. The identifier of the terminal scene is used to distinguish different terminal scenes, for example, number 1 is a city scene, number 2 is a mountain scene. Or,

[0151] The client of the APP of the first terminal device is pre-configured with the identifier of the terminal scene, and the server of the APP in the cloud is pre-configured with the identifier of the cloud scene. The terminal scene corresponds to the cloud scene, and the identifier of the terminal scene and the identifier of the cloud scene can be the same or different. It can be understood that the identifier of the terminal scene and the identifier of the cloud scene can correspond to the same scene. The purpose of such setting is that the terminal side and the cloud side have different data storage methods, so the identifiers for the storage of the same data can also be different. Such setting can adapt to the current storage capacity of the terminal side and the cloud side without modifying the current storage logic of the terminal side and the cloud side. Exemplarily, the terminal side can use numbers 1, 2, 3, etc. to distinguish different terminal side scenes, and the cloud side can use numbers 1-1, 1-2, 1-3, etc. to distinguish different cloud side scenes. Among them, for example, the terminal scene 1 can correspond to the cloud side scene 1-1, and the terminal scene 1 and the cloud side scene 1-1 are the same city scene. Similarly, for example, the terminal scene 2 can correspond to the cloud side scene 1-2, and the terminal scene 2 and the cloud side scene 1-2 are the same mountain scene.

[0152] In some embodiments, the information of the end-side scene may include an identifier of the end-side scene. The end-side scene may be a scene in an application, and the identifier of the end-side scene may include: an identifier of the application and an identifier of the scene in the application. For an application that includes an application copy, the same end-side scene may be different in different application copies, so the identifier of the end-side scene may also include an identifier of the application copy. In some embodiments, taking a game APP as an example, the game APP includes many game levels, and the scenes in different game levels are also different, so the identifier of the end-side scene may also include an identifier of the level.

[0153] It should be understood that the identifier can be regarded as information used to uniquely distinguish a certain content (such as a terminal scene, application, application copy, level, etc.). For example, the identifier can be information such as a number and a name.

[0154] In this embodiment, when the cloud is configured with an identifier of the end-side scenario, in response to the first request, the cloud can determine the identifier of the end-side scenario. When the cloud is configured with an identifier of the cloud-side scenario, in response to the first request, the cloud can convert the identifier of the end-side scenario into the identifier of the cloud-side scenario.

[0155] 2) In some embodiments, the end-side scene can be determined in the development state of the APP, and the end-side scene can be changed in the running state of the APP. Exemplarily, in a user-generated content (UGC) scenario, the end-side scene can be triggered to change by the user. For example, taking the home improvement APP as an example, the objects in the scene of the home improvement APP may include but are not limited to: TV, sofa, bookcase, etc., and the objects in the scene can be determined in the development state of the APP. In the process of using the home improvement APP, the user can place objects at will, and / or adjust the size of the objects, etc. The position of the objects in the scene and / or change, that is, the scene changes.

[0156] In this embodiment, the information of the end-side scene may include: the identification of the end-side scene, and the information of the object in the end-side scene. The object is an object preconfigured in the end-side scene. For example, in the scene of the home improvement APP, the object may be a house, and a TV, sofa, bookcase, etc. that can be placed in the house. The information of the object in the end-side scene may include, but is not limited to: the identification, style description, quantity, location, size, color, etc. of the object.

[0157] 3) In some embodiments, for an APP with a single end-side scene, such as a game APP for decorating rooms used by younger users, the APP may contain a scene, for example, a scene of a room. In this example, the end-side scene can be determined in the development state of the APP, and the end-side scene can be changed in the running state. Exemplarily, the user can change the position of the object in the room, and the scene will also change. In this scenario, the information of the objects in the end-side scene may include: the size, pattern, and distribution of the room, as well as the position of the characters, desks, beds, etc. in the room.

[0158] However, in the APP, since only one scene is included, the scene information on the terminal side may not include the identifier of the scene, and the cloud side can determine the scene of the APP. In this example, the information of the scene on the terminal side may include: information of objects in the scene on the terminal side.

[0159] For example, the scene of a design APP includes objects (basic objects) such as a sphere, a cone, and a cube. In the design APP, users can use the objects to arrange the scene. Figure 6 , the scene may include a sphere, three cones, and three cubes. Correspondingly, the information of the client-side scene may include: the number, size, color, position, etc. of the spheres, cones, and cubes. Figure 6 The first request sent by the terminal device to the cloud includes information about objects in the terminal scene.

[0160] Second, identification of the target area

[0161] In the embodiment of the present application, the identification of the target area can be the number, name, position of the target area in the scene, or the position of the target area relative to the center of the scene, etc., which is not limited in the embodiment of the present application. For example, taking the target area as a grid, the identification of the grid can be the number of the grid, such as G(2,2).

[0162] Third, status data

[0163] State data is the data used by the cloud to perform pre-calculation. In other words, the state data is used by the cloud to obtain pre-processed data, which is used for high-level rendering of the target area in the end-side scene. In other words, the terminal device can perform high-level rendering of the target area in the end-side scene based on the pre-processed data.

[0164] Different types of high-level rendering require different state data used by the cloud for pre-calculation. For example, if high-level rendering is a rendering task related to lighting, the state data may be lighting information. Lighting information may include, but is not limited to: the type, position, quantity, lighting direction, intensity, color, etc. of the light source. For example, if high-level rendering is a rendering task related to a character, the state data may be character information. Character information may include, but is not limited to: the character's age, clothing, skin condition, occupation, etc.

[0165] Taking high-level rendering as an example, which is a rendering task related to lighting, the state data may include lighting information.

[0166] 1) In some embodiments, the lighting in the scene is determined in the development state of the APP and remains unchanged in the running state of the APP. For example, the lighting in the scene can change dynamically according to a certain time rule. For example, the intensity, color, and direction of the lighting in the scene are pre-configured based on time changes. Exemplarily, in the morning, the intensity of the lighting in the scene is intensity 1, the direction of the lighting is from the east, and the color of the lighting is neutral light. In the evening, the intensity of the lighting in the scene is intensity 2, the direction of the lighting is from the west, and the color of the light is warm light. Exemplarily, the color temperature of neutral light can be, for example, 4000K, and the color temperature of warm light can be, for example, 3000K, which is not limited in the embodiments of the present application.

[0167] In this embodiment, because the illumination in the scene changes dynamically according to a certain time rule, the illumination in the scene is related to time. In some embodiments, the illumination information may include time.

[0168] In some embodiments, when the illumination in the scene changes dynamically according to a certain time rule, the illumination information may be empty. In this embodiment, in response to the illumination information being empty, the cloud may determine that the illumination information is the current time.

[0169] 2) In some embodiments, the lighting in the scene is determined in the development state of the APP, and the lighting can be changed in the running state of the APP. Exemplarily, taking a game APP as an example, for example, releasing skills, opening and closing doors, and lighting torches can all change the lighting in the scene. The release of skills, opening and closing doors, and lighting torches are often triggered by users. For example, a user operates a skill button to trigger the release of a skill. In this embodiment, in addition to the pre-configured lighting, the scene can also include lighting triggered by the user, which can affect the lighting in the scene. User-triggered lighting can also be understood as: lighting in the end-side scene triggered by user operation.

[0170] In this embodiment, the state data may include: time, and user trigger data. The user trigger data is used to affect the lighting of the end-side scene. Exemplarily, the user trigger data may include: operation data of a user clicking a button on the interface. Based on the operation data, the cloud can determine the lighting information triggered by the operation data, and the lighting information may include but is not limited to: the type, intensity, color, direction, and position of the lighting. The types of lighting can be divided into sunlight, light, torch light, and skill light.

[0171] Exemplarily, in response to a user clicking a button on the interface, the terminal device (eg, a game APP) may determine the lighting information triggered by the operation data. In this example, the user-triggered data may include: lighting information in the terminal scene triggered by the user operation.

[0172] In this embodiment, the state data may not include time. Accordingly, in response to the illumination information, the cloud may determine that the time is the current time.

[0173] S503: The cloud obtains the target area in the terminal side scene according to the information of the terminal side scene and the identifier of the target area.

[0174] In an embodiment of the present application, in response to the first request, the cloud can first obtain the target area in the end-side scene based on the information of the end-side scene and the identifier of the target area, and then pre-calculate the target area in the end-side scene based on the state data. In some embodiments, the process of the cloud obtaining the target area in the end-side scene can also be called the process of the cloud converting the end-side scene into the cloud-side scene.

[0175] The following describes a method for the cloud to obtain a target area in the terminal-side scenario in combination with the information of the terminal-side scenario in “One” in S502:

[0176] 1) In some embodiments, the terminal-side scenario can be determined in the development state of the APP and remains unchanged in the running state of the APP.

[0177] In this example, the information of the end-side scene may include an identifier of the end-side scene. In some embodiments, the cloud can query the target area in the end-side scene in the database according to the identifier of the end-side scene and the identifier of the target area. The database may be local to the cloud or independently set. The database may store various end-side scenes in different APPs.

[0178] In some embodiments, the cloud can convert the identifier of the terminal side scene into the identifier of the cloud side scene, and obtain the cloud side scene according to the identifier of the cloud side scene. The cloud can obtain the target area in the cloud side scene according to the identifier of the target area.

[0179] In some embodiments, the cloud or database may store information about the object corresponding to the identifier of the end-side scene, or information about the object corresponding to the identifier of the cloud-side scene. In this embodiment, the cloud can determine the information about the object corresponding to the identifier of the end-side scene based on the identifier of the end-side scene, that is, the information about the object contained in the end-side scene. The cloud can construct the end-side scene based on the information about the object contained in the end-side scene, and then obtain the target area in the end-side scene based on the identifier of the target area. Alternatively, in this embodiment, the cloud can first convert the identifier of the end-side scene into the identifier of the cloud-side scene, and then based on the information about the object corresponding to the identifier of the cloud-side scene, that is, the information about the object contained in the cloud-side scene. The cloud can construct the cloud-side scene (corresponding to the end-side scene) based on the information about the object contained in the cloud-side scene, and then obtain the target area in the cloud-side scene based on the identifier of the target area. Among them, the target area in the cloud-side scene can be regarded as the target area in the end-side scene.

[0180] 2) In some embodiments, the end-side scenario can be determined in the development state of the APP, and the end-side scenario can be changed in the running state of the APP.

[0181] Exemplarily, when the information of the end-side scene includes the identifier of the end-side scene and the information of the object in the end-side scene, the cloud can determine the end-side scene according to the identifier of the end-side scene, and refer to the description in 1). The cloud can place the object in the determined end-side scene according to the information of the object in the end-side scene and update the end-side scene. The cloud can obtain the target area in the end-side scene in the updated end-side scene according to the identifier of the target area.

[0182] Exemplarily, when the information of the terminal side scene includes information of objects in the terminal side scene, the cloud can first construct the terminal side scene according to the information of the objects in the terminal side scene, and then obtain the target area in the terminal side scene according to the identifier of the target area.

[0183] S504: The cloud obtains pre-processed data for high-level rendering of the target area in the terminal scene based on the status data.

[0184] In response to the first request, the cloud can obtain preprocessed data for high-order rendering of the target area in the terminal scene based on the state data. In other words, the cloud can perform precalculation based on the state data to obtain preprocessed data. The preprocessed data is used by the terminal device to perform high-order rendering on the target area in the terminal scene.

[0185] After obtaining the target area in the end-side scene, the cloud can pre-calculate the target area in the end-side scene based on the lighting information in the status data, and obtain pre-processed data for high-level rendering of the target area in the end-side scene. The following describes the process of obtaining pre-processed data for high-level rendering of the target area in the end-side scene based on different scenarios:

[0186] 1) In some embodiments, the lighting in the scene is determined in the development state of the APP and remains unchanged in the running state of the APP.

[0187] Case 1: In this embodiment, the pre-processed data of each end-side scenario or the pre-processed data of the cloud-side scenario corresponding to each end-side scenario can be pre-configured in the cloud. The following is an example of the pre-processed data of each end-side scenario being pre-configured in the cloud:

[0188] In case 1, the illumination in the end-side scene may change dynamically according to a certain time rule, and the pre-processed data of each end-side scene at each time may be pre-configured in the cloud. In this embodiment, the first request may include the time, or the first request may be empty. In addition, the first request also includes the information of the end-side scene and the identification of the target area.

[0189] In response to the first request, the cloud can determine the end-side scene and the target area in the end-side scene. In some embodiments, the cloud can query the preprocessed data of the end-side scene in the preconfigured preprocessed data. When the first request can include time, the cloud can obtain the first preprocessed data corresponding to the end-side scene at the time in the preprocessed data of the end-side scene. When the first request is empty, the cloud can obtain the first preprocessed data corresponding to the end-side scene at the current time in the preprocessed data of the end-side scene.

[0190] It should be understood that the division of the target area in the end-side scene is preconfigured, and the end-side and cloud-side are known in advance or can be queried. After obtaining the first preprocessed data, the cloud side can also obtain the preprocessed data corresponding to the target area in the first preprocessed data according to the identifier of the target area, which can be called the second preprocessed data. In the embodiment of the present application, the second preprocessed data can be used as "preprocessed data for high-order rendering of the target area in the end-side scene".

[0191] It is understandable that the cloud can also obtain preprocessed data corresponding to the target area of ​​the end-side scene in the preprocessed data of the end-side scene based on the information of the end-side scene and the identification of the target area. In the preprocessed data, the cloud can further obtain the preprocessed data corresponding to the target area of ​​the end-side scene at the time based on time, which is "preprocessed data for high-order rendering of the target area in the end-side scene".

[0192] Similar to pre-configuring the pre-processing data for each end-side scene, the cloud can pre-configure the pre-processing data for the cloud-side scene corresponding to each end-side scene. In this example, in response to the first request, the cloud can first convert the end-side scene into a cloud-side scene, and then obtain "pre-processing data for high-level rendering of the target area in the end-side scene" from the pre-processing data of the cloud-side scene.

[0193] Case 2: The cloud can calculate the preprocessed data of each end-side scene when it is offline, and store the preprocessed data of each end-side scene. Alternatively, the cloud can obtain the preprocessed data of the cloud-side scene corresponding to each end-side scene when it is offline, and store the preprocessed data of the cloud-side scene corresponding to each end-side scene. The following is an example of the cloud storing the preprocessed data of each end-side scene:

[0194] In case 2, the illumination in the end-side scene can change dynamically according to a certain time rule. The cloud can calculate the pre-processed data of each end-side scene at each time according to the illumination at each time when it is offline. The calculation process of the cloud can refer to Figure 7-Figure 9 Description in .

[0195] In this example, the first request may include time, or the first request is empty. In addition, the first request also includes information about the end-side scene and an identifier of the target area. In response to the first request, the cloud can obtain preprocessed data for high-level rendering of the target area in the end-side scene from the stored preprocessed data, and the description in Case 1 can be referred to.

[0196] Similar to calculating the preprocessing data of each end-side scene in an offline state, the cloud can calculate the preprocessing data of the cloud-side scene corresponding to each end-side scene in an offline state. In this example, in response to the first request, the cloud can first convert the end-side scene into a cloud-side scene, and then obtain the preprocessing data for high-order rendering of the target area in the end-side scene from the preprocessing data of the cloud-side scene, which can refer to the description in Case 1.

[0197] 2) In some embodiments, the lighting in the scene is determined in the development state of the APP, and the lighting can be changed in the running state of the APP.

[0198] In this example, in response to the first request, the cloud can calculate in real time preprocessing data for high-level rendering of the target area in the client scene. In this embodiment, the first request can include user trigger data and time, or the first request can include user trigger data.

[0199] In response to the first request, the cloud can determine the pre-configured lighting in the end-side scene based on the time (or the current time), and determine the user-triggered lighting based on the user-triggered data. The cloud can calculate the pre-processed data for high-level rendering of the target area in the end-side scene based on the pre-configured lighting in the end-side scene and the user-triggered lighting. The calculation process of the cloud can refer to Figure 7-Figure 9 Description in .

[0200] In some embodiments, high-level rendering is a rendering task related to lighting characteristics, and the state information includes lighting information. Accordingly, the preprocessed data is used to perform high-level rendering on a target area in a client-side scene to obtain a light and shadow effect of the high-level rendering.

[0201] The preprocessing data corresponding to different high-level rendering types may be different, and accordingly, the first rendering data may also be different. For example, if the high-level rendering type is GI rendering, the preprocessing data of GI includes the irradiance of each pixel after GI rendering, and the first rendering data may include the irradiance of each pixel after basic rendering. For example, if the high-level rendering type is reflection rendering, the preprocessing data of reflection rendering includes a cubemap captured by a reflection probe, and the first rendering data may include the color of each pixel after basic rendering.

[0202] S505, the cloud sends pre-processed data to the first terminal device.

[0203] S506: The first terminal device performs high-order rendering on the target area according to the preprocessed data to obtain second rendering data.

[0204] The first terminal device receives the preprocessed data and can perform high-level rendering on the target area according to the preprocessed data to obtain second rendering data. Exemplarily, the first terminal device can use a shader to color the target area in combination with the preprocessed data of the target area to complete high-level rendering.

[0205] S507: The first terminal device obtains an image of the target area according to the first rendering data and the second rendering data.

[0206] S508: The first terminal device displays an image of the target area.

[0207] In S507 and S508, the first rendering data and the second rendering data correspond to the target area in the terminal side scene, and the first rendering data and the second rendering data may both include rendering data of each pixel in the target area. The first terminal device may fuse the first rendering data and the second rendering data according to the corresponding relationship of the pixels to obtain an image of the target area. The first terminal device may send the image of the target area for display to display the image of the target area.

[0208] It should be understood that when the first terminal device displays the image of the target area, it also displays the images of other areas in the terminal side scene, but the image of the target area is the image updated by the cloud, and the images of other areas are not updated.

[0209] In an embodiment of the present application, the terminal device can request the cloud to assist in performing pre-calculation of high-order rendering, and the technical effect can refer to the description in the above embodiment. In addition, when the terminal device requests the cloud to perform pre-calculation, the terminal device can send data of a partial area in the scene. In this way, the cloud can calculate the pre-processed data of the partial area, and the terminal device can perform basic rendering on the partial area, and perform high-order rendering in combination with the pre-processed data of the partial area from the cloud. The terminal device can quickly load the screen of the partial area, and the loading speed is fast, which does not affect the display of the screen and the user experience.

[0210] The following takes the cloud computing DDGI based on state data (lighting information) as an example to describe the cloud pre-calculation process. Among them, DDGI can be regarded as pre-processed data for the target area of ​​the end-side scene. In some embodiments, when the cloud calculates DDGI based on lighting information, the pre-processed data can exist in the form of DDGI probe data. To facilitate the understanding of DDGI probe data, the following first introduces GI probes and GI probe data:

[0211] GI probes are essentially sampling points used to store lighting information at different locations in the scene. One way to place GI probes is to arrange them evenly in a 3D scene at a certain density. After that, each GI probe can collect or detect lighting from all directions with its location as the core, and record the lighting information in a certain cache format to obtain GI probe data. When the terminal device needs to render a shading point, it only needs to find several GI probes around the shading point, and interpolate the lighting information stored in these GI probes to obtain the lighting information at the shading point.

[0212] In some embodiments, the cache format may be, for example, an octahedral mapping format or a spherical harmonic coefficient format. For example, when the illumination information at the GI probe is cached in an octahedral mapping format, the GI probe data may include octahedral mapped texture data. When the illumination information at the GI probe is cached in a spherical harmonic coefficient format, the GI probe data may include spherical harmonic coefficients.

[0213] In summary, when the cloud calculates GI based on lighting information, the pre-processed data obtained can be GI probe data. Similarly, when the cloud calculates DDGI based on lighting information, the pre-processed data obtained is DDGI probe data. Among them, DDGI probe data can include lighting information of DDGI probes, which can also be called texture data of DDGI probes.

[0214] Figure 7 A schematic diagram showing the distribution of DDGI probes in the scene. Figure 7 , the scene includes the ground and houses. Figure 7 In the figure, the ground is represented by a quadrilateral, and the DDGI probes distributed in the scene are represented by black dots. Figure 7 The different sizes of black dots represent the DDGI probes distributed at different locations in the end-side scene.

[0215] In some embodiments, the DDGI probes in the scenario (end-side scenario) may be pre-deployed in the terminal device and the cloud, and both the cloud and the terminal device may determine the information of the DDGI probes in each end-side scenario. For example, the information of the DDGI probes may include, but is not limited to, the number and distribution of the DDGI probes.

[0216] In some embodiments, the DDGI probes in the end-side scenario can be pre-deployed in the terminal device, and the terminal device can determine the information of the DDGI probes in each end-side scenario. In this example, when the terminal device requests the cloud to perform pre-calculation, it can send the information of the DDGI probes in the end-side scenario to the cloud. In this example, for example, the first request can also include the information of the DDGI probes in the end-side scenario. In this way, in response to the first request, the cloud can determine the information of the DDGI probes in the end-side scenario.

[0217] In some embodiments, reference Figure 8 , the calculation process of DDGI probe data can include the following steps:

[0218] Step 1: Count the volumes that need to be updated.

[0219] Volumes can be understood as the 3D space where DDGI probes are distributed. In some embodiments, regions in the end-side scene have a mapping relationship with volumes. For example, one region corresponds to one volume, or one region corresponds to multiple volumes, or multiple regions correspond to one volume, and the mapping relationship between regions and volumes can be preconfigured.

[0220] In some embodiments, the size of the volume may be related to the GPU computing resources on the cloud side and the terminal side.

[0221] In some embodiments, the size of the volume may be constant, for example, the mapping relationship between the region and the volume may be preconfigured. In some embodiments, the size of the volume may change with the GPU computing resources on the cloud side and / or the end side, and the mapping relationship between the region and the volume may also change accordingly.

[0222] In some embodiments, in response to the first request, the cloud can determine the target area in the end-side scene that needs to be updated, and based on the mapping relationship between the area and the volume, determine the volumes that need to be updated. The volumes that need to be updated can be called target volumes or target spaces.

[0223] Step 2: Update Volumes in a loop to obtain the DDGI probe data in Volumes.

[0224] Looping and updating Volumes once can be understood as obtaining the DDGI probe data in Volumes once.

[0225] The following describes step 2 by taking the process of obtaining DDGI probe data in Volumes as an example. In some embodiments, updating Volumes may include the following steps:

[0226] Step 2-1: The DDGI probe emits light to find the intersection and returns the lighting result (RT radiance).

[0227] The DDGI probe in the volume emits light, and the light is reflected when it hits the object in the client scene. In this way, the location of the DDGI probe is used as the core, and the light is collected or detected in all directions to obtain the light at the DDGI probe in the volume. The light at the DDGI probe can include values ​​such as Irradiance and Distance. Among them, Irradiance represents the irradiance information at the DDGI probe, and Distance provides the DDGI probe offset information for the final rendering and shading.

[0228] Step 2-2: DDGI probe irradiance information blending (Irradiance Blend).

[0229] For each DDGI probe, the irradiance information at the DDGI probe and the surrounding DDGI probes may be mixed to obtain the comprehensive irradiance information at the DDGI probe.

[0230] Step 2-3: DDGI probe distance blending (Distance Blend).

[0231] This step involves calculating the visibility (or degree of occlusion) of the light at each DDGI probe. In some embodiments, distance blending may also be referred to as visibility blending. Exemplarily, for each DDGI probe, the visibility item of the light at the DDGI probe may be determined based on the distance of the surrounding DDGI probes or other factors.

[0232] Step 2-4: DDGI probe irradiance information edge copy (Distance Border Update).

[0233] After determining that the DDGI probe irradiance information is mixed, the irradiance information at the DDGI probe boundary needs to be updated to ensure information consistency.

[0234] Step 2-5, DDGI probe distance edge copy (Distance Border Update).

[0235] After determining the visibility of the light at the DDGI probe, the visibility at the DDGI probe boundary needs to be updated to ensure information consistency.

[0236] Step 2-6: Adjust the position of the DDGI probe (RelocateProbes).

[0237] In some embodiments, steps 2-6 are optional steps.

[0238] In some embodiments, the position of the DDGI probe may be adjusted to better capture the illumination at the DDGI probe.

[0239] Step 2-7, DDGI probe status mark (ClassifyProbes)

[0240] In some embodiments, steps 2-7 are optional steps.

[0241] In some embodiments, the status mark may, for example, mark whether the data at the DDGI probe has been updated or not updated, so as to timely update each DDGI probe data.

[0242] In some embodiments, step 2 above can also be briefly described as follows:

[0243] The cloud uses the DDGI algorithm to obtain DDGI probe data in Volumes. In some embodiments, DDGI probes can be used to store the lighting information of the scene and dynamically updated using ray tracing, thereby achieving real-time dynamic diffuse global illumination effects. The DDGI algorithm packages a group of DDGI probes into one or several Volumes. You only need to drag the Volume into the scene to be rendered, and Volumes will automatically place DDGI probes in it. The shading points in the Volume will automatically capture lighting information through the surrounding DDGI probes.

[0244] For any shading point in the volume (e.g., shading point 1), for example, the terminal device may obtain 8 DDGI probe data around shading point 1. The terminal device interpolates the irradiance according to the 8 DDGI probe data to obtain the irradiance of shading point 1. In some embodiments, the pre-processed data of DDGI may include the irradiance of the shading point corresponding to each pixel in the target area of ​​the terminal scene.

[0245] In some embodiments, the DDGI probe stores spherical information. The DDGI algorithm encodes the spherical data into a two-dimensional texture map through octahedral mapping. The minimum unit of the texture map is a texel, and one texel corresponds to one or more pixels. The DDGI probe data may include the irradiance received from the hemisphere in the direction of the texel (w), the distance r(w) from the nearest object seen from the texel direction to the probe, and the square of the distance r 2 (w). The irradiance is encoded as a three-dimensional vector texture, r(w) and r 2 (w) are encoded together as a two-dimensional vector texture (the x component stores r(w), and the y component stores r 2 (w)).

[0246] In one implementation, probe 1 is any one of the above eight DDGI probes. For example, the terminal device interpolates the irradiance according to the above eight DDGI probe data to obtain the irradiance of shading point 1. Specifically, it may include: obtaining three weight coefficients of probe 1, namely, trilinear difference coefficient, directional coefficient and Chebyshev coefficient, and taking the normalized value of the product of the above three coefficients as the weight of probe 1. Based on the weight of each probe, the irradiance of the above eight DDGI probes is weighted to obtain the irradiance of shading point 1.

[0247] Among them, the trilinear difference coefficient indicates the distance between probe 1 and shading point 1. If the coefficient is large, the weight of probe 1 is reduced; the direction coefficient indicates the angle between the direction of shading point 1 pointing to probe 1 and the surface normal of shading point 1. If the coefficient is too large, the weight of probe 1 is reduced. The Chebyshev coefficient indicates the probability that there is an occluder between probe 1 and shading point 1. If the coefficient is large, the weight of probe 1 is reduced. The Chebyshev coefficient is based on the above distance r(w) and the distance square r 2 (w) Determined.

[0248] Step 3: Fill the volumes with textures (discripter).

[0249] Based on the data of each DDGI probe in the Volume, the entire Volume can be filled with textures. Filling textures can be understood as filling the Volume with lighting textures.

[0250] Step 4: Render the Volumes (corresponding to the target area of ​​the end-side scene).

[0251] The resolution reduction may be configured in step 4. For example, in order to improve rendering efficiency, the DDGI rendering may be converted into a low-resolution version.

[0252] Step 5: Upsampling.

[0253] In some embodiments, because the DDGI rendering can be converted into a low-resolution version in step 4, raw sampling can be performed in step 5 to restore the target area of ​​the end-side scene to the original resolution to improve the picture clarity of the target area of ​​the end-side scene.

[0254] It should be understood that the process of obtaining DDGI probe data is briefly described in the embodiments of the present application, and the specific algorithm can refer to the detailed description of the DDGI algorithm in the prior art.

[0255] In some embodiments of the present application, refer to Fig. 9 , the cloud can perform steps 1 and 2 to obtain the DDGI probe data of the target area of ​​the end-side scene. After the cloud obtains the DDGI probe data of the target area of ​​the end-side scene, it can perform the following steps:

[0256] Step 3A: The cloud encodes the DDGI probe data in Volumes.

[0257] Step 3A may also be referred to as texture encoding.

[0258] In some embodiments, the cloud may use a preset encoding algorithm to encode the DDGI probe data in the Volumes.

[0259] Step 4A: The cloud sends the encoded DDGI probe data to the terminal device.

[0260] The encoded DDGI probe data may be understood as encoded texture data.

[0261] Step 5A: The terminal device decodes the DDGI probe data.

[0262] In some embodiments, the terminal device may use a decoding algorithm corresponding to a preset encoding algorithm to decode the received preprocessed data (DDGI probe data) to obtain the DDGI probe data of the target area.

[0263] Step 6A: The terminal device performs high-level rendering on the target area based on the DDGI probe data to obtain second rendering data.

[0264] In some embodiments, the target area of ​​the terminal scene includes at least one shading point. The terminal device performs high-order rendering on the target area as follows: the terminal device queries the DDGI probe at the shading point in the DDGI probe data, and shades the shading point according to the DDGI probe data at the shading point and the adjacent DDGI probe data, thereby completing the rendering of the shading point. Based on the same method, the terminal device can render each shading point in the target area to complete the high-order rendering of the target area.

[0265] Step 7A: The terminal device performs super-resolution processing.

[0266] In some embodiments, step 7A may be an optional step.

[0267] The purpose of super-resolution processing in step 7A is the same as that of upsampling in step 5, which is to restore the target area of ​​the end-side scene to the original resolution to improve the image clarity of the target area of ​​the end-side scene. Super-resolution refers to super resolution (SR), which is an underlying image processing task that maps a low-resolution image to a high-resolution image in order to enhance image details.

[0268] Reference Fig. 9 In an embodiment of the present application, the terminal device can send lighting information of a target area of ​​a terminal-side scene to the cloud, and the cloud can perform pre-calculation based on the lighting information to obtain pre-processed data such as DDGI probe data. The cloud can send pre-processed data to the terminal device so that the terminal device can perform high-order rendering on the target area of ​​the terminal-side scene based on the pre-processed data. In this embodiment, the calculation of DDGI probe data is achieved by relying on the high GPU computing power of the cloud, so that the terminal device can achieve high-order rendering of global illumination. In addition, the cloud performs pre-calculation, and the terminal device performs high-order rendering based on the pre-processed data. It can also comprehensively utilize the GPU resources of the cloud and the GPU resources of the terminal device to achieve joint use without wasting the GPU resources of the terminal device.

[0269] When playing games, users not only pay attention to the objects in the target area, but also pay attention to the objects under the field of view of the character in the target area. In order to obtain more refined pre-processing data and reduce the high-order rendering workload of the terminal device, in some embodiments, the first request may also include the field of view (FOV). The field of view is also called the viewing angle. Taking the game APP as an example, when the user plays the game, the objects in the terminal scene can be observed from different perspectives such as the first perspective, the third perspective, and the God perspective.

[0270] In this embodiment, the first request may include: information about the end-side scene, an identifier of the target area, state data, and the field of view. It should be understood that the target area may cover the field of view so that the cloud can fully obtain the pre-processed data for high-order rendering of the field of view. Accordingly, in response to the first request, the cloud can obtain the pre-processed data under the field of view for the target area.

[0271] In some embodiments, referring to the description in 1) and 2) of S504, after the cloud obtains the preprocessed data (such as the second preprocessed data) for high-order rendering of the target area of ​​the end-side scene, the cloud can also obtain the preprocessed data under the field of view angle from the preprocessed data (such as the second preprocessed data) for high-order rendering of the target area of ​​the end-side scene. Exemplarily, the cloud can retain the preprocessed data under the field of view angle in the target area, delete the preprocessed data under the non-field of view angle in the target area, and obtain the preprocessed data under the field of view angle.

[0272] It should be understood that the non-viewing angle in the target area can be understood as: the area other than the viewing angle in the target area.

[0273] In this embodiment, the cloud can send preprocessed data at the field of view angle in the target area to the terminal device. The terminal device can perform basic rendering on the field of view angle of the target area of ​​the end-side scene to obtain first rendering data. When the terminal receives the preprocessed data at the field of view angle in the target area from the cloud, it can perform high-order rendering on the field of view angle of the target area of ​​the end-side scene based on the preprocessed data at the field of view angle in the target area to obtain second rendering data. The terminal device can also obtain an image of the field of view angle of the target area of ​​the end-side scene based on the first rendering data and the second rendering data to display the image of the field of view angle of the target area of ​​the end-side scene.

[0274] In the embodiment of the present application, when the terminal device requests pre-processed data for high-level rendering from the cloud, the field of view angle can also be sent to the cloud so that the cloud can calculate the field of view angle for the target area of ​​the terminal scene and obtain the pre-processed data. Accordingly, the terminal device can perform high-level rendering on the field of view angle, and can obtain a more refined high-level rendering area. The terminal device can perform high-level rendering on the field of view angle, and does not perform high-level rendering on the non-field of view angle area in the target area, which can reduce the high-level rendering workload of the terminal device.

[0275] The above embodiments introduce a process in which the cloud obtains preprocessed data for high-order rendering of a target area of ​​a terminal-side scene, and preprocessed data for high-order rendering of a target area of ​​a terminal-side scene at a specific field of view. In some embodiments, after the cloud obtains the above preprocessed data, it can not only send the preprocessed data to the first terminal device, but also send the preprocessed data to the terminal devices of other players who use the APP, thereby avoiding repeated precalculations of the same target area of ​​the same scene by the cloud, thereby reducing the computing workload of the cloud.

[0276] For example, taking a game APP as an example, multiple players can be in the same game copy, and the multiple players can be in the same game scene. If each player's terminal device requests the pre-processed data of the game scene from the cloud, the cloud needs to perform multiple pre-calculations. Fig.10 Taking the terminal devices of multiple players including a first terminal device and a second terminal device as an example, in response to a first request from the first terminal device, the cloud can perform a pre-calculation to obtain pre-processed data of the target area (or the field of view angle of the target area) in the game scene.

[0277] In some embodiments, because the first request includes the identifier of the application copy, the cloud can query the second terminal device in the same game copy as the first terminal device based on the identifier of the game copy. The cloud can send the pre-processed data to the first terminal device and the second terminal device.

[0278] Among them, in response to the preprocessing data, the first terminal device can perform high-order rendering on the target area (or the field of view angle of the target area) and display the image of the target area (or the field of view angle of the target area). For the second terminal device, when the interface displayed by the second terminal device includes the target area (or the field of view angle of the target area), the second terminal device can also perform high-order rendering on the target area (or the field of view angle of the target area) according to the preprocessing data, so that the second terminal device can update the screen of the target area (or the field of view angle of the target area).

[0279] It is conceivable that when the target area (or the field of view angle of the target area) is not included in the interface displayed by the second terminal device, the second terminal device may not perform high-order rendering on the target area (or the field of view angle of the target area). As the second character corresponding to the second terminal device moves in the game scene, when the interface displayed by the second terminal device includes the target area (or the field of view angle of the target area), the second terminal device performs high-order rendering on the target area (or the field of view angle of the target area) based on the preprocessed data, so that the second terminal device can update the screen of the target area (or the field of view angle of the target area), which can speed up the speed at which the second terminal device updates the screen.

[0280] In the embodiment of the present application, when the end-side scene is a scene in which multiple users (or players) participate, the cloud does not need to perform a pre-calculation for each user, but only needs to perform a pre-calculation once and share it with other users. In this way, in the end-cloud collaborative rendering solution, the consumption of cloud-side GPU resources can be further reduced.

[0281] In the above embodiment, the rendering method for end-cloud collaboration provided by the embodiment of the present application is introduced from the perspective of the cloud and the first terminal device. Figure 11-Figure 12 The internal modules of the cloud and the first terminal device introduce the end-cloud collaborative rendering method provided by the embodiment of the present application from the perspective of internal module interaction.

[0282] Fig.11 A schematic diagram of an interaction between a terminal device and a cloud provided in an embodiment of the present application, wherein the terminal device may be a first terminal device or a second terminal device.

[0283] Reference Fig.11 , the terminal device may include: an APP. The APP may be a 3D APP. In some embodiments, the APP may include: a logic module and a rendering module. The cloud may include a high-order rendering pre-computation module and a virtual host / container. The pre-computation in the cloud may be performed in the virtual host / container.

[0284] The rendering module is used to perform basic rendering on the target area of ​​the end-side scene to obtain first rendering data. In some embodiments, the rendering module is also used to perform basic rendering on the field of view angle of the target area of ​​the end-side scene to obtain first rendering data.

[0285] The logic module is used to determine the information of the terminal scene, the identification of the target area, and the state data in response to the user's operation. In some embodiments, the logic module is also used to determine the field of view angle.

[0286] The logic module is further configured to send a first request to the high-order rendering pre-computation module. The first request may include information about the end-side scene, an identifier of the target area, and state data. Alternatively, the first request may include: information about the end-side scene, an identifier of the target area, state data, and a field of view.

[0287] The high-order rendering pre-computation module is used to obtain pre-processed data of the target area (or the field of view angle of the target area) for the terminal scene in response to the first request, and send the pre-processed data to the rendering module. The method for the high-order rendering pre-computation module to obtain pre-processed data can refer to the description in the above embodiment.

[0288] The rendering module is also used to perform high-order rendering on the target area (or the field of view angle of the target area) of the terminal scene according to the preprocessing data to obtain the second rendering data. The rendering module is also used to obtain the image of the target area (or the field of view angle of the target area) based on the first rendering data and the second rendering data for display, so that the terminal device displays the image of the target area (or the field of view angle of the target area).

[0289] The embodiments of the present application have the same technical principles and technical effects as the above embodiments, and reference may be made to the description in the above embodiments.

[0290] Fig.12 Another schematic diagram of interaction between a terminal device and the cloud provided in an embodiment of the present application. Fig.12 The terminal device includes: APP, terminal-side 3D engine, and terminal-cloud collaboration plug-in SDK. The cloud includes: high-level rendering service, terminal-cloud collaboration framework, and cloud-side 3D engine.

[0291] The APP may include logic code, which is the code used to implement the logic of the APP.

[0292] End-side 3D engine: The end-side 3D engine includes main modules such as engine framework, scene management, and rendering pipeline. The engine framework is used to provide logic analysis, model analysis, animation analysis and other related capabilities for the end-side APP. Scene management is used to manage the scene information of one or more scenes (i.e., end-side scenes) built by the APP. The rendering pipeline is used to convert the three-dimensional scene model into a two-dimensional space of screen pixels and output the scene image. Specifically, the rendering pipeline includes the following functions: converting the 3D coordinates of the object into 2D coordinates of the screen pixel space, and coloring each pixel on the screen.

[0293] The end-side 3D engine is divided into two modes: development state and running state. Among them, the end-side 3D engine provides developers with an integrated development environment (IDE) in the development state, allowing developers to develop the logic code of the APP and the rendering effect of the 3D scene. After the development is completed, the developer can package the developed related files into an application package of the game APP that can be run on the terminal device, such as an Android application package. The end-side 3D engine provides the APP with a Runtime runtime in the running state. The Runtime runtime generally has cross-platform capabilities and can support the APP to run on mobile devices with different operating systems such as Hongmeng, Android and iOS. In the embodiment of the present application, when the APP is running at Runtime, the above-mentioned rendering pipeline can be called to perform basic rendering on the end-side scene to be rendered and generate the first rendering data. In addition, when the APP is running at Runtime, it can also call the end-cloud collaborative plug-in SDK to request the cloud to perform pre-calculation of high-order rendering. And the first rendering data and the second rendering data are fused, and a displayable high-order rendering image can be obtained based on the fused data.

[0294] End-cloud collaboration plug-in SDK: This SDK encapsulates the key capabilities of end-cloud collaborative rendering, and based on the plug-in mechanism provided by the end-side 3D engine, enables the third-party end-side 3D engine to have the ability of end-cloud collaborative rendering. The end-cloud collaboration plug-in SDK includes some or all of the following sub-functions: state synchronization, transmission communication, decoding, and pipeline adaptation. In the embodiment of the present application, end-cloud collaborative rendering is achieved by providing an end-cloud collaboration plug-in SDK to the third-party 3D engine, thereby avoiding invasive modifications to the third-party 3D engine.

[0295] State synchronization on the client side: used to synchronize state data to the cloud side. Exemplarily, the state data may include some or all of the following: lighting information (such as the type, position, posture, brightness, and quantity of light sources), character information (such as the position, posture, animation, skills, and quantity of virtual characters), camera information (such as the position, posture, and field of view (FOV) of the camera), and scene update information (such as information related to object movement, object animation, and destruction).

[0296] Transmission communication on the client side: used to upload the client-side state synchronization data to the cloud side, and to receive the high-level rendering pre-processing data sent by the cloud side.

[0297] Decoding on the client side: used to decode the compressed and encoded pre-processed data sent from the cloud side into pre-processed data that can be consumed by the rendering pipeline on the client side.

[0298] End-side pipeline adaptation: used to add the consumption rendering capability of the decoded pre-processed data based on the original end-side rendering pipeline according to the extension mechanism of the end-side 3D engine rendering pipeline, that is, the capability to fuse the first rendering data and the second rendering data to obtain a high-level rendering effect.

[0299] In some embodiments, the high-level rendering service may include a pre-calculation module. The pre-calculation module is configured to perform pre-calculation based on the state data in response to the first request and obtain pre-processing data.

[0300] Cloud-side 3D engine: includes major functional modules such as scene management, rendering pipeline, and engine framework, and is used to provide a capability base for the end-cloud collaborative framework and high-level rendering services.

[0301] End-cloud collaboration framework: includes some or all of the following sub-functions: state synchronization, transmission communication, decoding, and session management.

[0302] State synchronization on the cloud side: used to receive a first request from the end side, and based on the information of the end side scene in the first request, update the scene information of the corresponding cloud side scene in the scene management module of the cloud side 3D engine.

[0303] Cloud-side encoding: used to compress and encode pre-processed data of high-level rendering services to reduce the amount of transmitted data.

[0304] Transmission communication on the cloud side: used to receive status data transmitted by the end side and send compressed and encoded pre-processed data to the end side.

[0305] Session management: used to assign clients connected to the end side to the corresponding high-level rendering services. For example, the assignment is based on the game ID, level / scene ID, copy ID, and other information uploaded by the end side. It can be understood that the cloud can establish sessions with multiple end-side clients and the high-level rendering services corresponding to the sessions, so that high-level rendering services can be provided to multiple clients (terminal devices) at the same time.

[0306] Based on the cloud-side 3D engine and the end-cloud collaborative framework, it provides high-level rendering services, that is, high-level rendering pre-calculation, such as GI pre-calculation, AO pre-calculation, reflection pre-calculation, soft shadow pre-calculation and other high-level rendering services.

[0307] In some embodiments, in order to save cloud-side GPU computing power and facilitate development and debugging, during the development stage of the APP, Fig.12 The modules in the terminal device and the cloud shown can be simulated, developed and tested on the same device, that is, the cloud-side 3D engine and the terminal-side 3D engine in development state can also be set in the same device, which can be a terminal device or a cloud, and no specific limitation is made here.

[0308] The embodiments of the present application have the same technical principles and technical effects as the above embodiments, and reference may be made to the description in the above embodiments.

[0309] The term "user interface (UI)" or interface in the following embodiments of the present application refers to a medium interface for interaction and information exchange between an application or operating system and a user, which realizes the conversion between the internal form of information and the form acceptable to the user. The user interface is a source code written in a specific computer language such as Java and extensible markup language (XML). The interface source code is parsed and rendered on an electronic device, and finally presented as content that can be recognized by the user. The commonly used form of user interface is a graphical user interface (GUI), which refers to a user interface related to computer operation displayed in a graphical manner. It can be a visual interface element such as text, icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets, etc. displayed on the display screen of an electronic device.

[0310] It should be noted that the data involved in this application (including but not limited to data used for analysis, stored data, displayed data, etc.) are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0311] In one embodiment, the present application also provides an electronic device, which may be the terminal device or the cloud in the above embodiment. Fig.13 , the electronic device may include: a processor 1301 (such as a CPU), and a memory 1302. The memory 1302 may include a high-speed random access memory (RAM), and may also include a non-volatile memory (NVM), such as at least one disk memory. The memory 1302 may store various instructions for completing various processing functions and implementing the method steps of the present application.

[0312] Optionally, the electronic device involved in the present application may further include: a power supply 1303, a communication bus 1304 and a communication port 1305. The above-mentioned communication port 1305 is used to realize the connection and communication between the electronic device and other peripherals. In the embodiment of the present application, the memory 1302 is used to store computer executable program code, and the program code includes instructions; when the processor 1301 executes the instruction, the instruction causes the processor 1301 of the electronic device to perform the action in the above-mentioned method embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.

[0313] Optionally, the electronic device involved in the present application may further include: a display screen 1306. The display screen 1306 is used to display an interface of the electronic device.

[0314] It should be noted that the modules or components described in the above embodiments may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASICs), or one or more digital signal processors (DSPs), or one or more field programmable gate arrays (FPGAs). For another example, when a module above is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processors that can call program codes, such as a controller. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0315] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loading and executing computer program instructions on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server, a data center, etc. that contains one or more available media integrated. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0316] The term "plurality" in this article refers to two or more than two. The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the previous and next associated objects are in an "or" relationship; in the formula, the character " / " indicates that the previous and next associated objects are in a "division" relationship. In addition, it should be understood that in the description of this application, words such as "first" and "second" are only used for the purpose of distinguishing descriptions, and cannot be understood as indicating or implying relative importance, nor can they be understood as indicating or implying order.

[0317] It should be understood that the various numerical numbers involved in the embodiments of the present application are only used for the convenience of description and are not used to limit the scope of the embodiments of the present application.

[0318] It can be understood that in the embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

Claims

1. A rendering method for end-cloud collaboration, characterized in that: Applied to a first terminal device, the method includes: Performing a first rendering on a target area in a terminal-side scene to obtain first rendering data, wherein the terminal-side scene includes a plurality of areas, and the target area is included in the plurality of areas; Sending a first request to the cloud, where the first request includes information about the terminal-side scene, an identifier of the target area, and status data, and the first request is used to instruct the cloud to obtain preprocessing data for second rendering of the target area in the terminal-side scene based on the status data; receiving the pre-processed data from the cloud; Performing a second rendering on the target area according to the preprocessed data to obtain second rendering data; Acquire an image of the target area according to the first rendering data and the second rendering data; The image is displayed.

2. The method according to claim 1, characterized in that The state data includes lighting information, where the lighting information is used for second rendering of the global illumination type, and the lighting information includes time and / or lighting information in the terminal-side scene triggered by a user operation.

3. The method according to claim 1 or 2, characterized in that: The information of the terminal side scene includes: an identifier of the terminal side scene, and / or information of an object in the terminal side scene, where the object is an object pre-configured in the terminal side scene.

4. The method according to claim 3, characterized in that The method further comprises: In response to a user's operation on the object, information about the object is acquired.

5. The method according to claim 3 or 4, characterized in that: The terminal side scene is a scene in an application, and the identifier of the terminal side scene includes: an identifier of the application and an identifier of the scene; or, The identifier of the terminal-side scenario includes: an identifier of the application, an identifier of the application copy, and an identifier of the scenario.

6. The method according to any one of claims 1 to 5, characterized in that The target area is related to a position of a first character in the terminal-side scene, and the first character corresponds to the first terminal device.

7. The method according to any one of claims 1 to 6, characterized in that The first request also includes: field of view angle, and the first request is specifically used to instruct the cloud to obtain pre-processed data for second rendering of the field of view angle in the target area based on the status data.

8. The method according to any one of claims 1 to 7, characterized in that The state data also includes at least one of the following: role information or scene update information.

9. The method according to any one of claims 1 to 8, characterized in that The second rendering is a global illumination type of rendering, and the global illumination type of rendering includes any one of the following: dynamic diffuse global illumination DDGI or global illumination based on spherical harmonics.

10. A rendering method for end-cloud collaboration, characterized in that: Applied to the cloud, the method includes: receiving a first request from a first terminal device, the first request including information of a terminal side scenario, an identifier of a target area, and status data, the terminal side scenario including multiple areas, and the target area being included in the multiple areas; Acquire the target area in the terminal side scene according to the information of the terminal side scene and the identifier of the target area; Based on the state data, obtaining preprocessing data for second rendering of the target area in the terminal side scene; Sending the preprocessed data to the first terminal device.

11. The method according to claim 10, characterized in that The state data includes lighting information, where the lighting information is used for second rendering of the global illumination type, and the lighting information includes time and / or lighting information in the terminal-side scene triggered by a user operation.

12. The method according to claim 10 or 11, characterized in that: The information of the terminal side scene includes: an identifier of the terminal side scene, and / or information of an object in the terminal side scene, where the object is an object pre-configured in the terminal side scene.

13. The method according to claim 12, characterized in that The terminal side scene is a scene in an application, and the identifier of the terminal side scene includes: an identifier of the application and an identifier of the scene; or, The identifier of the terminal-side scenario includes: an identifier of the application, an identifier of the application copy, and an identifier of the scenario.

14. The method according to claim 12 or 13, characterized in that When the information of the terminal side scene includes information of an object in the terminal side scene, acquiring the target area in the terminal side scene includes: Constructing the terminal side scene according to the information of the objects in the terminal side scene; According to the identifier of the target area, the target area in the terminal side scene is acquired.

15. The method according to claim 11, characterized in that The cloud includes: pre-processed data corresponding to the terminal-side scenario at different times, the pre-processed data is obtained offline by the cloud, or the pre-processed data is pre-configured in the cloud; When the illumination information includes time, acquiring, based on the state data, preprocessing data for second rendering of the target area in the terminal-side scene includes: According to the time, querying the cloud for first pre-processed data corresponding to the terminal-side scenario at the time; Acquire, from the first preprocessed data, second preprocessed data corresponding to the target area at the time; The second pre-processed data is used as pre-processed data for second rendering of the target area.

16. The method according to claim 15, characterized in that The first request further includes: a field of view angle, and after obtaining the second pre-processed data of the target area corresponding to the time, the step further includes: In the second preprocessed data, obtaining preprocessed data under the field of view angle; The using the second pre-processed data as pre-processed data for second rendering of the target area in the terminal-side scene includes: The preprocessed data under the field of view angle is used as preprocessed data for second rendering of the target area.

17. The method according to claim 11, characterized in that When the illumination information includes illumination information in the end-side scene triggered by the user operation, acquiring, based on the state data, preprocessing data for second rendering of the target area in the end-side scene includes: According to the preconfigured lighting at the current time and the lighting information in the terminal-side scene triggered by the user operation, preprocessing data for the second rendering of the target area is acquired.

18. The method according to claim 17, characterized in that The first request further includes: a field of view angle, and after obtaining the pre-processed data for the second rendering of the target area, the method further includes: In the preprocessed data for the second rendering of the target area, preprocessed data for the field of view angle is obtained.

19. The method according to any one of claims 10 to 18, characterized in that The state data also includes at least one of the following: role information or scene update information.

20. The method according to any one of claims 10 to 19, characterized in that The terminal side scenario is a scenario in an application, and the method further includes: The preprocessed data is sent to a second terminal device, where the application copy running on the second terminal device is the same as the application copy running on the first terminal device.

21. The method according to claim 11, characterized in that The second rendering is a global illumination type of rendering, and the global illumination type of rendering includes any one of the following: dynamic diffuse global illumination DDGI or global illumination based on spherical harmonics.

22. The method according to claim 21, characterized in that The target area corresponds to a target space, a plurality of DDGI probes are deployed in the target space, the preprocessed data includes a plurality of DDGI probe data, each DDGI probe data includes illumination information at the DDGI probe and around the DDGI probe; When the second rendering is DDGI, the acquiring, based on the state data, preprocessing data for the second rendering of the target area in the terminal-side scene includes: Determine a target space corresponding to the target area, and the plurality of DDGI probes in the target space; According to the illumination information, acquiring the plurality of DDGI probe data; encoding the plurality of DDGI probe data; The sending the pre-processed data to the first terminal device includes: The encoded DDGI probe data is sent to the first terminal device.

23. An electronic device, characterized in that: The electronic device comprises: one or more processors and a memory; The memory is coupled to the one or more processors, and the memory is used to store computer program codes, wherein the computer program codes include computer instructions, and the one or more processors call the computer instructions to enable the electronic device to perform the method according to any one of claims 1 to 22.

24. A chip system, characterized in that: The chip system is applied to an electronic device, and the chip system includes one or more processors, and the one or more processors are used to call computer instructions so that the electronic device executes the method described in any one of claims 1-22.

25. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes computer instructions, and when the computer instructions are executed on an electronic device, the electronic device executes the method according to any one of claims 1 to 22.

26. A computer program product, characterized in that The computer program product comprises a computer program code, and when the computer program code is run on an electronic device, the electronic device is caused to perform the method according to any one of claims 1 to 22.

Images