Image rendering method and terminal-cloud collaboration system

Abstract

The embodiments of the present application relate to the technical field of image processing. Provided are an image rendering method and a terminal-cloud collaboration system. The method comprises: a client controlling a server side to execute, according to computation processing requirements of the client for ray-tracing-related visual characteristics, a computation processing process for the corresponding visual characteristics, such that the client can obtain, from the server side, an intermediate rendering result of the visual characteristics which need to be rendered, so as to update a rendering effect of the visual characteristics in a rendered image of the client in real time by means of terminal-cloud collaboration, thereby improving the image rendering quality of the client.

Description

Image rendering method and end-cloud collaborative system

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on July 27, 2023, with application number 202310942831.X and application name “Image rendering method and end-cloud collaborative system”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The embodiments of the present application relate to the field of image processing technology, and in particular to an image rendering method and a terminal-cloud collaborative system. Background Art

[0003] In three-dimensional (3D) computer graphics, ray tracing (RT) is a rendering technique that simulates the propagation of light within a 3D scene and its interaction with the scene, thereby generating high-quality, realistic images. Common visual effects generated by ray tracing include direct lighting, indirect lighting (such as diffuse reflection), specular reflection, highlight reflection, shadows, scattering, and caustics. Images rendered using ray tracing can produce a high degree of visual realism, significantly enhancing image quality and thus improving the user experience.

[0004] However, the calculation of ray tracing technology requires a high computing power, and due to the limited computing power and memory capacity on mobile terminals (such as mobile phones and tablets), it is difficult to render images with ray tracing visual effects in real time.

[0005] Summary of the Invention

[0006] To solve the above technical problems, the present application provides an image rendering method and a device-cloud collaborative system. In this method, the client can control the server to perform the corresponding visual characteristic calculation and processing according to the client's ray tracing-related visual characteristic calculation and processing requirements, so that the client can obtain the intermediate rendering results of the required visual characteristics from the server, and update the rendering effect of the visual characteristics in the client's rendered image in real time in a device-cloud collaborative manner to improve the client's image rendering quality.

[0007] In one possible embodiment, the present application provides an end-cloud collaborative system. The end-cloud collaborative system includes at least one client and a server connected in communication, wherein the at least one client includes a first client; the first client is used to send first control information of a first image to the server; the server is used to control the calculation and processing process of m visual characteristics respectively according to the first control information, and obtain m first intermediate rendering results related to the m visual characteristics of the first image, m≥2; send the m first intermediate rendering results to the first client; the first client is also used to receive the m first intermediate rendering results, and obtain the first rendering result of the first image based on the m first intermediate rendering results.

[0008] The visual characteristics may include but are not limited to direct lighting, reflected lighting, shadows, etc.

[0009] The client can be deployed on a mobile terminal, and the server can be deployed on the cloud (such as a server).

[0010] The first control information, or the second control information described below, is the control information of the respective visual characteristics related to ray tracing.

[0011] Since the calculation of visual characteristics requires a high computing power, this application uses a server deployed on the cloud to perform the calculation of the intermediate rendering results of each visual characteristic, so that the calculation process that requires a high computing power is deployed on the cloud to achieve it, and uses a client deployed on the mobile terminal to perform the subsequent processing of the intermediate rendering results of each visual characteristic to complete the rendering of the corresponding visual characteristics, thereby deploying the processing process that requires lower computing power for the characteristics on the mobile terminal to achieve it. In this way, the calculation and processing process of the characteristics can be separated from the end-cloud to offload the high-load visual characteristic calculation process from the end-side to the cloud-side. In the case of low computing power on the end-side, real-time updates of various lighting characteristics in the image can be achieved by coordinating with the cloud-side.

[0012] In order to achieve the rendering of visual characteristics through end-cloud collaboration, the client synchronizes the control information of each visual characteristic to the server, thereby synchronizing the client's calculation and processing requirements for each characteristic to the server; the server can support the calculation and processing of multiple characteristics, and thus can control the calculation and processing process of the corresponding visual characteristics based on the control information synchronized by the client to obtain the intermediate rendering results of the corresponding visual characteristics (such as the illumination map of the direct lighting characteristic, etc.). In this way, the server can select the characteristics to be rendered and control the calculation and processing process of the characteristics according to the client's requirements for visual characteristics, so that the intermediate rendering results calculated and processed by the server can meet the client's characteristic rendering requirements. Although the server performs high-computing calculations on the characteristics, the client can control the server's characteristic calculation and processing process. And the client can render the corresponding visual characteristics based on the intermediate rendering results of the visual characteristics that meet its requirements issued by the server to obtain a rendered image with the corresponding visual characteristics. In this way, the client only needs to perform simple calculations on the received intermediate rendering results to achieve high-level image rendering effects, thereby improving the image drawing quality.

[0013] In a possible implementation, the number q of visual characteristics supported by the server for calculation and processing is greater than or equal to m, and the m visual characteristics are visual characteristics for which rendering is initiated in accordance with the first control information.

[0014] The number of visual features that the server supports computing is greater than or equal to the number of visual features that any client requests to start rendering, so that the server supports rendering of multiple features.

[0015] In one possible implementation, the server is configured with q calculation processing modules that perform calculation processing on q visual characteristics respectively, where q≥m; when the server controls the calculation processing processes of the m visual characteristics respectively according to the first control information, the server is specifically used to: call m calculation processing modules corresponding to the m visual characteristics among the q calculation processing modules, perform calculation processing on the m visual characteristics respectively according to the first control information, and obtain m first intermediate rendering results of the first image related to the m visual characteristics.

[0016] The server supports functional modules (also expressed as rendering components or calculation processing modules) with multiple features. By calling the functional modules, the server can combine the control information of the client to implement the calculation processing of the corresponding features.

[0017] In a possible embodiment, the at least one client also includes a second client; the second client is used to send second control information of the second image to the server; the server is further used to control the calculation and processing processes of n visual characteristics respectively according to the second control information, to obtain n second intermediate rendering results of the second image related to the n visual characteristics, q≥n≥1, wherein the n visual characteristics are the visual characteristics indicated by the second control information to start rendering; the n second intermediate rendering results are sent to the second client; the second client is further used to receive the n second intermediate rendering results, and obtain a second rendering result of the second image based on the n second intermediate rendering results.

[0018] The server can support multiple clients to simultaneously request to calculate and process characteristics of an image, and the sum of the types of characteristics requested to be calculated by the multiple clients is less than or equal to the q types of visual characteristics that the server supports calculation and processing.

[0019] The server can calculate and process the visual characteristics of each client's request to start rendering according to the client's respective control information, and refer to the respective control information to respectively control the calculation and processing of the required visual characteristics.

[0020] In a possible implementation, when the server controls the calculation and processing processes of n visual characteristics respectively according to the second control information to obtain n second intermediate rendering results of the second image related to the n visual characteristics, the server is specifically used to: call n calculation and processing modules corresponding to the n visual characteristics among the q calculation and processing modules, and perform calculation and processing on the n visual characteristics respectively according to the second control information to obtain n second intermediate rendering results of the second image related to the n visual characteristics.

[0021] The server supports functional modules (also expressed as rendering components or calculation processing modules) with multiple features. It can select and call the corresponding functional modules based on the features requested by the client, and use the corresponding functional modules in combination with the client's control information to implement the calculation processing of the corresponding features, and obtain the intermediate rendering results of the corresponding features that meet the client's requirements.

[0022] In a possible implementation, when the server controls the calculation and processing process of m visual characteristics respectively according to the first control information to obtain m first intermediate rendering results related to the m visual characteristics of the first image, the server is specifically used to: determine the m original intermediate results obtained by respectively calculating the m visual characteristics; and process the m original intermediate results respectively according to the first control information to obtain the m first intermediate rendering results of the m visual characteristics of the first image, wherein the data volume of the m first intermediate rendering results is less than the data volume of the m original intermediate results.

[0023] The “calculating the visual characteristics” described in various embodiments of the present application means that when the visual characteristic is direct lighting, it means calculating the direct lighting; when the visual characteristic is shadow, it means calculating the shadow.

[0024] The original intermediate result obtained by calculating a visual characteristic represents that the corresponding visual characteristic is calculated for the entire scene. For example, if the visual characteristic is direct lighting, the original intermediate result obtained by calculating the direct lighting is the intermediate rendering result that illuminates the entire game scene.

[0025] The control of the calculation process of visual characteristics in this application may include control of the calculation process of visual characteristics, as well as control of the extraction and / or encoding of the original intermediate results obtained by calculation (which can reduce the amount of data), thereby reducing the bandwidth pressure for transmitting the intermediate rendering results.

[0026] In one possible implementation, the first control information includes first indication information indicating an extraction strategy for the intermediate rendering results of each of the m visual characteristics. When the server processes the m original intermediate results separately based on the first control information to obtain the m first intermediate rendering results for the m visual characteristics of the first image, the server is specifically configured to: extract m groups of partial intermediate results from the m original intermediate results separately based on the first indication information to obtain the m first intermediate rendering results for the m visual characteristics.

[0027] The first indication information may be a resolution requirement for an intermediate rendering result and / or a viewing angle of a client, etc., which is not limited here.

[0028] The server can extract and process the original intermediate result based on the client's control information (such as viewing angle, resolution requirements for the intermediate rendering result, etc.) to obtain an intermediate rendering result with corresponding visual characteristics that meets the client's requirements.

[0029] In this way, regardless of whether there are one or multiple clients, the server can obtain the original intermediate results of the required features. Then, the server, combined with the client's control information (understood as the client's demand for feature resources), performs certain extraction processing on the original intermediate results to obtain the intermediate rendering results of the corresponding features that meet the client's perspective requirements. This can reduce the bandwidth requirements of the client's mobile terminal.

[0030] In a possible implementation, the first control information includes third indication information indicating an encoding strategy for the intermediate rendering results of each of the m visual characteristics. When the server processes the m original intermediate results separately according to the first control information to obtain the m first intermediate rendering results of the m visual characteristics of the first image, the server is specifically configured to: encode the m original intermediate results or the m groups of partial intermediate results extracted from the m original intermediate results separately according to the third indication information to obtain the m first intermediate rendering results of the m visual characteristics.

[0031] The third indication information may be a compression rate, an encoding mode, etc., which is not limited here.

[0032] In this way, regardless of whether there are one or more clients, the server can obtain the original intermediate results of the required rendering features. Then, the server combines the client's control information (understood as the client's demand for feature resources) to perform certain extraction processing and / or encoding processing on the original intermediate results to obtain a code stream of the intermediate rendering results of the corresponding features that meet the client's perspective requirements, which can reduce the bandwidth requirements of the mobile terminal on the client side.

[0033] In a possible implementation, when the server controls the calculation and processing process of n visual characteristics respectively according to the second control information to obtain n second intermediate rendering results of the second image related to the n visual characteristics, the server is specifically used to: determine n original intermediate results obtained by respectively calculating the n visual characteristics; and process the n original intermediate results respectively according to the second control information to obtain the n second intermediate rendering results of the n visual characteristics of the second image, wherein the data volume of the n second intermediate rendering results is less than the data volume of the n original intermediate results.

[0034] The control of the calculation process of visual characteristics in this application may include control of the calculation process of visual characteristics, as well as control of the extraction and / or encoding of the original intermediate results obtained by calculation (which can reduce the amount of data), thereby reducing the bandwidth pressure for transmitting the intermediate rendering results.

[0035] In one possible implementation, the second control information includes second indication information indicating an extraction strategy for the intermediate rendering results of each of the n visual characteristics. When the server processes the n original intermediate results separately according to the second control information to obtain the n second intermediate rendering results of the n visual characteristics of the second image, the server is specifically configured to: extract n groups of partial intermediate results from the n original intermediate results separately according to the second indication information to obtain the n second intermediate rendering results of the n visual characteristics.

[0036] The second indication information may be a resolution requirement for the intermediate rendering result and / or a viewing angle of the second client, etc., which is not limited here.

[0037] Furthermore, the original intermediate result is, for example, a calculation result of corresponding visual characteristics of a complete 360-degree viewing angle in a three-dimensional scene (also applicable to a two-dimensional scene).

[0038] Then, the server can extract and process the original intermediate result based on the client's control information (such as viewing angle, resolution requirements for intermediate rendering results, etc.) to obtain an intermediate rendering result with corresponding visual characteristics that meets the client's requirements.

[0039] In this way, regardless of whether there are one or multiple clients, the server can obtain the original intermediate results of the required features. Then, the server, combined with the client's control information (understood as the client's demand for feature resources), performs certain extraction processing on the original intermediate results to obtain the intermediate rendering results of the corresponding features that meet the client's perspective requirements. This can reduce the bandwidth requirements of the client's mobile terminal.

[0040] In one possible implementation, the second control information includes fourth indication information indicating an encoding strategy for the intermediate rendering results of each of the n visual characteristics. When the server processes the n original intermediate results separately according to the second control information to obtain the n second intermediate rendering results of the n visual characteristics of the second image, the server is specifically configured to: encode the n original intermediate results or the n groups of partial intermediate results extracted from the n original intermediate results separately according to the fourth indication information to obtain the n second intermediate rendering results of the n visual characteristics.

[0041] The third indication information may be a compression rate, an encoding mode, etc., which is not limited here.

[0042] In this way, regardless of whether there are one or more clients, the server can obtain the original intermediate results of the required rendering features. Then, the server combines the client's control information (understood as the client's demand for feature resources) to perform certain extraction processing and / or encoding processing on the original intermediate results to obtain a code stream of the intermediate rendering results of the corresponding features that meet the client's perspective requirements, which can reduce the bandwidth requirements of the mobile terminal on the client side.

[0043] In a possible implementation, when the server determines the m original intermediate results obtained by respectively calculating the m visual characteristics, and when determining the n original intermediate results obtained by respectively calculating the n visual characteristics, the server is specifically used to: determine k visual characteristics based on the first control information and the second control information, wherein the k visual characteristics are the union of the m visual characteristics and the n visual characteristics, m≤k, n≤k, and q≥k; calculate the k visual characteristics respectively (for example, call the calculation processing module of the k visual characteristics to perform calculation) to obtain k original intermediate results of each of the k visual characteristics; determine the m original intermediate results among the k original intermediate results that correspond to the m visual characteristics respectively; and determine the n original intermediate results among the k original intermediate results that correspond to the n visual characteristics respectively.

[0044] This eliminates the need for the server to repeatedly perform calculations for the same feature, allowing multiple clients to share the server's calculation results and computing resources. Furthermore, by processing (e.g., extracting and compressing) the raw intermediate results of the corresponding visual features calculated uniformly according to the control information of the corresponding client, the server obtains a lower data volume for the intermediate rendering results, thereby reducing the bandwidth requirements placed on the client when transmitting the code stream of the intermediate rendering results to the client.

[0045] In this embodiment, when there are many clients, the k visual characteristics may be determined first and calculated uniformly to obtain k original intermediate results, which improves processing efficiency.

[0046] In a possible implementation, when the server determines the m original intermediate results obtained by respectively calculating the m visual characteristics, the server is specifically used to: respectively calculate the m visual characteristics indicated by the first control information to start rendering, and obtain m original intermediate results of each of the m visual characteristics.

[0047] In this embodiment, the response processing of the control information may be performed on different clients in a certain order. For example, the characteristic of the first client is first calculated to obtain m original intermediate results.

[0048] In a possible implementation, when the server determines the n original intermediate results obtained by respectively calculating the n visual characteristics, the server is specifically used to: determine the p visual characteristics that are different from the m visual characteristics among the n visual characteristics indicated by the second control information to start rendering, 1≤p<n; calculate the p visual characteristics respectively to obtain p original intermediate results for each of the p visual characteristics; determine i original intermediate results corresponding to i visual characteristics respectively among the m original intermediate results; wherein the i visual characteristics are the same visual characteristics between the m visual characteristics and the n visual characteristics, i+p=n.

[0049] In this embodiment, control information responses can be processed for different clients in a specific order. For example, characteristics of the first client are first calculated to obtain m original intermediate results. Repeated characteristics between the first and second clients are then compared, and non-repeated characteristics are also calculated to obtain original intermediate results. For repeated characteristics, the corresponding i original intermediate results calculated for the first client can be reused, thus achieving computing resource sharing.

[0050] In a possible implementation, the first image and the second image are rendered images of the same scene from different perspectives.

[0051] The scene may be a two-dimensional scene or a three-dimensional scene.

[0052] For example, if the two-dimensional scene is an A4 paper and the visual characteristic is direct lighting, the first image may be an image viewed from the left half of the A4 paper with direct lighting, and the second image may be an image viewed from the right half of the A4 paper with direct lighting.

[0053] In one possible implementation, the visual characteristics include at least one of the following:

[0054] Direct lighting, bounce lighting, shadows, indirect lighting, spherical reflections, ambient occlusion.

[0055] In one possible implementation, the present application provides an image rendering method. The method includes: a server receiving first control information of a first image from a first client; the server controlling the calculation and processing of m visual characteristics based on the first control information to obtain m first intermediate rendering results related to the m visual characteristics of the first image, where m ≥ 2; and sending the m first intermediate rendering results to the first client; wherein the m first intermediate rendering results are used to provide the first client with rendering resources for rendering the first image.

[0056] The rendering resource is a rendering resource of the m visual characteristics, such as an intermediate rendering result.

[0057] In a possible implementation, the number q of visual characteristics supported by the server for calculation and processing is greater than or equal to m, and the m visual characteristics are visual characteristics for which rendering is initiated in accordance with the first control information.

[0058] In one possible implementation, the server is configured with q calculation processing modules that perform calculation processing on q visual characteristics respectively, where q≥m; the server controls the calculation processing processes of the m visual characteristics respectively based on the first control information, including: the server calls m calculation processing modules corresponding to the m visual characteristics among the q calculation processing modules, performs calculation processing on the m visual characteristics respectively based on the first control information, and obtains m first intermediate rendering results of the first image related to the m visual characteristics.

[0059] In a possible embodiment, the method further includes: the server receives second control information of the second image from the second client; the server controls the calculation and processing processes of n visual characteristics respectively according to the second control information, and obtains n second intermediate rendering results related to the n visual characteristics of the second image, q≥n≥1, wherein the n visual characteristics are the visual characteristics indicated by the second control information to start rendering; and sends the n second intermediate rendering results to the second client; wherein the n second intermediate rendering results are used to provide the second client with rendering resources for rendering the second image.

[0060] In a possible implementation, the server controls the calculation and processing processes of the n visual characteristics respectively according to the second control information to obtain n second intermediate rendering results of the second image related to the n visual characteristics, including: the server calls n calculation and processing modules corresponding to the n visual characteristics among the q calculation and processing modules, and performs calculation and processing on the n visual characteristics respectively according to the second control information to obtain n second intermediate rendering results of the second image related to the n visual characteristics.

[0061] In a possible implementation, the server controls the calculation and processing processes of the m visual characteristics respectively according to the first control information to obtain m first intermediate rendering results of the first image related to the m visual characteristics, including: the server determines the m original intermediate results obtained by respectively calculating the m visual characteristics; the server processes the m original intermediate results respectively according to the first control information to obtain the m first intermediate rendering results of the m visual characteristics of the first image, wherein the data volume of the m first intermediate rendering results is less than the data volume of the m original intermediate results.

[0062] In a possible implementation, the first control information includes first indication information indicating an extraction strategy for the intermediate rendering results of each of the m visual characteristics. The server processes the m original intermediate results respectively based on the first control information to obtain the m first intermediate rendering results of the m visual characteristics of the first image, including: the server extracts m groups of partial intermediate results from the m original intermediate results respectively based on the first indication information to obtain the m first intermediate rendering results of the m visual characteristics.

[0063] In a possible implementation, the first control information includes third indication information indicating an encoding strategy for the intermediate rendering results of each of the m visual characteristics, and the server processes the m original intermediate results separately based on the first control information to obtain the m first intermediate rendering results of the m visual characteristics of the first image, including: the server encodes the m original intermediate results or the m groups of partial intermediate results extracted from the m original intermediate results separately based on the third indication information to obtain the m first intermediate rendering results of the m visual characteristics.

[0064] In a possible implementation, the server controls the calculation and processing process of n visual characteristics respectively according to the second control information to obtain n second intermediate rendering results of the second image related to the n visual characteristics, including: the server determines n original intermediate results obtained by respectively calculating the n visual characteristics; the server processes the n original intermediate results respectively according to the second control information to obtain the n second intermediate rendering results of the n visual characteristics of the second image, wherein the data volume of the n second intermediate rendering results is less than the data volume of the n original intermediate results.

[0065] In a possible implementation, the second control information includes second indication information indicating an extraction strategy for the intermediate rendering results of each of the n visual characteristics. The server processes the n original intermediate results respectively based on the second control information to obtain the n second intermediate rendering results of the n visual characteristics of the second image, including: the server extracts n groups of partial intermediate results from the n original intermediate results respectively based on the second indication information to obtain the n second intermediate rendering results of the n visual characteristics.

[0066] In a possible implementation, the second control information includes fourth indication information indicating an encoding strategy for the intermediate rendering results of each of the n visual characteristics, and the server processes the n original intermediate results separately according to the second control information to obtain the n second intermediate rendering results of the n visual characteristics of the second image, including: the server encodes the n original intermediate results or the n groups of partial intermediate results extracted from the n original intermediate results separately according to the fourth indication information to obtain the n second intermediate rendering results of the n visual characteristics.

[0067] In a possible implementation, the server determines m original intermediate results obtained by respectively calculating the m visual characteristics, including: the server determines k visual characteristics based on the first control information and the second control information, wherein the k visual characteristics are the union of the m visual characteristics and the n visual characteristics, m≤k, n≤k, and q≥k; the server calculates the k visual characteristics respectively to obtain k original intermediate results of the k visual characteristics; the server determines m original intermediate results among the k original intermediate results that correspond to the m visual characteristics respectively.

[0068] In a possible implementation, the server determines n original intermediate results obtained by respectively calculating the n visual characteristics, including: the server determines n original intermediate results among the k original intermediate results that respectively correspond to the n visual characteristics.

[0069] In a possible implementation, the server determines the m original intermediate results obtained by respectively calculating the m visual characteristics, including: the server respectively calculates the m visual characteristics indicated by the first control information to start rendering, and obtains m original intermediate results of each of the m visual characteristics.

[0070] In a possible implementation, the server determines n original intermediate results obtained by respectively calculating the n visual characteristics, including: the server determines that, among the n visual characteristics for which the second control information indicates to start rendering, there are p visual characteristics that are different from the m visual characteristics, 1≤p<n; the server calculates the p visual characteristics respectively to obtain p original intermediate results for each of the p visual characteristics; the server determines i original intermediate results corresponding to i visual characteristics respectively among the m original intermediate results; wherein, the i visual characteristics are the same visual characteristics between the m visual characteristics and the n visual characteristics, i+p=n.

[0071] In a possible implementation, the first image and the second image are rendered images of the same scene from different perspectives.

[0072] In a possible implementation, the visual characteristics include at least one of the following: direct lighting, reflected lighting, shadow, indirect lighting, spherical reflection, and ambient occlusion.

[0073] The effects of the image rendering methods in the above embodiments are similar to the effects of the image rendering methods executed by the server in the above embodiments, and are not described in detail here.

[0074] In one possible implementation, the present application provides an image rendering method. The method includes: a client sending first control information of a first image to a server, wherein the first control information is used to provide the server with information for controlling the computational processing of each of m visual characteristics, where m ≥ 2; the client receiving m first intermediate rendering results from the server in response to the first control information; and the client obtaining a first rendering result of the first image based on the m first intermediate rendering results of the m visual characteristics.

[0075] In a possible implementation, the client sending the first control information of the first image to the server includes: the client sending the first control information of the first image to the server when determining that the scene is updated and / or the camera posture is updated.

[0076] In a possible embodiment, the client sends first control information of the first image to the server, including: the client sends control information of m visual characteristics indicating the start of rendering to the server; and / or, the client sends control information of an extraction strategy of the intermediate rendering results of each of the m visual characteristics to the server; and / or, the client sends control information of an encoding strategy of the intermediate rendering results of each of the m visual characteristics to the server.

[0077] In a possible implementation, before the client sends the first control information of the first image to the server, the method further includes: the client obtaining target environment information; wherein, the target environment information includes at least one of the client's network environment information, scene information of the first image, and user input information; and the client determines the first control information of the first image based on the target environment information.

[0078] In a possible implementation, the m visual characteristics include at least one of the following: direct lighting, reflected lighting, shadow, indirect lighting, spherical reflection, and ambient occlusion.

[0079] The effects of the image rendering methods in the above embodiments are similar to the effects of the image rendering methods executed by the client in the above embodiments, and are not described in detail here.

[0080] In one possible embodiment, the present application provides an image rendering method, which includes: a control module receives first control information of a first image from a first client; the control module outputs first information based on the first control information, and the first information is used to instruct the rendering module to start rendering of m visual characteristics; the control module receives m original intermediate results from the rendering module, wherein the m original intermediate results are respective characteristic calculation results obtained by the rendering module respectively calculating the m visual characteristics; the control module processes the m original intermediate results respectively based on the first control information to obtain m first intermediate rendering results of the m visual characteristics of the first image, wherein the data volume of the m first intermediate rendering results is less than the data volume of the m original intermediate results, and m≥2; wherein the m first intermediate rendering results are used to provide rendering resources for rendering the first image to the first client.

[0081] In a possible implementation, the number q of visual characteristics supported by the control module for calculation and processing is greater than or equal to m, and the m visual characteristics are visual characteristics for which rendering is initiated in accordance with the first control information.

[0082] In one possible embodiment, the control module is configured with q sub-control modules that process q visual characteristics respectively, where q≥m; the control module processes the m original intermediate results respectively according to the first control information to obtain m first intermediate rendering results of the m visual characteristics of the first image, including: m sub-control modules corresponding to the m visual characteristics among the q sub-control modules, processing the m original intermediate results respectively according to the first control information to obtain m first intermediate rendering results of the m visual characteristics of the first image.

[0083] In one possible implementation, the first control information includes first indication information indicating an extraction strategy for the intermediate rendering results of each of the m visual characteristics; the control module processes the m original intermediate results respectively based on the first control information to obtain m first intermediate rendering results for the m visual characteristics of the first image, including: the control module extracts m groups of partial intermediate results from the m original intermediate results respectively based on the first indication information to obtain the m first intermediate rendering results for the m visual characteristics.

[0084] In a possible implementation, the first control information includes third indication information indicating an encoding strategy for the intermediate rendering results of each of the m visual characteristics; the control module processes the m original intermediate results separately based on the first control information to obtain m first intermediate rendering results of the m visual characteristics of the first image, including: the control module encodes the m original intermediate results or the m groups of partial intermediate results extracted from the m original intermediate results separately based on the third indication information to obtain the m first intermediate rendering results of the m visual characteristics.

[0085] In a possible embodiment, the method further includes: the control module receives second control information of the second image from the second client; the control module outputs second information based on the second control information, and the second information is used to instruct the rendering module to start rendering of n visual characteristics; the control module receives n original intermediate results from the rendering module, wherein the n original intermediate results are respective characteristic calculation results obtained by the rendering module respectively calculating the n visual characteristics; the control module processes the n original intermediate results respectively based on the second control information to obtain n second intermediate rendering results of the n visual characteristics of the second image, wherein the data volume of the n second intermediate rendering results is less than the data volume of the n original intermediate results, q≥n≥1; wherein the n second intermediate rendering results are used to provide rendering resources for rendering the second image to the second client.

[0086] In one possible implementation, the second control information includes second indication information indicating an extraction strategy for the intermediate rendering results of each of the n visual characteristics; the control module processes the n original intermediate results respectively based on the second control information to obtain n second intermediate rendering results for the n visual characteristics of the second image, including: the control module extracts n groups of partial intermediate results from the n original intermediate results respectively based on the second indication information to obtain the n second intermediate rendering results for the n visual characteristics.

[0087] In a possible implementation, the second control information includes fourth indication information indicating an encoding strategy for the intermediate rendering results of each of the n visual characteristics, and the control module processes the n original intermediate results separately based on the second control information to obtain n second intermediate rendering results of the n visual characteristics of the second image, including: the control module encodes the n original intermediate results or the n groups of partial intermediate results extracted from the n original intermediate results separately based on the fourth indication information to obtain the n second intermediate rendering results of the n visual characteristics.

[0088] In a possible embodiment, the method further includes: the control module determines k visual characteristics based on the first control information and the second control information, wherein the k visual characteristics are the union of the m visual characteristics and the n visual characteristics, m≤k, n≤k, and q≥k; the control module outputs third information, and the third information is used to instruct the rendering module to start rendering for the k visual characteristics respectively; wherein the first information and the second information are merged into the third information; the control module receives k original intermediate results from the rendering module, and the k original intermediate results are the respective characteristic calculation results obtained by the rendering module by respectively calculating the k visual characteristics; wherein the m original intermediate results are the original intermediate results among the k original intermediate results that correspond to the m visual characteristics respectively; and the n original intermediate results are the original intermediate results among the k original intermediate results that correspond to the n visual characteristics respectively.

[0089] In a possible embodiment, the present application also provides an image rendering method, which includes: a rendering module receives first information from a control module, wherein the first information is used to instruct the rendering module to start rendering of m visual characteristics; the rendering module calculates the m visual characteristics respectively based on the first information to obtain m original intermediate results, where m≥2; and the rendering module sends the m original intermediate results to the control module.

[0090] In a possible implementation, the number q of visual features supported for rendering by the rendering module is greater than or equal to m.

[0091] In one possible implementation, the rendering module is configured with q sub-rendering modules that perform calculations on q visual characteristics respectively, where q ≥ m; the rendering module calculates the m visual characteristics respectively based on the first information to obtain m original intermediate results, including: m sub-rendering modules among the q sub-rendering modules corresponding to the m visual characteristics respectively calculate the m visual characteristics to obtain m original intermediate results of the first image related to the m visual characteristics.

[0092] In a possible embodiment, the method further includes: the rendering module receives second information from the control module, the second information is used to instruct the rendering module to start rendering of n visual characteristics; the rendering module calculates the n visual characteristics respectively according to the second information to obtain n original intermediate results, q≥n≥1; and the rendering module sends the n original intermediate results to the control module.

[0093] In a possible implementation, the rendering module calculates the n visual characteristics separately based on the second information to obtain n original intermediate results, including: n sub-rendering modules among the q sub-rendering modules corresponding to the n visual characteristics, respectively calculating the n visual characteristics to obtain n original intermediate results of the second image related to the n visual characteristics.

[0094] In a possible embodiment, the method further includes: the rendering module receives third information from the control module, and the third information is used to instruct the rendering module to start rendering for k visual characteristics respectively; wherein the k visual characteristics are the union of the m visual characteristics and the n visual characteristics, m≤k, n≤k, and q≥k; wherein the first information and the second information are merged into the third information; the rendering module calculates the k visual characteristics respectively to obtain k original intermediate results of each of the k visual characteristics, wherein the k visual characteristics are the union of the m visual characteristics and the n visual characteristics, m≤k, n≤k, and q≥k; wherein the m original intermediate results are the original intermediate results among the k original intermediate results that correspond to the m visual characteristics respectively; the n original intermediate results are the original intermediate results among the k original intermediate results that correspond to the n visual characteristics respectively; and the rendering module sends the k original intermediate results to the control module.

[0095] In one possible embodiment, the present application provides an image rendering device. The device includes: a receiving module for receiving first control information of a first image; a control module for controlling the calculation and processing of m visual characteristics based on the first control information, respectively, to obtain m first intermediate rendering results associated with the m visual characteristics of the first image, where m ≥ 2; and sending the m first intermediate rendering results to a first client; wherein the m first intermediate rendering results are used to provide the first client with rendering resources for rendering the first image.

[0096] The image rendering device may be the aforementioned server, and the specific functional modules in the device may be modules on the server that implement the aforementioned method.

[0097] The effects of the image rendering apparatus in each of the above embodiments are similar to the effects of the image rendering method executed by the server in each of the above embodiments, and will not be described in detail here.

[0098] In one possible embodiment, the present application provides an image rendering device. The device includes: a sending module for sending first control information of a first image to a server, wherein the first control information is used to provide the server with information for controlling the computational processing of each of m visual characteristics, where m ≥ 2; a receiving module for receiving, from the server, m first intermediate rendering results in response to the first control information; and a processing module for obtaining a first rendering result of the first image based on the m first intermediate rendering results of the m visual characteristics.

[0099] The image rendering device may be the above-mentioned client, and the specific functional modules in the device may be modules of the client that implement the above-mentioned method.

[0100] The effects of the image rendering apparatus in each of the above embodiments are similar to the effects of the image rendering method executed by the client in each of the above embodiments, and are not described in detail here.

[0101] In one possible implementation, the present application provides an image rendering device. The image rendering device includes one or more interface circuits and one or more processors; the interface circuits are configured to receive signals from a memory and send the signals to the processors, the signals including computer instructions stored in the memory; when the processors execute the computer instructions, the processors can implement the method executed by the server or client in any of the aforementioned implementations.

[0102] The effects of the image rendering device of this embodiment are similar to the effects of the image rendering methods of the above embodiments, and are not described in detail here.

[0103] In one possible implementation, the present application provides a computer-readable storage medium storing a computer program that, when executed on a computer or processor, causes the computer or processor to execute the method executed by the server or client in any of the above implementations.

[0104] The effect of the computer-readable storage medium of this embodiment is similar to the effect of the image rendering method of the above embodiments, and will not be repeated here.

[0105] In one possible implementation, the present application provides a computer program product. The computer program product includes a software program, which, when executed by a computer or a processor, enables the client-side method or the server-side method in any of the above implementations to be executed.

[0106] The effects of the computer program product of this embodiment are similar to the effects of the image rendering methods of the above embodiments, and will not be described in detail here. BRIEF DESCRIPTION OF THE DRAWINGS

[0107] FIG1 is a schematic diagram illustrating an exemplary system architecture;

[0108] FIG2 is a schematic diagram illustrating an exemplary interaction process;

[0109] FIG3 is a schematic diagram illustrating an exemplary interaction process;

[0110] FIG4 a is a schematic diagram illustrating an exemplary processing process;

[0111] FIG4 b is a schematic diagram illustrating an exemplary processing process;

[0112] FIG5 is a schematic diagram illustrating an exemplary interaction process;

[0113] FIG6 a is a flowchart illustrating an exemplary image rendering;

[0114] FIG6 b is a flowchart illustrating an exemplary image rendering;

[0115] FIG7 is a schematic diagram illustrating an exemplary interaction process;

[0116] FIG8 a is a schematic diagram illustrating an exemplary processing process of a server;

[0117] FIG8 b is a schematic diagram illustrating an exemplary processing process of the server;

[0118] FIG8c is a schematic diagram illustrating an exemplary processing process of the server;

[0119] FIG8 d is a schematic diagram illustrating an exemplary processing process of the server;

[0120] FIG8e is a schematic diagram illustrating an exemplary processing process of the server;

[0121] FIG9a is a schematic diagram illustrating an exemplary processing procedure of a client;

[0122] FIG9 b is a schematic diagram illustrating an exemplary processing procedure of a client;

[0123] FIG9c is a schematic diagram illustrating an exemplary processing process of a client;

[0124] FIG10 is a schematic diagram illustrating an exemplary interaction process;

[0125] FIG11 is a schematic diagram illustrating an exemplary development process;

[0126] FIG12 is a schematic structural diagram of a device provided in an embodiment of the present application;

[0127] FIG13 is a schematic diagram of the structure of a chip provided in an embodiment of the present application. DETAILED DESCRIPTION

[0128] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0129] The term "and / or" in this article is merely a description of the association relationship between associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0130] In the description and claims of the embodiments of this application, the terms "first" and "second" are used to distinguish different objects, rather than to describe a specific order of objects. For example, the terms "first target object" and "second target object" are used to distinguish different objects, rather than to describe a specific order of objects.

[0131] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0132] In the description of the embodiments of this application, unless otherwise specified, "multiple" means two or more. For example, "multiple processing units" means two or more processing units; "multiple systems" means two or more systems.

[0133] In three-dimensional (3D) computer graphics, ray tracing (RT) is a rendering technique used to generate images by tracing the path of light on a pixel-by-pixel basis in an image plane and simulating the effects of its encounters with virtual objects. Ray tracing generates high-quality, realistic images by simulating the propagation of light in a 3D scene and its interaction with the 3D scene. Common visual effects calculated by ray tracing include direct lighting (Direct Light), indirect lighting (such as diffuse reflection), specular reflection, highlight reflection, shadows, scattering, caustics, etc. Ray tracing technology is mainly used in fields such as gaming and virtual reality. Images rendered using ray tracing technology can produce a high degree of visual realism and can significantly enhance the quality of the picture, thereby improving the user experience.

[0134] However, ray tracing requires significant computing power. Currently, ray tracing visuals are primarily displayed on personal computers (PCs) with high computing power. However, mobile devices (such as phones and tablets) have limited computing power and memory capacity, making it difficult to render images with ray tracing visual effects in real time.

[0135] In order to get rid of the limitations of the hardware capabilities of mobile terminals and realize PC-level ray tracing image rendering effects on the mobile terminal side, the present application provides a method, device and system. The method realizes real-time rendering of the visual effects (such as direct lighting, plane reflection, shadow, etc.) of ray tracing (abbreviated as "ray tracing") on the mobile terminal side through collaboration between the cloud and the mobile terminal, and can enhance the real-time rendering performance of the mobile terminal for the picture and the rendering quality of the picture, thereby improving the user experience.

[0136] The method and system of this application can be applied to various scenarios that require the display of images with ray tracing effects, such as gaming scenarios and virtual reality scenarios, but this embodiment of the application does not limit this.

[0137] The following uses a game scenario as an example to illustrate the technical solution of this application. When this application is applied to other scenarios, the implementation principle of the solution is similar and will not be repeated here.

[0138] The system of the present application may include a client 100 and a server 200, wherein the client 100 may be a game instance (e.g., a game executable program) installed and running on a mobile terminal (referred to as the "terminal side"), and the server 200 may be a game instance (e.g., a game executable program) installed and running on the cloud (referred to as the "cloud side") for use with the client.

[0139] The cloud can be a server or a virtual instance (such as a virtual machine or container) running on a server, and is not limited here. For example, the cloud is a computing server.

[0140] In some embodiments, the client 100 and the server 200 may each be implemented in hardware, software, or a combination of hardware and software including one or more signal processing and / or application specific integrated circuits.

[0141] The implementation process of the method of the present application may include a development state process and an operation state process, wherein, in the development state, the device 300 of the present application may be used to expand the functionality of an existing game engine to develop a game instance, such as the above-mentioned client 100 and server 200.

[0142] In the running state, terminal-cloud collaboration can be performed by running the client 100 on the mobile terminal and the server 200 on the cloud, so as to render the visual effect of ray tracing in real time on the mobile terminal.

[0143] The visual effects of ray tracing of the present application are described below as "characteristics" or "visual characteristics".

[0144] Visual characteristics can be rendering characteristics created by simulating the actual bouncing of light in a scene based on a ray tracing algorithm. Therefore, these characteristics may include, but are not limited to: direct lighting, indirect lighting, planar reflections, curved surface reflections, soft shadows, hard shadows, ambient occlusion, transparent refraction, subsurface scattering, etc.

[0145] FIG1 is a schematic diagram illustrating an exemplary system architecture of the present application.

[0146] When developing a game instance (eg, client 100 and server 200 ) based on an existing game engine, the present application provides an apparatus 300 as shown in FIG1 .

[0147] As shown in Figure 1, in the development state, the engine layer can be the engine system of an existing game engine, and the device 300 of the present application can include a function layer and an interface layer. The device 300 can be built on top of the original module (such as the engine layer) of the existing game engine.

[0148] In a specific implementation, the device 300 composed of the functional layer and the interface layer can be used as a plug-in for an existing game engine, so that game developers can use the device 300 to expand the functions of the existing game engine, thereby obtaining a game instance that can be installed on a mobile terminal (such as the above-mentioned client 100) and a game instance that can be installed in the cloud (such as the server 200).

[0149] The internal module structure of the developed client 100 and server 200 in operation can be referred to FIG3 , which will be described in detail later.

[0150] As shown in FIG1 , the engine layer may include but is not limited to: a rendering pipeline module, a game logic module, a camera module, etc.

[0151] In addition, in the development state, the engine layer can also provide an application programming interface (API) for the upper layer (such as the interface layer of this application) to call, for example, to expand the functions of the game engine; in addition, in the development state, the engine layer also provides a graphical user interface (GUI) for the entire system, presenting the interactive interface of each module in the system shown in Figure 1 in a user-friendly manner.

[0152] The specific module structure within the engine layer can refer to the internal structure of the engine layer in existing game engines or future game engines, and is not limited here.

[0153] It should be understood that the API and GUI in the engine layer shown in Figure 1 can be used to develop the client 100 and server 200, while the packaged and released client 100 and server 200 no longer include the above API and GUI used in the above development state.

[0154] The interface layer may include the collaboration module of the present application.

[0155] The functional layers of this application may include but are not limited to: rendering module, encoding module, decoding module, transmission module, etc.

[0156] Among them, the collaborative module can provide an interface (referred to as "plug-in interface") of the above-mentioned plug-in (here is device 300) of this application to realize the interaction between the functional layer and the engine layer, and to realize the interaction between the modules within the functional layer.

[0157] In addition, the collaborative module can also provide collaborative functions on the end side (such as end-cloud synchronization, collaborative scheduling, etc.), as well as collaborative functions on the cloud side (such as end-cloud synchronization, collaborative scheduling, etc.). The collaborative functions of the collaborative module on the end side and the cloud side will be elaborated in detail in the running state embodiments and will not be repeated here.

[0158] Figure 1 is a schematic diagram illustrating an exemplary system framework. It should be understood that the system shown in Figure 1 is merely an example, and the system of the present application may have more or fewer components than shown, may combine two or more components, or may have a different component configuration. The various components shown in Figure 1 may be implemented in hardware, including one or more signal processing and / or application-specific integrated circuits, software, or a combination of hardware and software.

[0159] FIG2 exemplarily shows a schematic diagram of the interaction process between the client 100 deployed on the mobile terminal and the server 200 deployed on the cloud.

[0160] In combination with the system architecture shown in Figure 1, Figure 3 exemplarily shows the interaction process between the internal modules of the client 100 shown in Figure 2, the interaction process between the internal modules of the server 200, and the interaction process between the client 100 and the server 200.

[0161] As shown in Figure 3, the client 100 developed using the device 300 shown in Figure 1 of the present application may include but is not limited to: a rendering module 101a, a decoding module 102a, a collaboration module 103a, a transmission module 104a, a rendering pipeline module 105a, and a game logic module 106a.

[0162] As shown in FIG3 , the server 200 developed using the apparatus 300 shown in FIG1 of the present application may include but is not limited to: a rendering module 101b, an encoding module 102b, a collaboration module 103b, a transmission module 104b, a rendering pipeline module 105b, and a game logic module 106b.

[0163] FIG3 is a schematic diagram showing the structure of the client and server. It should be understood that the structure of the client 100 and server 200 shown in FIG3 is only an example, and the various modules in the client 100 can be split, merged, and include more or fewer modules, or can have different module configurations. The various modules in the server 200 can be split, merged, and include more or fewer modules, or can have different module configurations. The various modules in the client 100 shown in FIG3 can be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application-specific integrated circuits. And the various modules in the server 200 shown in FIG3 can be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application-specific integrated circuits.

[0164] The following describes the interaction process between the client 100 and the server 200 shown in FIG. 2 in conjunction with FIG. 3 , FIG. 4 a , FIG. 4 b and FIG. 5 .

[0165] 5 exemplarily shows the interaction process between multiple clients (eg, client 100 and client 101 ) and a server 200 .

[0166] As shown in Figure 2, the process may include the following steps:

[0167] S101, the client 100 detects a change in the game scene.

[0168] This embodiment takes application in a game as an example, and therefore the rendering scene here is a game scene.

[0169] In some embodiments, the rendering scene may be a three-dimensional scene or a two-dimensional scene, which is not limited here.

[0170] S101 is described below with reference to FIG. 3 .

[0171] As shown in FIG3 , the client 100 can run on a mobile terminal. The game logic module 106a in the client 100 can provide complete game logic. The game logic module 106a can receive user operations on the game and execute the game logic in response to the user operations.

[0172] During the process of running a frame of game screen, the client 100 may execute game logic (such as the movement of virtual characters in the game, etc., which is not limited here) to cause the game state to change.

[0173] The collaboration module 103a can monitor changes in the game status from the game logic module 106a. For example, changes in character animation in the game scene, changes in the status of light sources in the game scene, changes in the status of objects in the game scene, scene changes, etc. are all changes in the game status (not limited here). In this way, the collaboration module 103a can obtain the changed game status from the game logic module 106a.

[0174] Changes in the game state may cause changes in the game scene. Therefore, the client side needs to render the changed game scene to display the updated game screen due to the user's operation on the game.

[0175] The embodiments of Figures 2 to 6 of the present application are explained by taking the case where the game state changes but the camera posture does not change as an example.

[0176] As shown in FIG3 , when the game scene changes, the game logic module 106a may also send the camera posture to the collaboration module 103a for synchronization to the cloud, and send the camera posture to the rendering pipeline module 105a for rendering the game screen.

[0177] It should be understood that any change in information such as the change in the game scene and the change in the camera posture can trigger the client 100 to control the server 200 to perform calculations and processing of the characteristics required by the client 100 according to the control information of the client 100, and control the calculation and processing process.

[0178] In a possible implementation, as shown in FIG3 , the game logic module 106 a may also monitor changes in the camera posture of the client 100 , and synchronize the changed camera posture to the collaboration module 103 a .

[0179] Among them, the rendering pipeline module 105a of the client 100 can obtain the game scene and camera posture that need to be rendered due to changes in the game state from the game logic module 106a, so as to utilize the basic rendering capabilities configured in the development state and combine the camera posture to render the game scene.

[0180] S102 , the client 100 may determine control information of various characteristics.

[0181] Among them, the multiple characteristics may be part or all of the characteristics supporting rendering configured for the rendering module 101a in the development state, such as direct lighting, planar reflection, soft shadow and other characteristics. The specific characteristics can be flexibly determined according to the control strategy configured by the client 100 (for example, as shown in Figure 4a).

[0182] The control information is image control information, which can be used to control the calculation and processing process of the server 200 on the above-mentioned multiple characteristics.

[0183] In a possible implementation, when the client 100 detects a change in the game scene, the client 100 may obtain a frame state of the current frame (ie, the above-mentioned image), which is an example of control information.

[0184] As shown in Figures 3 and 4a, the collaboration module 103a can obtain the changed game state from the game logic module 106a, such as the changed character animation, scene changes (such as the position of objects in the scene, changes in light sources, etc.), camera posture, etc.

[0185] In this embodiment, the game state packaged in the frame state obtained by the client 100 may be the game state after the change. In other embodiments, it may also be the game state before the change and the game state after the change (for example, the position before the character animation moves and the position after the movement).

[0186] In addition, as shown in FIG4a , the collaboration module 103a can also obtain frame information of the current frame to be rendered (such as information on which frame the game is currently running to, etc.) and terminal information (such as information that can identify a mobile terminal or a player account).

[0187] As shown in Figures 3 and 4a, the collaborative module 103a can also obtain the current network status information of the mobile terminal to which the client 100 belongs from the transmission module 104a, where the network status information can be any parameter that can indicate the current network quality, such as packet loss rate, loop delay, etc., which is not limited here.

[0188] The transmission module 104a of the client 100 may determine the current network status information according to the transmission status of the code stream, and synchronize the information to the coordination module 103a.

[0189] In a possible implementation, as shown in FIG4 a , the collaboration module 103 a may also obtain scene information of the game, which may be information about scene changes in the aforementioned game state, that is, scene information of the changed game scene.

[0190] For example, if a character in the game walks into a scene with many reflections, the reflection effect of this scene needs to be prioritized. In specific implementation, the game developer can set certain logic, such as setting a certain boundary for such a scene. When the character in the game enters the area defined by the boundary, the collaborative module 103a is triggered to obtain the game scene information.

[0191] As shown in Figure 4a, in the development state, game developers can configure a feature control policy for the collaboration module 103a. These features are the various features supported by the rendering module and game pipeline module in the client 100 during the development state. These various features supported by the client 100 can be rendered in collaboration with the server 200, and are therefore referred to as "collaborative rendering features" below.

[0192] The control policy can control the state (on or off) of each of the above-mentioned multiple features supported by the client 100, as well as the state of the feature resource.

[0193] Among them, although the client 100 after development can support multiple features, the collaboration module 103a can flexibly determine which features to enable or disable, and determine the status of the feature resources of the enabled features according to the current scenario information and the current network status.

[0194] The characteristic resource is also called an intermediate rendering result (to be rendered by the server 200 ), and the status of the characteristic resource may include but is not limited to: at least one of the resolution and compression rate of the intermediate rendering result.

[0195] In one possible implementation, the user of the client 100 may also manually input the configuration of the control strategy during operation, such as keeping a certain feature in the game always on, or keeping a certain feature in the off state, etc. There is no specific limitation, and the switch of the feature and the status of the feature resources of the turned-on feature can be flexibly set according to user needs.

[0196] As shown in Figure 4a, the collaborative module 103a can refer to the scene information and the current network status based on the control strategy to determine whether each feature of the current frame is turned on or off, as well as the status of the feature resources of the turned-on feature, so as to control the cloud server 200's calculation and processing process of the feature.

[0197] For example, when the client 100 is disconnected from the network, the client 100 can control to turn off all collaborative rendering features (such as some features with ray tracing visual effects), so that multiple features are in a closed state. In this way, only the end-side rendering pipeline module 105a can be used to render the current frame to ensure the basic game experience, without the need to obtain the intermediate rendering results of the above-mentioned multiple features of the current frame from the server 200 deployed in the cloud.

[0198] For example, when the network bandwidth of the client 100 is insufficient to support high-quality rendering of the corresponding feature, the collaborative module 103a can adjust and control the quality (such as resolution, compression rate, etc.) of the intermediate rendering results of the corresponding feature transmitted from the cloud to the mobile terminal.

[0199] For example, when the enabled features include reflected lighting, the collaborative module 103a can control the reduction of the resolution of the reflection map; when the enabled features include or shadows, the collaborative module 103a can control the reduction of the resolution of the shadow map or increase the compression rate of the shadow layer; when the enabled features include direct lighting, the collaborative module 103a can control the reduction of the resolution of the probe that reduces direct lighting, etc.

[0200] For example, when the network bandwidth of the client 100 is insufficient to guarantee the rendering of all collaborative rendering features, the collaborative module 103a may choose to turn off certain lower-priority features based on the scene information shown in FIG4a (eg, the scene information of the changed game scene to be rendered).

[0201] For example, in a water surface reflection scene (scene information of a game scene), the priority of the reflected lighting characteristics of the game scene is higher than the priority of the shadow characteristics. Then the collaborative module 103a can choose to turn off the shadow characteristics and turn on the reflected lighting characteristics in combination with the scene information to ensure that the current frame rendered in the cloud has characteristics that match the game scene, and ensure that the lighting characteristics in the game screen match the lighting scene of the game screen.

[0202] Continuing with reference to Figures 3 and 4a, the collaboration module 103a can package the frame information, terminal information, game status (such as character animation, scene changes, etc.), camera posture, feature status (equivalent to the requirement to turn each feature on or off), and feature resource status (equivalent to the encoding requirements for turned-on features, such as compression rate, resolution, etc.) into the frame status of the current frame and send it to the server 200 deployed in the cloud through the transmission module 104a.

[0203] In an embodiment of the present application, when the collaborative module 103a of the client 100 monitors changes in the game scene, it can dynamically adjust the feature state of the current frame to be rendered (for example, whether the feature is turned on or off), and the encoding requirements of the turned-on feature when the server 200 in the cloud is rendered according to at least one piece of information such as user manual control or the current network status or the current game scene. In this way, not only are the rendering of the features on the end side and the cloud side independent of each other, but they also have their own independent rendering processes and calculation processes. In addition, by carrying control information for the features to be rendered on the cloud side in the frame state, the client 100 can allow the end side to independently control the calculation and processing process of each feature on the cloud side in real time according to its own network environment and other operating environment (optionally, the computing power environment of the end side, the computing power environment of the cloud side, etc.), so as to achieve synchronization and control of the rendering state of the features rendered on the cloud side (whether the features are turned on or off), the extraction strategy of the feature resources (for example, strategies related to perspective information such as camera posture, and / or, resolution requirements for feature resources, etc.), and the encoding strategy of the feature resources (compression rate of feature resources, etc.).

[0204] In one possible implementation, the collaborative module in the device 300 can provide a scripting interface during development, allowing game developers to customize the code logic for acquiring feature control information for the collaborative module 103a in the client 100. Thus, the developed collaborative module 103a can follow the processes shown in Figures 3 and 4a to acquire various status information and package it into a frame state for upload to the cloud when the game scene is updated.

[0205] In the above embodiment, the control information in S102 in FIG. 2 is taken as an example to illustrate the frame state.

[0206] In some embodiments, the control information in S102 may also be at least one of the camera posture, feature status, and feature resource status shown in FIG4a , so as to control the rendering process of the server 200 when rendering the corresponding feature requested by the client 100. Specifically, the feature status may be used to control which features the server 200 renders. Optionally, the client 100 may control the server 200 through the camera posture to render an intermediate rendering result of the corresponding feature that conforms to the user perspective of the client 100. Optionally, the client 100 may control the resolution of the intermediate rendering result of the feature rendered by the server 200 and the compression rate when encoding the intermediate rendering result through the status of the feature resource.

[0207] In other embodiments, the control strategy shown in FIG4a can also be pre-configured or configured in the development state within the collaborative module 103b of the server 200 shown in FIG3. In this way, the control information sent by the client 100 to the server 200 may include, but is not limited to, the camera posture shown in FIG4a, the scene information of the game scene, the network status information, etc. The server 200 with higher computing power utilizes the scene information of the game scene and the network status information according to the pre-configured control strategy to obtain the characteristic state and the state of the characteristic resource when the client 100 renders the current frame. The specific implementation principle is similar to the principle of using the control strategy on the client 100 side to obtain the characteristic state and the state of the characteristic resource, and will not be repeated here. In this way, if the computing power of the mobile terminal deployed by the client 100 is relatively low, the cloud with higher computing power can also combine the network environment of the mobile terminal, the changed game scene and the perspective of the game user (indicated by the camera posture here, or by other information that can indicate the user's perspective) to control the calculation process of the characteristics of the current frame of the client 100.

[0208] S103 , the client 100 sends the control information to the server 200 .

[0209] 3 , the coordination module 103 a may send the packaged frame status to the server 200 via the transmission module 104 a .

[0210] In one possible implementation, multiple users may play games in a game lobby. Figure 5 illustrates the interaction between the respective clients of two users (client 100 and client 101) and server 200 in the game lobby. Clients 100 and 101 play the same game in the same game lobby, and at the same moment, the game screens displayed by them share a large portion of the same content. In other words, the game screens displayed by clients 100 and 101 match the perspectives of the game users, presenting different perspectives of the same game scene.

[0211] It should be understood that this application does not limit the number of clients that can interact with the server 200 at the same time.

[0212] The principles of the client 100 and the client 101 in executing the above-mentioned process from S101 to S103 are similar. The following describes the specific process of the client 100 executing the above-mentioned process from S101 to S103 with reference to FIG5 , taking the client 100 as an example.

[0213] As shown in Figure 5, the client 100 can be configured with feature components of three characteristics in the development state, namely feature component 301, feature component 302 and feature component 303. For example, feature component 301 is used to implement the basic rendering process of direct lighting characteristics, feature component 302 implements the basic rendering process of reflected lighting characteristics, and feature component 303 is used to implement the basic rendering process of shadow characteristics.

[0214] Similarly, as shown in FIG5 , the server 200 may be configured with feature components having three features in the development state, namely, feature component 201, feature component 202, and feature component 203. For example, feature component 201, feature component 202, and feature component 203 are respectively used to implement the calculation and processing processes of the three features supported by the cloud side, namely, direct lighting, reflected lighting, and shadow.

[0215] In this way, the client 100 can support end-cloud collaborative rendering of three characteristics: direct lighting, reflected lighting, and shadows.

[0216] Exemplarily, in combination with FIG3 , the feature component of each feature in the server 200 shown in FIG5 may be a combination of the rendering module 101b , the encoding module 102b , and the collaboration module 103b of the corresponding feature shown in FIG3 .

[0217] Similarly, in combination with FIG3 , the feature component of each feature in the client 100 shown in FIG5 may be a combination of the rendering module 101a , the decoding module 102a , and the collaboration module 103a of the corresponding feature shown in FIG3 .

[0218] In this way, the calculation and rendering processes for each feature are independent of each other, whether on the client side or the cloud side.

[0219] As shown in Figure 5, the client 100 can execute game logic to cause the game state to change. When the client 100 runs a frame image (game screen), it can extract the changed game state (such as character animation, etc.), camera posture (whether it changes or not, it can be extracted), and obtain the characteristic state of the frame game screen (the above three characteristics are respectively in the on or off state), the state of the characteristic resource of each turned-on characteristic (such as the compression rate and resolution of the intermediate rendering results, etc.), and package this information to obtain the frame state of the current frame (the above frame image), and transmit it to the server 200 through the network.

[0220] The execution principle is the same as that of the client 100 , and the client 101 also transmits the frame status of the current frame it is running to the server 200 via the network.

[0221] In an embodiment of the present application, the frame state of the client can be synchronized to the server on the cloud side, and the feature rendering processes on the client side and the cloud side are independent of each other, and have their own independent rendering and calculation processes; by additionally including control information for the features calculated and processed by the cloud side in the frame state, the client side can be allowed to independently configure the various features on the cloud side in real time according to the operating environment (such as the network status of the client side, the computing power of the client side, the computing power of the cloud side, etc.), so as to dynamically adjust the rendering status of the cloud side features (such as whether the features are turned on or off), the extraction strategy of the feature resources (such as based on the viewing angle or network status, the resolution requirements of the feature resources, etc.) and the encoding and transmission quality of the feature resources (such as based on the compression rate requirements, etc.).

[0222] S104: The server 200 controls the calculation and processing processes of the multiple characteristics respectively according to the control information, and obtains an intermediate rendering result of each characteristic of the current frame.

[0223] Taking the control information in S104 as the frame status as an example, and continuing to refer to Figure 3, the server 200 can receive the frame status of the current frame uploaded by each client through the transmission module 104b. As shown in Figure 5, the server 200 can receive the frame status uploaded by the client 100 and the client 101 respectively.

[0224] In some embodiments, as shown in FIG3 , the transmission module 104 b may send the frame status received from each client to the coordination module 103 b .

[0225] FIG4 b exemplarily shows the process of the cooperation module 103 b processing the received frame status of the client 100 and the frame status of the client 101 .

[0226] As shown in FIG4b , the collaboration module 103b can parse the received frame states of each client to obtain the game states such as scene changes and character animations of client 100, and the game states such as scene changes and character animations of client 101. The collaboration module 103b can then summarize the changes in the game states of all clients (here, client 100 and client 101) to determine the updated game scene.

[0227] For example, if game character A on client 100 moves one step and game character B on client 101 moves two steps, the game scene can be updated by summarizing the changes in the game states of all clients. Here, collaboration module 103b can determine the updated game scene by summarizing the changes in the game states of all clients. Because the game scene has been updated, clients 100, 101, and server 100 will render the corresponding features of the game scene.

[0228] In some embodiments, the collaborative module 103b of the server 200 can parse the frame state of each client received to determine the scene information of the game scene (such as character animation) and the camera pose information carried in the frame state. The server 200 can cache the intermediate rendering results (full perspective) of various features of the previous frame, the scene information, and the camera pose information. After receiving the frame state of the current frame, the server 200 can compare the camera pose information corresponding to the previous frame with the camera pose information corresponding to the current frame to determine whether the camera pose has been updated, and can compare the scene information of the game scene corresponding to the previous frame with the scene information of the game scene corresponding to the current frame to determine whether the game scene has been updated.

[0229] In addition, as shown in FIG4 b , the collaboration module 103 b can also obtain the characteristic status uploaded by each client by parsing the frame status of each client, which are the characteristic status of client 100 and the characteristic status of client 101 .

[0230] In conjunction with the description of the feature components of client 100 and client 101 shown in Figure 5 , both client 100 and client 101 support rendering of three features: direct lighting, reflected lighting, and shadows. Therefore, when a game scene is updated, client 100 can control the rendering of these three features for that scene. In this way, the feature status of these three features uploaded by client 100 will all be enabled.

[0231] Similarly, when the game scene is updated, the client 101 can control the two features of direct lighting and reflected lighting for the game scene, and turn off the rendering of the shadow feature. In this way, the characteristic states of the above three features uploaded by the client 101 are: the direct lighting feature is on, the reflected lighting feature is on, and the shadow feature is off.

[0232] In some embodiments, for features that the client does not need to request rendering this time, the status of the feature may not be uploaded to the server. For example, the client 101 may only upload the feature of direct lighting as being on and the feature of reflected lighting as being on. Then the server 200 may default to the client 101's rendering requirement for the shadow feature as not to render.

[0233] Optionally, as shown in FIG4 b , the collaboration module 103 b of the server 200 may aggregate the characteristic states of all clients to obtain the characteristic state of each characteristic.

[0234] Among them, when the collaborative module 103b summarizes the feature status, for a feature, as long as there is a client reporting that the feature status is on, the summarized feature status is on; for a feature, when all clients report that the feature status is off, the summarized feature status is off.

[0235] Then, in this embodiment, the summarized characteristic states include the direct illumination characteristic being in an on state, the reflected illumination characteristic being in an on state, and the shadow characteristic being in an on state.

[0236] In addition, as shown in Figure 4b, the collaborative module 103b can also obtain the status of the characteristic resources and camera posture of client 100 (an example of extracting control information) and the status of the characteristic resources and camera posture of client 101 by parsing the frame status of each client.

[0237] Among them, the status of the feature resources uploaded by the client 100 reflects the rendering requirements (such as the resolution of the intermediate rendering results) and / or encoding requirements (such as the compression rate of the intermediate rendering results) of the client 100 for each of the above three features in the enabled state.

[0238] The status of the feature resource uploaded by the client 101 reflects the client 101's extraction requirements (such as the resolution of the intermediate rendering results) and / or encoding requirements (such as the compression rate of the intermediate rendering results) for the intermediate rendering results of each of the two features (direct lighting feature and reflected lighting feature) that are in the enabled state.

[0239] And the camera posture uploaded by the client can reflect the perspective of the game screen viewed by the client, so when the server 200 pushes the rendered intermediate rendering results to the client, it only needs to push the intermediate rendering results that match the perspective.

[0240] In some embodiments, the extraction control information uploaded by the client for extracting part of the intermediate rendering result from the unified intermediate rendering result is not limited to information indicating the viewing angle, but can also be area information (such as a reflection area), etc. The specific content of the extraction control information may be related to the characteristics. The extraction control information can be used to reduce the bandwidth occupied by the client by the intermediate rendering results sent by the server 200 to the client.

[0241] Optionally, the server 200 may store the control information of each client parsed by the collaborative module 103b, such as the requirements for feature resources (such as feature status, camera posture, status of feature resources, etc.), so that the server 200 can control the calculation and processing process of each feature, thereby obtaining the intermediate rendering results of the corresponding features that meet the requirements of each client and pushing them to the corresponding client.

[0242] In combination with Figure 4b, referring to Figure 3, the collaboration module 103b can synchronize the changed game scene, feature status, feature resource status, camera posture, etc. to the rendering module 101b, so that the rendering module 101b can render each feature that is in the turned-on state after aggregation separately based on the changed game scene.

[0243] In an embodiment of the present application, in order to save cloud computing resources, when multiple clients (such as the above-mentioned client 100 and client 101) request the server 200 to render the same game scene, the server 200 of the present application aggregates the feature states reported by multiple clients, thereby uniformly rendering each feature that is in the enabled state after aggregation, enabling multiple clients to share computing resources for the same feature. In this way, when multiple clients request to render the same feature, the server 200 does not need to repeatedly render the same feature multiple times, but only needs to render it once, allowing multiple clients to share cloud computing resources.

[0244] The rendering module shown in FIG3 can initiate the calculation process of each feature based on the updated game scene, feature status, feature resource status, camera pose, etc., to obtain a feature stream for each feature. The encoding module 102b can encode the feature stream for each feature based on the status of the feature resource to obtain a feature code stream, and transmit it to the client 100 via the transmission module 104b.

[0245] In a possible implementation, there are multiple clients requesting rendering of the same game scene, such as client 100 and client 101 shown in FIG5 . The processing process of the server 200 is described below with reference to FIG5 .

[0246] Combined with Figure 4b, as shown in Figure 5, the server 200 can receive the frame status uploaded by the client 100 and the client 101 respectively, and summarize the frame status to obtain the summarized characteristic status, for example, the characteristic of direct lighting is on, the characteristic of indirect lighting is on, and the characteristic of shadow is on.

[0247] As shown in FIG5 , the server 200 can also obtain the status of the characteristic resources and the camera posture of each client from the frame status uploaded by the client 100 and the client 101 .

[0248] As shown in FIG5 , the server 200 may also aggregate the game states in the frames uploaded by the client 100 and the client 101 to determine the changed game scene (also referred to as the updated game scene) when the game state is updated.

[0249] As shown in FIG5 , the characteristic states summarized by the server 200 are the characteristic states of the three characteristics supported by the server 200. Therefore, the server 200 can perform the calculation process for the characteristics of direct lighting through the characteristic component 201 and control the calculation process according to the respective requirements of the client 100 and the client 101. The requirements can be the changed game scene shown in FIG5 , the characteristic state of each client, the status of the characteristic resource of each client, the camera posture of each client, etc.

[0250] Similarly, the server 200 can also perform a calculation process of the characteristics of the reflected light through the characteristic component 202, and control the calculation process according to the respective requirements of the client 100 and the client 101.

[0251] Furthermore, the server 200 may also perform a calculation process of the characteristics of the shadow through the characteristic component 203 , and control the calculation process according to the respective requirements of the client 100 and the client 101 .

[0252] The following takes the execution process of the feature component 201 as an example to illustrate how the server 200 controls the calculation and processing process of the direct lighting features according to the needs of each client. The execution principles of other feature components are similar to the execution principles of the feature component 201 introduced here, and will not be repeated here.

[0253] As shown in Figure 5, the aggregated direct lighting feature state is on, so feature component 201 can execute the direct lighting rendering process. In some embodiments, for example, if the aggregated feature state of a certain feature (such as the shadow feature) is off, the corresponding feature component 203 does not need to render the shadow feature for the game scene after this change.

[0254] As shown in FIG5 , first, the feature component 201 may perform feature rendering of direct lighting based on the changed game scene to obtain an intermediate rendering result of the feature (eg, an irradiance map).

[0255] Then, the feature component 201 may write the intermediate rendering result of the feature to the shared cache so that the client 100 and the client 101 may share the intermediate rendering result.

[0256] In the prior art, the feature rendering method can only provide the required feature rendering for a specific perspective. In an embodiment of the present application, the rendering method used by each feature component in the server 200 can support rendering of a shareable intermediate rendering result. That is, multiple clients can share the intermediate rendering results of the corresponding features rendered by the server 200, and use the perspective of each client (an example of extracting control information) to extract a part of the results from the intermediate rendering results obtained by the server 200 to obtain the intermediate rendering results of the corresponding features that conform to the perspective of each client, thereby realizing the sharing of computing resources for the same feature in the cloud.

[0257] In some embodiments, the intermediate rendering result may be stored in a format such as an irradiance map or a lightmap, so as to support multiple clients to extract intermediate rendering results that meet their client requirements from the cached unified intermediate rendering result.

[0258] Next, the feature component 201 can extract the cached intermediate rendering results of the direct lighting according to the status of the feature resources of each client and the camera posture of each client, so as to extract the intermediate rendering results that meet the requirements of the status of the feature resources and camera posture of each client as the feature flow of each client.

[0259] For example, since the characteristics of the direct lighting rendered by the feature component 201 are based on the overall changed game scene, the game scene is a three-dimensional scene, but the perspectives (such as camera postures) of each client may be different, resulting in the game screen of the game scene displayed on each client needing to be the game screen of the game scene under that perspective. Therefore, the feature component 201 can extract the intermediate rendering results that match the camera posture of the client 100 from the intermediate rendering results of the cached direct lighting, so as to facilitate subsequent rendering to obtain the game screen of the changed game scene that conforms to the perspective of the client 100. For example, the extracted intermediate rendering results do not involve the characteristic effect of the invalid area blocked by the user's line of sight of the client. The specific effect may vary according to the difference in logic set by the game developer, which is not limited here.

[0260] It should be understood that the extraction control information carried by each client in the frame state is not limited to the camera posture, but can also be a viewing cone, a reflection area, etc.

[0261] The status of the feature resource uploaded by the client 100 may include the resolution of the intermediate rendering result of the direct lighting feature, wherein the intermediate rendering result of the direct lighting feature rendered by the feature component 201 has a higher resolution. Then the feature component 201 may continue to downsample the intermediate rendering result that matches the camera posture of the client 100 according to the resolution required by the client 100 to obtain an intermediate rendering result that meets the resolution required by the client 100.

[0262] In this way, the feature component 201 can extract the irradiance map (represented by the feature flow in FIG5 ) that meets the viewing angle of the client 100 and the network quality (here represented by the resolution of the intermediate rendering result) through the cache.

[0263] In this embodiment, the server 200 can use shared computing resources to uniformly render certain features requested by multiple clients to save computing resources; and write the rendered intermediate rendering results into a shared cache, and combine the perspective information of different clients to extract the intermediate rendering results that meet the client's perspective from the cache, for example, so that the extracted intermediate rendering results do not involve the characteristic effects of invalid areas blocked by the user's line of sight of the client; and the resolution of the extracted intermediate rendering results also meets the resolution requirements of the client, so that the quality of the extracted intermediate rendering results can meet the client's current network status.

[0264] The principle of feature component 201 extracting the intermediate rendering result of the direct lighting from the cache according to the needs of client 101 is similar to the principle of extracting the irradiance map that meets the viewing angle and resolution of client 100 introduced here, and will not be repeated here.

[0265] Finally, as shown in Figure 5, the feature component 201 (for example, the encoding module 102b shown in Figure 3) can compress and encode the feature stream of each client extracted from the cache according to the status of the feature resources of each client (for example, the compression rate) to obtain the code stream of the direct lighting feature that meets the requirements of the client 100, and the code stream of the direct lighting feature that meets the requirements of the client 101.

[0266] Similarly, the characteristic component 202 can output a code stream with the reflected illumination characteristic that meets the requirements of the client 100 , and a code stream with the reflected illumination characteristic that meets the requirements of the client 101 .

[0267] Similarly, the characteristic component 203 can output a code stream that obtains the shadow characteristics of the client 100.

[0268] In some embodiments, the feature component in the server 200 (e.g., encoding module 102b as shown in FIG3 ) can compress the feature stream of each feature using a matching encoding algorithm based on the data characteristics of the feature stream. For example, spherical harmonic Gaussian encoding can be used for irradiance maps, motion vector-based encoding can be used for quasi-static content, and region of interest (ROI) encoding can be used for specular reflections, etc., to minimize the network bandwidth occupied by the transmission of the feature stream.

[0269] In other embodiments, the state of the client's feature resource or the client's camera posture may be combined separately to extract the intermediate rendering result of the corresponding feature. The principle is similar and will not be repeated here.

[0270] Returning to Figure 3, in this embodiment, the rendering module 101b in the server 200 can be controlled by the collaboration module 103b to adjust the calculation and processing process of different rendering characteristics according to the control information carried in the frame state, and drive the rendering pipeline module 107b to render the characteristics of the changed game scene; and the rendering module 101b can cache the intermediate rendering results rendered by the rendering pipeline module 107b, and extract the cached intermediate rendering results according to the requirements of the client 100 to obtain a feature stream that meets the requirements of the client 100; then, the feature stream is encoded through the encoding module 102b, and the encoded feature code stream is transmitted back to the client 100 through the transmission module 104b.

[0271] In this embodiment, based on the game scene, rendering an intermediate rendering result of a certain characteristic requires higher computing power. This application configures the process of rendering the intermediate rendering result to the rendering module 101b and the game pipeline 105b in the server 200 as shown in Figure 3 in the development state. In this way, the calculation of the ray tracing special effects can be transferred from the mobile terminal to the cloud in the running state, so that mobile terminals with lower computing power can also run games with ray tracing special effects.

[0272] S105 , the server 200 sends the corresponding intermediate rendering result to the client 100 according to the control information.

[0273] In the embodiment of FIG. 5 , the intermediate rendering result in S105 may be an encoded code stream of corresponding characteristics, wherein the code stream may be a data stream composed of the intermediate rendering results compressed according to the compression rate required by the client 100 in the control information.

[0274] As shown in FIG5 , the server 200 may perform split control on the characteristic code stream sent to each client according to the characteristic status of each client carried in the frame status uploaded by the client 100 and the client 101 .

[0275] As described above, the client 100 requests to enable all three of the above features, while the client 101 only enables the direct lighting feature and the reflected lighting feature corresponding to the feature component 201 and the feature component 202 respectively. Then, the server 200 can transmit the code streams of the above three features requested by the client 100 that meet the requirements of the client 100 to the client 100 through the network.

[0276] The server can transmit the code stream of direct lighting characteristics and the code stream of reflected lighting characteristics that meet the needs of the client 101 to the client 101 through the network, without distributing the code stream of shadow characteristics to the client 101.

[0277] In some embodiments, the code streams of multiple characteristics distributed by the server 200 to the same client (eg, client 100) may be distributed to the client 100 in a parallel manner as shown in FIG5 or in a serial manner, without limitation.

[0278] In addition, when the server 200 distributes the required characteristic code streams to multiple clients, it can send the required code streams to multiple clients in parallel to improve the rendering efficiency of multiple clients for game screens with characteristics and reduce game delays and freezes.

[0279] In one embodiment, the server 200 may establish a network transmission channel for each client through the transmission module 104b shown in FIG. 3 , and transmit the characteristic code stream of each client and its related stream information to each client.

[0280] S106: The client 100 renders the game scene with corresponding characteristics based on the intermediate rendering result to obtain a target rendering result of the characteristics in the game scene.

[0281] As shown in Figure 3, the client 100 can receive three types of characteristic code streams (optionally including stream information) from the server 200 through the transmission module 104a. The transmission module 104a can obtain the data format based on the stream information, and then call the decoding module 102a to use the decoding algorithm matching the data format to decode the code streams of the corresponding characteristics to obtain characteristic streams of the three characteristics.

[0282] In some embodiments, the decoding module 102a may call hardware decoding according to specific hardware characteristics of the mobile terminal to improve the decoding efficiency of the characteristic code stream.

[0283] In addition, as shown in FIG3 , the coordination module 103 a may receive current network status information from the transmission module 104 a and may combine the network status information to update the frame status of the next frame.

[0284] In addition, as shown in Figure 3, the collaboration module 103a in the client 100 can control the rendering module 101a to inject (for example, replace the intermediate rendering results, etc.) the feature streams of the above three features synchronized with the server 200 (specifically, the intermediate rendering results of the features) into the basic rendering pipeline of the rendering pipeline module 105a, so as to achieve the update and enhancement of the rendering effect of the features in the end-side game screen when the rendering scene (here is the game scene) is updated. In this way, when the user operates the game and causes the game scene to be updated, the output game screen can also be updated with the ray tracing special effects.

[0285] In a possible implementation, the process of the client 100 processing the code streams with the above three characteristics received from the server 200 is described with reference to FIG. 5 .

[0286] As shown in FIG5 , the client 100 can receive three types of characteristic code streams from the server 200 . Specifically, the code stream of direct lighting characteristics, the code stream of reflected lighting characteristics, and the code stream of shadow characteristics are input into the characteristic component 301 , the characteristic component 302 , and the characteristic component 303 of the client 100 , respectively.

[0287] The processing of the three feature components is similar. Let's take the bitstream processing of the direct illumination feature by feature component 301 as an example to illustrate:

[0288] The feature component 301 can decode the code stream of the direct lighting feature to obtain the feature stream of the direct lighting (here is the irradiance map); then, the feature component 301 can convert the feature stream to obtain the converted intermediate rendering result (also called rendering resource) that is compatible with the basic pipeline in the rendering pipeline module 105a.

[0289] For example, feature component 301 can convert an irradiance map into rendering resources such as spherical harmonic Gaussian coefficients to be compatible with the basic pipeline in rendering pipeline module 105a. For another example, feature component 302 can convert a feature stream of reflected lighting (such as a reflection map) into a cubemap to be compatible with the basic pipeline in rendering pipeline module 105a.

[0290] Finally, the feature component 301 may inject the converted intermediate rendering result into the rendering pipeline module 105 a .

[0291] As shown in FIG5 , feature component 301 , feature component 302 , and feature component 303 can respectively inject the converted intermediate rendering results of the three features into the rendering pipeline module 105 a .

[0292] For example, the feature component 301 may replace the preset spherical harmonic Gaussian coefficients in the rendering pipeline module 105a with the spherical harmonic Gaussian coefficients obtained by the above conversion. For another example, the feature component 302 may replace the preset cubemap in the rendering pipeline module 105a with the cubemap obtained by the above conversion.

[0293] The rendering pipeline module 105a can render the above three characteristics in combination with the changed game scene, and replace the preset intermediate rendering results of the three characteristics with the converted intermediate rendering results of the above three characteristics from the cloud, thereby obtaining the target rendering results of the above three characteristics under the game scene. The rendering pipeline module 105a obtains the game screen that can be displayed on the screen by superimposing and post-processing the target rendering results of the three characteristics.

[0294] For ease of understanding, please refer to the processes shown in Figures 6a and 6b to understand the process of feature injection.

[0295] FIG6 a shows a process of rendering a screen-capable image with three characteristics of direct lighting, reflected lighting, and shadow by the client 100 in the related art.

[0296] FIG6 b shows a process in which the client 100 and the server 200 of the present application collaborate to render a screen-capable image with three characteristics: direct lighting, reflected lighting, and shadows.

[0297] As shown in FIG6 a , when the client 100 detects a change in the game scene, it may start the rendering process for the three characteristics of direct lighting, reflected lighting, and shadows.

[0298] For example, the client 100 renders the characteristics of direct lighting, reflected lighting, and shadows for the game scene in sequence.

[0299] The client 100 can render the direct lighting characteristics of the game scene according to the direct lighting rendering process to obtain an image 11 with direct lighting; then, the client 100 can render the reflected lighting for the image 11 according to the reflected lighting rendering process to obtain an image 21 with not only direct lighting but also reflected lighting; then, the client 100 can render the shadow for the image 21 according to the shadow rendering process 1 to obtain an image 31 with direct lighting, reflected lighting and shadow at the same time; finally, the client 100 can post-process the image 31 (such as tone mapping, etc., without limitation) to obtain an image 41 that can be displayed on the screen.

[0300] It should be understood that the rendering order of each feature by the client 100 may vary based on the difference in games. The rendering order here is only an illustrative example and is not intended to limit this application.

[0301] The principle of the rendering process of each feature by the client 100 is similar, and the process of rendering direct lighting is taken as an example for explanation.

[0302] As shown in Figure 6a, since rendering to obtain an irradiance map requires high computing power, and the computing power of the client 100 is relatively low, the client 100 is configured with a preset irradiance map (the intermediate rendering result of the direct lighting characteristics) in the development state. Then, the client 100 can calculate the lighting for the preset irradiance map to obtain the above-mentioned image 11 to realize the direct lighting rendering process.

[0303] In the process of Figure 6a, the intermediate rendering results of the corresponding characteristics used in the rendering process of each characteristic are preset intermediate rendering results (such as preset irradiance maps). Then, when the game scene changes, any characteristics of direct lighting, reflected lighting, and shadows in the game screen will not change. In other words, the characteristic effects in each frame of the game screen of the client 100 are the same during operation and will not change.

[0304] However, in the technical solution of the present application as shown in Figure 6b, as shown in Figure 6b, the client 100 can render the direct lighting characteristics of the game scene according to the direct lighting rendering process configured in the development state, and obtain an image 12 with direct lighting; then, the client 100 can render the reflected lighting for the image 12 according to the reflected lighting rendering process configured in the development state, and obtain an image 22 with not only direct lighting but also reflected lighting; then, the client 100 can render the shadow for the image 22 according to the shadow rendering process configured in the development state, and obtain an image 32 with direct lighting, reflected lighting and shadow at the same time; finally, the client 100 can post-process the image 32 to obtain an image 42 that can be displayed on the screen (for example, the game screens shown in Figures 3 and 5).

[0305] The principle of the rendering process of each feature by the client 100 is similar, and the process of rendering direct lighting is taken as an example for explanation.

[0306] As shown in FIG6b, since rendering to obtain an irradiance map requires a relatively high computing power, the server 200 can perform direct lighting rendering on the above-mentioned game scene to obtain an intermediate rendering result of direct lighting, which is here the target irradiance map; then, the client 100 can replace the rendering resources used in the direct lighting rendering process (here the preset irradiance map) with the target irradiance map rendered by the server 200; then, the client 100 calculates the lighting for the target irradiance map, thereby completing the injection and fusion of the target irradiance map to obtain image 12, wherein image 12 can be an example of the target rendering result in S106. Similarly, the rendering resources used in the reflected lighting rendering process shown in FIG6b are also replaced with the intermediate rendering result of the reflected lighting characteristics generated by the server 200, and the rendering resources used in the shadow rendering process shown in FIG6b are also replaced with the intermediate rendering result of the shadow characteristics generated by the server 200.

[0307] Thus, compared to Figure 6a, in the solution shown in Figure 6b of the present application, the game screen displayed by client 100 can achieve real-time updates of the three characteristics of direct lighting, reflected lighting, and shadows in the changed game scene when the game scene changes, rather than constantly maintaining the same direct lighting effect, the same reflected lighting effect, and the same shadow effect. Furthermore, the visual effects of the three characteristics updated in real time can match the needs of client 100, such as the current network status of client 100, the changed game scene, and the user's perspective of client 100.

[0308] In the development state, the game pipeline module 105a in the client 100 can be configured with the rendering processes of the above three features, so that the client 100 can support the injection of the above three features.

[0309] In summary, the client 100 of the present application (e.g., a game executable program installed on a mobile terminal) can be directly controlled by the user; it has complete game logic, can generate changes in the game scene, and synchronizes the client 100's rendering requirements for features to the cloud through the collaborative module 103a, thereby driving the server 200 deployed on the cloud to render the changed game scene to meet the above-mentioned features of the rendering requirements. Furthermore, if the client 100 only has basic rendering capabilities (e.g., the ability to calculate lighting as shown in FIG6b , but not the ability to calculate intermediate rendering results of features), the rendering module 101a in the client 100 can enhance the effects of the various features rendered by the client 100 based on the intermediate rendering results of the features sent from the cloud, and the resulting synthetic rendering results meet the game screen requirements of the client 100.

[0310] The server 200 of the present application (for example, a game executable program deployed on a cloud-side device) only has basic game logic (for example, the logic of calculating the intermediate rendering results of the characteristics), which can be controlled by the collaborative module 103b. The characteristics will be rendered only when the game scene is updated (wherein, the game scene is not updated and there is no need to render the above characteristics). And when rendering the characteristics, it can be rendered according to the rendering requirements of the client 100 synchronized with the collaborative module 103b to obtain an intermediate rendering result of the characteristics that meet the requirements of the client 100. In addition, compared to the prior art in which the lighting characteristics of each frame of the game screen are fixed, the present application can adjust the lighting characteristics of the changed game scene in real time by the cloud when the game scene changes, and has the complete rendering capability of the intermediate rendering result, including ray tracing rendering characteristics or high-quality rendering resources, etc. The server 200 sends the intermediate rendering results of the characteristics calculated by the rendering module 101b to the client 100 for the injection of characteristics, thereby realizing the transfer of computing power for calculating the intermediate rendering results from the mobile terminal to the cloud.

[0311] In the above embodiment, the client 100 detects that the game scene has changed and needs to update the characteristics of the game screen. In another possible implementation, when the client 100 detects that the game scene has not changed and only the camera posture has changed, the client 100 can synchronize the changed camera posture to the server 200 through the collaboration module 103a and the transmission module 104a. The server 200 can cache the intermediate rendering results of various characteristics rendered in the previous frame of the game screen.

[0312] Different from the embodiment in which the game scene changes, in this embodiment, the server 200 does not need to recalculate the characteristics. The server 200 can extract the intermediate rendering results of the perspective that matches the camera posture from the cached intermediate rendering results related to the last calculated frame of the game screen based on the camera posture synchronized with the client 100, and send it to the client 100 to update the game screen of the client 100 to the game screen within the changed perspective.

[0313] In this way, when the game scene of the client 100 has not changed but only the user's perspective has changed, the client 100 can still update the game screen to the game screen within the changed perspective by collaborating with the cloud. The specific implementation process is mostly the same as the above-mentioned embodiment in which the game scene changes. The difference is that the server 200 does not need to recalculate the characteristics (such as the characteristic rendering operation of each characteristic component in the server 200 as shown in Figure 5), but only needs to extract the intermediate rendering results related to the cached previous frame of the game screen according to the changed user perspective. Optionally, the server 200 can compress the extracted intermediate rendering results according to the encoding requirements required by the client 100. Among them, the update of the game screen will not cause the change of the type of characteristics in the game screen, but the perspective of various characteristics in the game screen has changed.

[0314] The following takes the rendering characteristics of direct lighting, reflected lighting, and shadow as an example, and describes the collaboration process between the above-mentioned client 100 and server 200 of the present application in conjunction with Figure 7.

[0315] The processing in Figure 7 is largely identical to the processing in Figures 3 through 5 and Figure 6b described in the previous embodiment, and will not be detailed here. This example aims to detail the calculation, extraction, and compression of the intermediate rendering results of the three features by the server 200, as well as the decompression and feature injection processes by the client 100. Figure 7 uses a single client as an example. When the game scene is updated and there are multiple clients, the principles are similar. Please refer to the above for details and will not be detailed here.

[0316] Feature 1: Direct lighting. In rendering, direct lighting refers to the lighting effect obtained when the light emitted by the light source directly reaches the surface of the object without ricocheting. In game applications, considering the complexity of calculations and the number of light sources, the direct dynamic lighting effects of particle light sources (hundreds to thousands) are generally not considered, especially on mobile platforms. Due to the huge number of particles, real-time lighting calculations are unrealistic, but particle effects are often crucial to the expressiveness of the game. In this case, you can consider offloading the direct lighting part of the particles to the cloud to achieve the lighting effects of a large number of particles, thereby enhancing the rendering atmosphere of the game.

[0317] Feature 2: Planar reflection (specular or direct). In rendering, reflection describes the phenomenon in which light waves are reflected and propagated back when they reach the interface of a medium. When representing materials such as smooth surfaces (metal, smooth ground) and water surfaces (lakes, water on the ground), adding reflection can significantly improve the image quality. When the reflective surface is a plane, the reflected content can be obtained by rendering once by mirroring the rendering camera along the reflective plane. However, if it needs to be updated in real time, an additional full-scene draw call is required, which increases the rendering burden on the client side. Therefore, it is considered to offload its calculations to the cloud to assist the client side in rendering the reflection effect.

[0318] Feature 3: Soft Shadows. In rendering, shadows describe the effect of light being blocked by an object and unable to illuminate the surface of the object. When the light source is a surface light source, the shadow formed is not black or white, but has a certain gradient transition area, which is called penumbra or soft shadow. To achieve the soft shadow effect, it is generally approximated by transforming the hard shadow, but this implementation method often has certain flaws; more stable and realistic soft shadows need to be achieved through ray tracing algorithms. However, the computing power and power consumption required by the ray tracing algorithm are unaffordable for current mobile terminals, so it is considered to offload its calculations to the cloud to assist the terminal side in achieving soft shadow rendering effects.

[0319] In summary, this embodiment constructs end-cloud collaborative rendering for three rendering features: direct illumination (DI), planar reflection, and shadow. Its operation design is shown in Figure 7, and the specific operation logic will be described in conjunction with Figures 8a to 8c and Figures 9a to 9c.

[0320] Optionally, in the development state, the device 300 of the present application as a plug-in as shown in Figure 1 can provide different user interfaces for the above-mentioned multiple characteristics (DI, planar reflection, shadow) for developers to configure the rendering module, encoding module, transmission module and collaborative module of the server 200, for example, the resolution of the intermediate rendering result (such as a map) rendered by the rendering module can be configured.

[0321] 7 , the client 100 executes the game logic based on the user's operation instructions, so that the game scene is updated.

[0322] When the game scene is updated, the client 100 can package the frame status and transmit it to the server 200 via the network.

[0323] In addition, when the game scene is updated, the client 100 can also perform physical or animation calculations, and perform feature rendering based on the intermediate rendering results of the above three characteristics after replacement to obtain an image with the above three characteristics.

[0324] After the server 200 is synchronized to the frame state from the client 100, the calculation, caching, extraction and encoding of direct illumination (DI), reflected illumination and shadow can be performed according to the frame state.

[0325] The following describes the process of calculating, caching, extracting and encoding direct lighting.

[0326] FIG8 a shows a process of the rendering module and the encoding module of the server 200 rendering direct lighting characteristics for the changed game scene.

[0327] As shown in FIG8a , for direct lighting calculations, the rendering module may determine light sources and light probes based on the game scene, where light probes are points (or spheres) in three-dimensional space, and direct lighting light sources may illuminate these spheres.

[0328] First, the rendering module can traverse the light probes in the game scene;

[0329] Then, for each light probe, the rendering module can iterate over the list of direct light sources that need to be calculated;

[0330] Next, the rendering module can calculate the direct lighting of each light probe for the list of direct light sources, and obtain the direct lighting result of each direct light source for all light probes;

[0331] Considering the radiance distribution of each direct light source to the light probe after attenuation at a certain distance, for each light probe, an integral filter can be performed based on the direct lighting results corresponding to all direct light sources to obtain the irradiance map of the light probe;

[0332] The rendering module can then store the irradiance map for each light probe in a probe irradiance database.

[0333] The irradiance of the light probe can be cached in a spatial sparse structure. The irradiance map of the light probe can store the irradiance of the point in all directions. The irradiance I can be calculated by formula 1: I = ∫ Ω L(x i ,wi )(w i ·n)dw i , formula 1;

[0334] In formula 1, x, w, n, and Ω represent the position, unit solid angle direction, surface normal, and spherical area domain respectively; L is the radiance, which is used to describe the radiation intensity at a certain position x along a certain unit solid angle direction w, per unit time, and per unit area, that is, the color and brightness perceived by the human eye, and L e Specifically refers to the self-luminous radiance; f(x o ,w o ,x i ,w i ) is a bidirectional distribution function, which is used to describe how light interacts with the medium, that is, the incident light (x i ,w i ) for the outgoing ray (x o ,w o ) considers the direct illumination result from the light source to the probe.

[0335] As shown in Figure 8a, the rendering module can select the irradiance map of the light probe within the corresponding client's perspective (which can be called the irradiance map of the effective light probe) from the probe irradiance database based on the camera posture of each client (position and perspective relationship, etc.), taking client 100 as an example here, to eliminate unnecessary irradiance maps of light probes and thus save data transmission bandwidth.

[0336] The rendering module can combine the irradiance maps of the valid light probes to obtain a complete irradiance map and send it to the encoding module.

[0337] Optionally, when performing DI extraction, the rendering module of the server 200 can also downsample the complete irradiance map based on the resolution requirements of the irradiance map in the frame state uploaded by the client 100 to extract an irradiance map that meets its resolution requirements.

[0338] Finally, the encoding module can compress and encode the extracted irradiance map according to the video stream mode and the compression rate indicated by the client 100 based on the requirements for the bitstream in the frame status uploaded by the client 100 (such as the compression rate in the status of the characteristic resource), so as to adjust the bitrate of the bitstream, obtain the DI bitstream of the client 100 and send it to the client 100.

[0339] As shown in FIG7 , after receiving the code stream of DI, the client 100 may decode it to obtain the above-mentioned complete irradiance map and replace the irradiance map.

[0340] Specifically, FIG9a shows the injection process of the direct illumination characteristics:

[0341] For direct lighting, considering that the final shading calculation is performed directly based on the irradiance map, there are certain problems in sampling and compatibility. Therefore, the rendering module of the client 100 can first convert the irradiance map into a spherical harmonic Gaussian expression (such as spherical harmonic Gaussian coefficients). Then, the client 100 can replace the preset spherical harmonic Gaussian coefficients in the rendering process of its own direct lighting with the above-mentioned converted spherical harmonic Gaussian coefficients from the server 200 to achieve the replacement of the intermediate rendering results of the direct lighting. The specific principle is similar to the principle described in the embodiment of Figure 6b above, and will not be repeated here. In this way, the intermediate rendering results of the direct lighting sent from the cloud are injected.

[0342] In this way, when the game scene changes, the special effects of direct lighting in the game screen displayed by the client 100 may also change.

[0343] The following describes the process of calculating, caching, extracting and encoding reflected lighting.

[0344] FIG8 b shows the process of the rendering module and the encoding module of the server 200 rendering the reflected lighting characteristics of the changed game scene.

[0345] As shown in Figure 8b, for planar reflection calculations, the rendering module first performs texture expansion on objects in the game scene that can be reflected and parameterizes them onto a Texture Space Shading (TSS) map. Then, after the game scene is updated, the rendering module performs shading calculations in this TSS texture space and stores them in the TSS texture map database. The rendering module then calculates the mirror camera based on the camera position and viewing angle of each client. It then performs raster sampling from the TSS texture map database based on the mirror camera to obtain the rendering result of the reflection camera (here, the reflection texture map). The rendering module then removes invalid areas from the reflection texture map based on the viewing angle information of each client (e.g., camera pose). For example, it removes content in non-reflective areas or content that can be efficiently calculated on the mobile terminal side, such as skybox content, to minimize bandwidth. The removed reflection texture map is then sent from the rendering module to the encoding module for encoding and distribution to each client. For example, the encoding module can compress and encode the reflection texture map according to the video stream mode and the compression rate indicated by the client 100 based on the code stream requirements in the frame state uploaded by the client 100 (such as the compression rate in the state of the characteristic resource), obtain the code stream of the reflection lighting required by the client 100 and send it to the client 100.

[0346] As shown in FIG. 7 , after receiving the code stream of the reflected light, the client 100 may decode the code stream to obtain the aforementioned reflected texture map, and then replace the reflected texture map.

[0347] Specifically, FIG9b shows the injection process of the characteristics of the reflected light:

[0348] As for reflected lighting, considering that the use of commonly used planar reflection probes is very common, the game pipeline on the mobile terminal side only needs to be pre-configured with a rendering process for calculating planar reflection probes. Then, the client 100 can replace the preset reflection probe texture resources in its own upper rendering process with the reflection texture map sent by the cloud server 200, and the planar reflection characteristics can be made effective in the rendering results of the client 100 in a compatible manner, thereby realizing the effect of dynamic planar reflection in the game screen.

[0349] In this way, when the game scene changes, the special effects of the plane reflection in the game screen displayed by the client 100 may also change.

[0350] The following describes the process of shadow calculation, caching, extraction and encoding.

[0351] FIG8 c shows the process of the rendering module and the encoding module of the server 200 rendering the shadow characteristics of the changed game scene.

[0352] As shown in Figure 8c, for soft shadow calculations, all objects in the scene that can be reflected are first texture-unwrapped and parameterized onto a TSS texture map. After the game scene is updated, the rendering module determines the area light sources based on the updated scene. The rendering module then traverses the area light sources and calculates the shadow coefficient (between 0 and 1) caused by each area light source in texture space. This shadow system can be defined as the ratio of the shading result with occlusion considered (between 0 and 1) to the shading result without occlusion considered (between 0 and 1). After calculating the shadow coefficient, it can be stored as a shadow texture in the TSS texture map database. Next, raster sampling is performed in the TSS texture map database based on the client's camera pose (position and viewing angle) to obtain the shadow texture from the client's camera's perspective. Considering that there can be multiple area light sources (for example, there are four lights in this game scene), shadow textures need to be merged. Among them, since the shadow coefficient is a single channel and a single layer is formed for a single light source, when the rendering module merges the shadow textures, it can first merge the shadow layers according to the number of surface light sources to obtain a commonly used 4-channel layer. Afterwards, the rendering module sends the merged shadow texture to the encoding module. The encoding module can compress and encode the merged shadow texture according to the video stream mode and the compression rate indicated by the client 100 based on the code stream requirements (such as the compression rate in the state of the characteristic resource) in the frame state uploaded by each client (such as the client 100), obtain the code stream of the shadow characteristics required by the client 100, and send it to the client 100.

[0353] As shown in FIG7 , after receiving the shadow code stream, the client 100 can decode it to obtain the shadow texture (specifically, the shadow coefficient, represented by the soft shadow coefficient map in FIG9 c ), and replace the shadow coefficient.

[0354] Specifically, Figure 9c shows the injection process of shadow characteristics:

[0355] For soft shadow shading, since the rendering method based on the shadow coefficient does not have a corresponding module in the game engine, in the development state, the present application can use the device 300 as described in Figure 1 to expand the functionality of the existing game engine, so as to configure the shader (such as a function) in the device 300 to the game pipeline module of the client 100, so that the client 100 can customize a soft shadow shader based on a given shadow coefficient.

[0356] When injecting shadow features, after the client 100 obtains the soft shadow coefficient map from the server 200, the above-mentioned soft shadow shader in the client 100 can sample the coefficient map from the soft shadow coefficient map to obtain the corresponding shadow coefficient; then, the client 100 can obtain a shading result containing soft shadow special effects based on the obtained shadow coefficient and the lighting calculation result of the client 100 side for the game scene without considering object occlusion.

[0357] In this way, when the game scene changes, the game screen displayed by the client 100 may have the characteristics of shadows and may change as the game scene is updated.

[0358] Continuing back to FIG7 , after the client 100 decodes, generates and replaces the above three features, it can use the intermediate rendering results of the replaced features in combination with physical / animation calculations to perform feature rendering. Optionally, particle effects, post-processing and other processing can also be performed to obtain an image that can be displayed on the screen.

[0359] In a possible implementation, the client 100 and server 200 of the present application may also support collaborative rendering of more features, which may include but are not limited to: indirect lighting, spherical reflection, and ambient occlusion (AO).

[0360] For indirect lighting, the process of calculating the characteristics of the indirect lighting by the server 200 is similar to the process of calculating the characteristics of the direct lighting shown in FIG8 a .

[0361] FIG8 d shows the process of the rendering module and the encoding module of the server 200 rendering the indirect lighting characteristics of the changed game scene.

[0362] As shown in Figure 8d, first, the rendering module can traverse the probes based on the game scene, emit light for each probe, and then color the light to obtain the radiance of each probe, and then perform integral filtering on the radiance of each probe to obtain the irradiance map of each probe. Then, the rendering module can store the irradiance map of each probe in the probe irradiance database. Taking into account that the results of indirect lighting can also be stored as irradiance maps, the server 200 can superimpose and issue the intermediate rendering results of indirect lighting (here is the irradiance map) and the intermediate rendering results of direct lighting (the format is also the irradiance map) before issuing the intermediate rendering results of indirect lighting (here is the irradiance map). For the extraction, encoding and injection process of the intermediate rendering results of indirect lighting by the client 100, please refer to the relevant introduction of Figures 8a and 9a above on the characteristics of direct lighting, which will not be repeated here.

[0363] In this way, when the game scene changes, the game screen displayed by the client 100 may have the characteristics of indirect lighting and may change as the game scene is updated.

[0364] For spherical reflection, it is equivalent to a panoramic camera. Figure 8e shows the process of the rendering module and the encoding module of the server 200 rendering the spherical reflection characteristics of the changed game scene.

[0365] As shown in Figure 8e, the server 200 can traverse the spherical reflection probes based on the game scene and calculate the spherical reflection texture for each spherical reflection probe. Then, the spherical reflection texture of each spherical reflection probe is stored in the spherical reflection texture database; then the server 200 selects the spherical reflection texture of the spherical reflection probe that meets the perspective of the client 100 based on the camera posture (position and perspective) of the client 100; then, the server 200 merges all the selected spherical reflection textures and encodes them through the encoding module in combination with the compression rate requirements of the client 100 for the spherical reflection texture; finally, the server 200 sends the compressed spherical reflection characteristic code stream (such as the spherical reflection texture) to the client 100.

[0366] Similar to the processing process of Figures 9a to 9c, after the client 100 obtains the spherical reflection texture with spherical reflection characteristics from the server 200, it can replace the preset spherical reflection texture in the rendering process of the spherical reflection texture of the client 100 with the spherical reflection texture issued by the server 200, thereby realizing the rendering of dynamic spherical reflection characteristics of the game screen in a compatible manner.

[0367] In this way, when the game scene changes, the game screen displayed by the client 100 may have the characteristics of spherical reflection and may change as the game scene is updated.

[0368] AO can be understood as soft shadows from indirect lighting (e.g., shadows from corners). Regarding the rendering of AO characteristics, since it can be understood as a special shadow coefficient, its calculation process within the cloud server 200 can refer to the soft shadow rendering calculation process shown in Figure 8c above. Furthermore, the server 200 can merge the intermediate rendering results of AO with the intermediate rendering results of soft shadows, compress them, and send them to the client 100.

[0369] After receiving the code stream of the intermediate rendering result of the AO, the client 100 can decompress the code stream to extract the AO map from the multi-channel image; thereafter, the client 100 can replace the preset AO texture map of the game engine with the extracted AO map, which can be compatible with the given method to realize the dynamic AO characteristics of the game screen.

[0370] In this way, when the game scene changes, the game screen displayed by the client 100 may have the characteristics of AO and may change as the game scene is updated.

[0371] In the embodiment of FIG. 7 , the server 200 may be deployed on a server in the cloud. In this embodiment, considering that multiple users access cloud services, the computing pressure on a single server may be too high, and the server may be unable to carry the computing tasks of all clients.

[0372] In this embodiment, different from FIG7 , as shown in FIG10 , the computing tasks of each feature can be separately executed on the cloud side through independent graphics card hardware or independent server hardware, thereby improving the rendering efficiency of feature rendering on the cloud side, and reducing the rendering delay of game screens with ray tracing features in scenarios with massive users.

[0373] As shown in FIG. 10 , the cloud may include a control server 400 and rendering servers 401 to 403 .

[0374] Among them, the rendering server 401 can be used to perform rendering and compression processes on the characteristics of direct lighting, the rendering server 402 can be used to perform rendering and compression processes on the characteristics of planar reflection, and the rendering server 403 can be used to perform rendering and compression processes on the characteristics of shadows.

[0375] The control server 400 can be used to synchronize frame states with each client and calculate physics or animation in combination with the synchronized frame states, so as to control the rendering servers 401 to 403 to respectively perform rendering and compression processes of corresponding characteristics.

[0376] The processing process of each server in Figure 10 is basically the same as the processing process of the server 200 in Figure 7. The only difference is that the workflow of the server 200 in Figure 7 is executed by the four servers in Figure 10 to implement the overall function of the server 200 shown in Figure 7.

[0377] As shown in FIG10 , the control server 400 and each rendering server can be connected and communicated with each other through a network. In this way, the control server 400 can control each rendering server to render features respectively, and each rendering server can also communicate with each other to improve the rendering efficiency of features.

[0378] 11 , the following describes a process of developing a game instance using the apparatus 300 shown in FIG. 1 to obtain the client 100 and the server 200 , respectively. In this way, the client 100 and the server 200 can execute the processes of the various method embodiments described above.

[0379] As shown in FIG11 , the client 100 developed using the apparatus 300 shown in FIG1 (e.g., as a plug-in) of the present application can run the complete game logic and have the basic rendering capabilities of the feature. Furthermore, the server 200 developed using the apparatus 300 shown in FIG1 of the present application can run the basic game logic and have the complete rendering capabilities of the feature.

[0380] In combination with the system architecture shown in FIG1 , the implementation process of the development state of the method of the present application will be described below with reference to FIG11 .

[0381] As shown in Figure 11, existing game engines can provide game developers with complete game development functions such as rendering pipelines, game logic, etc. For example, Figure 11 shows a game logic module and a rendering pipeline module.

[0382] FIG11 schematically illustrates only the coordination module and rendering module in apparatus 300, highlighting the differences between the rendering module and the coordination module between client 100 and server 200. It should be understood that when game developers use apparatus 300 to expand the functionality of an existing game engine, apparatus 300 also includes the encoding module, decoding module, and transmission module shown in FIG1 . FIG11 does not illustrate the encoding module, decoding module, and transmission module of apparatus 300 in the expanded game engine.

[0383] Among them, the encoding module and decoding module in the device 300 can provide specific software and hardware encoding and decoding algorithm functions for data sources with different characteristics, so as to reduce the volume of intermediate results sent from the cloud side and reduce the impact of network bandwidth on the user experience.

[0384] The transmission module in the device 300 can establish a stable, low-latency network transmission channel, and perform multi-stream adaptive control based on the provided compressed data and type, and transmit the compression results in the cloud to the end side, such as the client 100, in the form of a code stream; and upload the client's frame status to the server.

[0385] Game developers can use the plug-in interface (e.g., rendering settings interface) provided by the collaborative module in the device 300 of the present application to configure the functions of the existing game engine. For example, developers call the plug-in interface provided by the collaborative module to interact with various modules in the device 300 and various modules in the existing game engine to configure modules such as the rendering module, game logic module, and rendering pipeline module.

[0386] The plug-in interface provided by the collaboration module may include but is not limited to: an encoding interface for calling an encoding module, a decoding interface for calling a decoding module, a network transmission interface for calling a transmission module, and a rendering setting interface.

[0387] Among them, in the running state, the encoding module in the functional layer can be called through the encoding interface provided by the collaborative module to encode the intermediate rendering results of the feature.

[0388] In the running state, the decoding module in the functional layer can be called through the decoding interface provided by the collaborative module to decode the characteristic code stream.

[0389] In the running state, the transmission module in the functional layer can be called through the network transmission interface provided by the collaboration module to perform network transmission of the encoded characteristic code stream between the mobile terminal and the cloud.

[0390] In the development state, the rendering setting interface provided by the collaborative module can provide game developers with a series of interactive feature plug-ins for configuring the feature functions of the rendering module in the functional layer.

[0391] The functional layer can provide rendering enhancement services in a manner compatible with existing game engines through the above-mentioned plug-in interface provided by the collaborative module.

[0392] For example, the collaboration module may provide a feature setting interface to the game developer, and the game developer may input in the feature setting interface to call the rendering setting interface to configure the feature functions of the rendering module; wherein the rendering module may provide a feature rendering setting function.

[0393] Among them, the process of directly configuring the rendering module is relatively complicated. The game developer of this application can configure the rendering module by calling the rendering setting interface provided by the collaborative module to perform configuration operations such as switching features, which reduces the complexity of the configuration process of the rendering module.

[0394] In addition, the rendering pipeline module can provide a rendering process control development interface, and the rendering module can embed the feature functions of feature rendering into the rendering pipeline module in a customized or compatible manner.

[0395] In addition, the game logic module can provide a game logic development interface, so the collaboration module can extract the required game state from the game logic module and provide the game scene to the rendering pipeline module through the game logic module.

[0396] The following focuses on the differences between the development process of FIG11 , when developing the client 100 and the server 200:

[0397] When developing the client 100, the game developer can set the characteristic state of the rendering module through the collaboration module. The characteristic state is the characteristics of rendering supported by the rendering module, such as supporting direct lighting, shadows and other characteristics.

[0398] When developing the client 100 , the game developer may customize the execution logic of the collaborative module to generate control information (such as the above-mentioned frame status).

[0399] Furthermore, the rendering pipeline module configured by the rendering module can have independent rendering processes for each feature supported by the rendering module. However, the intermediate rendering results for that feature used in each independent rendering process are preset intermediate rendering results. In this way, the game pipeline on the client side only includes the basic game pipeline for the corresponding feature.

[0400] Therefore, when client 100 renders the above features independently without coordination with server 200, the visual effects of each frame remain the same, and do not change in real time with changes in the game scene. For example, the direct lighting effect on objects in each frame remains the same. In this embodiment, the rendering pipeline module in client 100 only guarantees the most basic rendering effects for each feature to minimize the load on the client side.

[0401] Furthermore, the rendering module in client 100 may have a feature injection function. Specifically, the rendering module in client 100 may replace the preset intermediate rendering result of the corresponding feature based on the intermediate rendering result of the corresponding feature sent by server 200, and then use the replaced intermediate rendering result to perform the rendering calculation of the feature to obtain the target rendering result of the feature. In this way, client 100 can coordinate with server 200 to ensure that the effect of the feature supported by the client in each frame of the game screen is updated in real time as the game scene changes, thereby enhancing the display effect of the feature.

[0402] The collaborative module in the device 300 may have the collaborative functions of the collaborative module 103a and the collaborative module 103b shown in Figure 3. When packaging and publishing the client 100, the functions of the collaborative module 103a in the device 300, as well as the configured rendering module 101a, decoding module 102a, transmission module 104a, rendering pipeline module 105a, and game logic module 106a may be packaged and published to obtain the client 100 shown in Figure 3.

[0403] When developing the server 200, game developers can use the collaboration module to configure the rendering module with various rendering and calculation functions to obtain high-quality intermediate rendering results of corresponding characteristics. Furthermore, game developers can use the collaboration module to configure the rendering module to share and cache the intermediate rendering results, as well as configure the rendering module to extract intermediate rendering results of corresponding characteristics based on control information from the collaboration module.

[0404] In addition, the rendering pipeline module configured by the rendering module can have the complete rendering capabilities of each feature supported by the rendering module (for example, high-quality intermediate rendering results of the corresponding features can be obtained), thereby extracting the calculation process of the intermediate rendering results of complex rendering features from the rendering process in the rendering pipeline on the end side, and realizing the transfer of computing power from the end side to the cloud side.

[0405] When packaging and publishing the client 100, the functions of the collaboration module 103b in the device 300, as well as the configured rendering module 101b, decoding module 102b, transmission module 104b, rendering pipeline module 105b, and game logic module 106b can be packaged and published to obtain the server 200 shown in Figure 3.

[0406] The present application can separate the computing process from the end-cloud based on the characteristics, and can offload the high-load characteristic calculation to the cloud, calculate the intermediate rendering result of the characteristic, and the end side can realize the effect rendering of the high-order characteristics of the game interface through simple calculation based on the intermediate rendering result sent from the cloud, thereby improving the drawing experience. And the client can synchronize the characteristic rendering requirements to the server, and the server can install the characteristic rendering requirements to perform characteristic rendering to obtain the intermediate rendering results of the characteristics that meet the client's characteristic rendering requirements, so that the client can control the calculation and processing process of the characteristics of the server in real time, such as turning on or off the rendering of the characteristics, controlling the resolution and compression rate of the intermediate rendering results of the characteristics, etc. And the client can replace the rendering resources used in its own characteristic rendering process (such as the preset intermediate rendering results) with the intermediate rendering results of the characteristics sent by the server to perform characteristic rendering, so that the characteristic effects in the game screen displayed by the client can be updated in real time as the game scene is updated.

[0407] In one possible implementation, an embodiment of the present application provides an image rendering device. The image rendering device includes a receiving module for receiving first control information of a first image from a first client; a control module for controlling the calculation and processing of m visual characteristics based on the first control information, respectively, to obtain m first intermediate rendering results related to the m visual characteristics of the first image, where m ≥ 2; and sending the m first intermediate rendering results to the first client; wherein the m first intermediate rendering results are used to provide the first client with rendering resources for rendering the first image.

[0408] In a possible implementation, the number q of visual characteristics supported for calculation and processing by the image rendering apparatus is greater than or equal to m, and the m visual characteristics are visual characteristics for which rendering is initiated in accordance with the first control information.

[0409] In one possible embodiment, the image rendering device is configured with q calculation processing modules that perform calculation processing on q visual characteristics respectively, where q≥m; the control module is used to call m calculation processing modules corresponding to the m visual characteristics among the q calculation processing modules, and perform calculation processing on the m visual characteristics respectively according to the first control information to obtain m first intermediate rendering results of the first image related to the m visual characteristics.

[0410] In a possible embodiment, the receiving module is further used to receive second control information of the second image from the second client; the control module is further used to control the calculation and processing processes of n visual characteristics respectively according to the second control information, to obtain n second intermediate rendering results of the second image related to the n visual characteristics, q≥n≥1, wherein the n visual characteristics are the visual characteristics indicated by the second control information to start rendering; the sending module is further used to send the n second intermediate rendering results to the second client; wherein the n second intermediate rendering results are used to provide the second client with rendering resources for rendering the second image.

[0411] In a possible embodiment, the control module is further used to call n computing and processing modules among the q computing and processing modules corresponding to the n visual characteristics, and perform computing and processing on the n visual characteristics respectively according to the second control information to obtain n second intermediate rendering results of the second image related to the n visual characteristics.

[0412] In a possible implementation, the control module is further used to determine m original intermediate results obtained by respectively calculating the m visual characteristics; and based on the first control information, process the m original intermediate results respectively to obtain the m first intermediate rendering results of the m visual characteristics of the first image, wherein the data volume of the m first intermediate rendering results is less than the data volume of the m original intermediate results.

[0413] In one possible implementation, the first control information includes first indication information indicating an extraction strategy for the intermediate rendering results of each of the m visual characteristics. The control module is further configured to extract m groups of partial intermediate results from the m original intermediate results based on the first indication information to obtain the m first intermediate rendering results of the m visual characteristics.

[0414] In one possible embodiment, the first control information includes third indication information indicating the encoding strategy of the intermediate rendering results of each of the m visual characteristics. The control module is further used to encode the m original intermediate results or the m groups of partial intermediate results extracted from the m original intermediate results according to the third indication information to obtain the m first intermediate rendering results of the m visual characteristics.

[0415] In one possible implementation, the control module is further configured to determine n original intermediate results obtained by respectively calculating the n visual characteristics; and based on the second control information, process the n original intermediate results respectively to obtain the n second intermediate rendering results of the n visual characteristics of the second image, wherein the data volume of the n second intermediate rendering results is smaller than the data volume of the n original intermediate results.

[0416] In one possible implementation, the second control information includes second indication information indicating an extraction strategy for the intermediate rendering results of each of the n visual characteristics, and the control module is further configured to extract n groups of partial intermediate results from the n original intermediate results based on the second indication information to obtain the n second intermediate rendering results of the n visual characteristics.

[0417] In one possible implementation, the second control information includes fourth indication information indicating an encoding strategy for the intermediate rendering results of each of the n visual characteristics. The control module is further configured to encode the n original intermediate results or the n groups of partial intermediate results extracted from the n original intermediate results, respectively, based on the fourth indication information, to obtain the n second intermediate rendering results of the n visual characteristics.

[0418] In a possible implementation, the control module is further used to determine k visual characteristics based on the first control information and the second control information, wherein the k visual characteristics are the union of the m visual characteristics and the n visual characteristics, m≤k, n≤k, and q≥k; the image rendering device calculates the k visual characteristics respectively to obtain k original intermediate results of the k visual characteristics; and determines m original intermediate results among the k original intermediate results that correspond to the m visual characteristics respectively.

[0419] In a possible implementation, the control module is further configured to determine n original intermediate results, among the k original intermediate results, that respectively correspond to the n visual characteristics.

[0420] In a possible implementation, the control module is further configured to respectively calculate the m visual characteristics for which the first control information indicates to initiate rendering, to obtain m original intermediate results of each of the m visual characteristics.

[0421] In a possible implementation, the control module is further configured to determine, among the n visual characteristics for which the second control information indicates to initiate rendering, p visual characteristics that are different from the m visual characteristics, 1≤p<n; calculate the p visual characteristics separately to obtain p original intermediate results for each of the p visual characteristics; and determine, among the m original intermediate results, i original intermediate results corresponding to the i visual characteristics respectively; wherein the i visual characteristics are the same visual characteristics between the m visual characteristics and the n visual characteristics, i+p=n.

[0422] In a possible implementation, the first image and the second image are rendered images of the same scene from different perspectives.

[0423] In a possible implementation, the visual characteristics include at least one of the following: direct lighting, reflected lighting, shadow, indirect lighting, spherical reflection, and ambient occlusion.

[0424] The effects of the image rendering methods in the above embodiments are similar to the effects of the image rendering methods executed by the server in the above embodiments, and are not described in detail here.

[0425] In one possible embodiment, the present application provides an image rendering device. The device includes: a sending module for sending first control information of a first image to a server, wherein the first control information is used to provide the server with information for controlling the computational processing of each of m visual characteristics, where m ≥ 2; a receiving module for receiving, from the server, m first intermediate rendering results in response to the first control information; and a processing module for obtaining a first rendering result of the first image based on the m first intermediate rendering results of the m visual characteristics.

[0426] In a possible implementation, the sending module is further configured to send the first control information of the first image to the server when it is determined that the scene is updated and / or the camera posture is updated.

[0427] In one possible embodiment, the sending module is further used to send control information indicating the start of rendering of m visual characteristics; and / or, to send control information indicating an extraction strategy for the intermediate rendering results of each of the m visual characteristics; and / or, to send control information indicating an encoding strategy for the intermediate rendering results of each of the m visual characteristics.

[0428] In a possible embodiment, the device also includes: an acquisition module for acquiring target environment information; wherein the target environment information includes at least one of the network environment information of the client, the scene information of the first image, and user input information; and a determination module for determining the first control information of the first image based on the target environment information.

[0429] In a possible implementation, the m visual characteristics include at least one of the following: direct lighting, reflected lighting, shadow, indirect lighting, spherical reflection, and ambient occlusion.

[0430] The effects of the image rendering methods in the above embodiments are similar to the effects of the image rendering methods executed by the client in the above embodiments, and are not described in detail here.

[0431] The following describes a device provided in an embodiment of the present application. As shown in FIG12 :

[0432] FIG12 is a schematic diagram of the structure of an image rendering device provided by an embodiment of the present application. As shown in FIG12 , the device 500 may include: a processor 501 , a transceiver 505 , and optionally a memory 502 .

[0433] The transceiver 505 may be referred to as a transceiver unit, a transceiver, or a transceiver circuit, etc., and is configured to implement transceiver functions. The transceiver 505 may include a receiver and a transmitter. The receiver may be referred to as a receiver or a receiving circuit, etc., and is configured to implement a receiving function; the transmitter may be referred to as a transmitter or a transmitting circuit, etc., and is configured to implement a transmitting function.

[0434] The memory 502 may store a computer program or software code or instruction 504, which may also be referred to as firmware. The processor 501 may implement the image rendering method provided in each embodiment of the present application by running the computer program or software code or instruction 503 therein, or by calling the computer program or software code or instruction 504 stored in the memory 502. The processor 501 may be a central processing unit (CPU), and the memory 502 may be, for example, a read-only memory (ROM) or a random access memory (RAM).

[0435] The processor 501 and transceiver 505 described in this application can be implemented on an integrated circuit (IC), an analog IC, a radio frequency integrated circuit RFIC, a mixed signal IC, an application specific integrated circuit (ASIC), a printed circuit board (PCB), an electronic device, etc.

[0436] The above-mentioned device 500 may further include an antenna 506. The modules included in the device 500 are only for illustration and are not limited in this application.

[0437] For example, the structure of the image rendering device may not be limited to FIG12. The image rendering device may be an independent device or may be part of a larger device. For example, the image rendering device may be implemented as follows:

[0438] (1) An independent integrated circuit IC, or chip, or chip system or subsystem; (2) A collection of one or more ICs, optionally including a storage component for storing data and instructions; (3) A module that can be embedded in other devices; (4) In-vehicle equipment, etc.; (5) Others, etc.

[0439] If the image rendering device is implemented as a chip or a chip system, please refer to the chip structure diagram shown in Figure 13. The chip shown in Figure 13 includes a processor 601 and an interface 602. The number of processors 601 can be one or more, and the number of interfaces 602 can be multiple. Optionally, the chip or chip system can include a memory 603.

[0440] Among them, all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module and will not be repeated here.

[0441] Based on the same technical concept, an embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. The computer program includes at least one section of code, and the at least one section of code can be executed by a computer to control the computer to implement the above method embodiment.

[0442] Based on the same technical concept, an embodiment of the present application also provides a computer program, which, when executed, is used to implement the above method embodiment.

[0443] The program may be stored in whole or in part on a storage medium packaged with the processor, or may be stored in whole or in part on a memory not packaged with the processor.

[0444] Based on the same technical concept, an embodiment of the present application further provides a chip including a processor. The processor can implement the above method embodiment.

[0445] The steps of the method or algorithm described in conjunction with the disclosure of the embodiments of the present application can be implemented in a hardware manner, or can be implemented by a processor executing a software instruction. The software instruction can be composed of corresponding software modules, and the software module can be stored in a random access memory (Random Access Memory, RAM), a flash memory, a read-only memory (Read Only Memory, ROM), an erasable programmable read-only memory (Erasable Programmable ROM, EPROM), an electrically erasable programmable read-only memory (Electrically EPROM, EEPROM), a register, a hard disk, a mobile hard disk, a read-only compact disc (CD-ROM) or any other form of storage medium well known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and can write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0446] Those skilled in the art will appreciate that in one or more of the above examples, the functions described in the embodiments of the present application can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any media that facilitates the transmission of computer programs from one place to another. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0447] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. A terminal-cloud collaborative system, characterized in that: The end-cloud collaborative system includes at least one client and a server in communication connection, wherein the at least one client includes a first client; The first client is used to send first control information of a first image to the server; The server is configured to control the calculation and processing processes of the m visual characteristics respectively according to the first control information, obtain m first intermediate rendering results related to the m visual characteristics of the first image, where m≥2; and send the m first intermediate rendering results to the first client; The first client is further used to receive the m first intermediate rendering results, and obtain a first rendering result of the first image based on the m first intermediate rendering results.

2. The system according to claim 1, characterized in that The number q of visual characteristics supported by the server for calculation and processing is greater than or equal to m, and the m visual characteristics are visual characteristics for which the first control information indicates to start rendering.

3. The system according to claim 1 or 2, characterized in that: The server is configured with q calculation processing modules for respectively calculating and processing q visual characteristics, q≥m; When the server controls the calculation and processing processes of the m visual characteristics respectively according to the first control information, the server is specifically configured to: Call m computing and processing modules corresponding to the m visual characteristics among the q computing and processing modules, perform computing and processing on the m visual characteristics respectively according to the first control information, and obtain m first intermediate rendering results of the first image related to the m visual characteristics.

4. The system according to claim 2 or 3, characterized in that: The at least one client further comprises a second client; The second client is used to send second control information of a second image to the server; The server is further configured to control the calculation and processing processes of the n visual characteristics respectively according to the second control information, obtain n second intermediate rendering results of the second image related to the n visual characteristics, q≥n≥1, wherein the n visual characteristics are the visual characteristics for which the second control information indicates to start rendering; and send the n second intermediate rendering results to the second client; The second client is further used to receive the n second intermediate rendering results, and obtain a second rendering result of the second image based on the n second intermediate rendering results.

5. The system according to claim 4, characterized in that When the server controls the calculation and processing processes of the n visual characteristics respectively according to the second control information to obtain n second intermediate rendering results of the second image related to the n visual characteristics, the server is specifically configured to: Calling n computing and processing modules corresponding to the n visual characteristics among the q computing and processing modules, performing computing and processing on the n visual characteristics respectively according to the second control information, and obtaining n second intermediate rendering results of the second image related to the n visual characteristics.

6. The system according to any one of claims 1 to 5, characterized in that: When the server controls the calculation and processing processes of the m visual characteristics respectively according to the first control information to obtain m first intermediate rendering results of the first image related to the m visual characteristics, the server is specifically configured to: Determine m original intermediate results obtained by respectively calculating the m visual characteristics; According to the first control information, the m original intermediate results are processed respectively to obtain the m first intermediate rendering results of the m visual characteristics of the first image, wherein the data volume of the m first intermediate rendering results is smaller than the data volume of the m original intermediate results.

7. The system according to claim 6, characterized in that The first control information includes first indication information indicating an extraction strategy of the intermediate rendering results of the m visual characteristics. When the server processes the m original intermediate results respectively according to the first control information to obtain the m first intermediate rendering results of the m visual characteristics of the first image, the server is specifically configured to: According to the first indication information, m groups of partial intermediate results are respectively extracted from the m original intermediate results to obtain the m first intermediate rendering results of the m visual characteristics.

8. The system according to claim 6 or 7, characterized in that: The first control information includes third indication information indicating a coding strategy of the intermediate rendering results of the m visual characteristics. When the server processes the m original intermediate results respectively according to the first control information to obtain the m first intermediate rendering results of the m visual characteristics of the first image, the server is specifically configured to: According to the third indication information, the m original intermediate results or the m groups of partial intermediate results extracted from the m original intermediate results are encoded respectively to obtain the m first intermediate rendering results of the m visual characteristics.

9. The system according to any one of claims 4 to 8, characterized in that: When the server controls the calculation and processing processes of the n visual characteristics respectively according to the second control information to obtain n second intermediate rendering results of the second image related to the n visual characteristics, the server is specifically configured to: Determine n original intermediate results obtained by respectively calculating the n visual characteristics; According to the second control information, the n original intermediate results are processed respectively to obtain the n second intermediate rendering results of the n visual characteristics of the second image, wherein the data volume of the n second intermediate rendering results is less than the data volume of the n original intermediate results.

10. The system according to claim 9, characterized in that The second control information includes second indication information indicating an extraction strategy of the intermediate rendering results of the n visual characteristics. When the server processes the n original intermediate results respectively according to the second control information to obtain the n second intermediate rendering results of the n visual characteristics of the second image, the server is specifically configured to: According to the second indication information, n groups of partial intermediate results are respectively extracted from the n original intermediate results to obtain the n second intermediate rendering results of the n visual characteristics.

11. The system according to claim 9 or 10, characterized in that: The second control information includes fourth indication information indicating an encoding strategy of the intermediate rendering results of the n visual characteristics. When the server processes the n original intermediate results respectively according to the second control information to obtain the n second intermediate rendering results of the n visual characteristics of the second image, the server is specifically configured to: According to the fourth indication information, the n original intermediate results or the n groups of partial intermediate results extracted from the n original intermediate results are encoded respectively to obtain the n second intermediate rendering results of the n visual characteristics.

12. The system according to any one of claims 9 to 11, characterized in that: When the server determines the m original intermediate results obtained by respectively calculating the m visual characteristics, and when determining the n original intermediate results obtained by respectively calculating the n visual characteristics, the server is specifically configured to: Determine k visual characteristics according to the first control information and the second control information, wherein the k visual characteristics are a union of the m visual characteristics and the n visual characteristics, m≤k, n≤k, and q≥k; Calculating the k visual characteristics respectively to obtain k original intermediate results of the k visual characteristics; Determine m original intermediate results corresponding to the m visual characteristics respectively among the k original intermediate results; Determine n original intermediate results among the k original intermediate results that respectively correspond to the n visual characteristics.

13. The system according to any one of claims 9 to 11, characterized in that: When the server determines the m original intermediate results obtained by respectively calculating the m visual characteristics, the server is specifically configured to: The m visual characteristics indicated by the first control information to start rendering are calculated respectively to obtain m original intermediate results of the m visual characteristics respectively.

14. The system according to claim 13, characterized in that When the server determines the n original intermediate results obtained by respectively calculating the n visual characteristics, the server is specifically configured to: Determine p visual characteristics that are different from the m visual characteristics among the n visual characteristics indicated by the second control information to start rendering, 1≤p<n; Calculating the p visual characteristics respectively to obtain p original intermediate results of the p visual characteristics; Determining i original intermediate results corresponding to the i visual characteristics respectively from the m original intermediate results; The i visual characteristics are the same visual characteristics between the m visual characteristics and the n visual characteristics, i+p=n.

15. The system according to any one of claims 4 to 14, characterized in that The first image and the second image are rendered images of the same scene from different perspectives.

16. The system according to any one of claims 1 to 15, characterized in that The visual characteristics include at least one of the following: Direct lighting, bounce lighting, shadows, indirect lighting, spherical reflections, ambient occlusion.

17. An image rendering method, characterized in that: The method comprises: The server receives first control information of a first image from a first client; The server controls the calculation and processing processes of the m visual characteristics respectively according to the first control information to obtain m first intermediate rendering results related to the m visual characteristics of the first image, where m≥2; and sends the m first intermediate rendering results to the first client; The m first intermediate rendering results are used to provide rendering resources for rendering the first image to the first client.

18. The method according to claim 17, characterized in that The number q of visual characteristics supported by the server for calculation and processing is greater than or equal to m, and the m visual characteristics are visual characteristics for which the first control information indicates to start rendering.

19. The method according to claim 17 or 18, characterized in that The server is configured with q calculation processing modules for respectively calculating and processing q visual characteristics, q≥m; The server controls the calculation and processing processes of the m visual characteristics respectively according to the first control information, including: The server calls m computing and processing modules among the q computing and processing modules corresponding to the m visual characteristics, and performs computing and processing on the m visual characteristics respectively according to the first control information, to obtain m first intermediate rendering results of the first image related to the m visual characteristics.

20. The method according to claim 18 or 19, characterized in that The method further comprises: The server receives second control information of a second image from a second client; The server controls the calculation and processing processes of the n visual characteristics respectively according to the second control information, obtains n second intermediate rendering results of the second image related to the n visual characteristics, q≥n≥1, wherein the n visual characteristics are the visual characteristics for which the second control information indicates to start rendering; and sends the n second intermediate rendering results to the second client; The n second intermediate rendering results are used to provide rendering resources for rendering the second image to the second client.

21. An image rendering method, characterized in that: The method comprises: The client sends first control information of the first image to the server, wherein the first control information is used to provide the server with information for controlling a calculation process of each of m visual characteristics, where m≥2; The client receives m first intermediate rendering results in response to the first control information from the server; The client obtains a first rendering result of the first image based on the m first intermediate rendering results of the m visual characteristics.

22. The method according to claim 21, characterized in that The client sends first control information of a first image to the server, including: When determining that the scene is updated and / or the camera posture is updated, the client sends first control information of the first image to the server.

23. The method according to claim 21 or 22, characterized in that The client sends first control information of a first image to the server, including: The client sends control information of m visual characteristics indicating starting rendering to the server; and / or, The client sends control information indicating an extraction strategy of the intermediate rendering results of the m visual characteristics to the server; and / or, The client sends control information indicating an encoding strategy of an intermediate rendering result of each of the m visual characteristics to the server.

24. An image rendering method, characterized in that: The method comprises: The control module receives first control information of a first image from a first client; The control module outputs first information according to the first control information, wherein the first information is used to instruct the rendering module to start rendering for the m visual characteristics; The control module receives m original intermediate results from the rendering module, wherein the m original intermediate results are respective characteristic calculation results obtained by the rendering module respectively calculating the m visual characteristics; The control module processes the m original intermediate results respectively according to the first control information to obtain m first intermediate rendering results of the m visual characteristics of the first image, wherein the data volume of the m first intermediate rendering results is less than the data volume of the m original intermediate results, and m≥2; The m first intermediate rendering results are used to provide rendering resources for rendering the first image to the first client.

25. The method according to claim 24, characterized in that The number q of visual characteristics supported by the control module for calculation and processing is greater than or equal to m, and the m visual characteristics are visual characteristics for which the first control information indicates to start rendering.

26. The method according to claim 24 or 25, characterized in that The control module is configured with q sub-control modules for processing q visual characteristics respectively, q ≥ m; The control module processes the m original intermediate results respectively according to the first control information to obtain m first intermediate rendering results of the m visual characteristics of the first image, including: The m sub-control modules corresponding to the m visual characteristics among the q sub-control modules respectively process the m original intermediate results according to the first control information to obtain m first intermediate rendering results of the m visual characteristics of the first image.

27. An image rendering method, characterized in that: The method comprises: The rendering module receives first information from the control module, where the first information is used to instruct the rendering module to start rendering for m visual features; The rendering module calculates m visual characteristics respectively according to the first information to obtain m original intermediate results, where m≥2; The rendering module sends the m original intermediate results to the control module.

28. The method according to claim 27, characterized in that The number q of visual features supported for rendering by the rendering module is greater than or equal to m.

29. The method according to claim 27 or 28, characterized in that The rendering module is configured with q sub-rendering modules for respectively calculating and processing q visual characteristics, q ≥ m; The rendering module calculates the m visual characteristics respectively according to the first information to obtain m original intermediate results, including: The m sub-rendering modules corresponding to the m visual characteristics among the q sub-rendering modules respectively calculate the m visual characteristics to obtain m original intermediate results of the first image related to the m visual characteristics.

30. An image rendering device, characterized in that: The image rendering device comprises: A receiving module, configured to receive first control information of a first image; A control module, configured to control the calculation and processing processes of the m visual characteristics respectively according to the first control information, to obtain m first intermediate rendering results of the first image related to the m visual characteristics, where m≥2; and to send the m first intermediate rendering results to the first client; The m first intermediate rendering results are used to provide rendering resources for rendering the first image to the first client.

31. An image rendering device, characterized in that: The image rendering device comprises: A sending module, used for sending first control information of the first image to a server, wherein the first control information is used to provide the server with information for controlling a calculation process of each of m visual characteristics, where m≥2; A receiving module, configured to receive, from the server, m first intermediate rendering results in response to the first control information; A processing module is used to obtain a first rendering result of the first image based on the m first intermediate rendering results of the m visual characteristics.

32. A computer-readable storage medium, characterized in that: Comprising a computer program which, when run on a computer or a processor, enables the computer or the processor to execute a method as described in any one of claims 17 to 20, or any one of claims 21 to 23, or any one of claims 24 to 26, or any one of claims 27 to 29.

33. An image rendering device, characterized in that: It comprises one or more interface circuits and one or more processors; the interface circuit is used to receive a signal from a memory and send the signal to the processor, the signal including a computer instruction stored in the memory; when the processor executes the computer instruction, the processor is used to execute the method as claimed in any one of claims 17 to 20, or any one of claims 21 to 23, or any one of claims 24 to 26, or any one of claims 27 to 29.

34. A computer program product, characterized in that The computer program product comprises a software program, which, when executed by a computer or a processor, enables the steps of the method described in any one of claims 17 to 20 to be executed, the steps of the method described in any one of claims 21 to 23 to be executed, or the steps of the method described in any one of claims 24 to 26, or the steps of the method described in any one of claims 27 to 29 to be executed.

35. An image rendering device, characterized in that: The image rendering device includes a processor, which is used to execute instructions. When the instructions are executed by the processor, the processor is used to execute the method described in any one of claims 17 to 20, or any one of 21 to 23, or any one of 24 to 26, or any one of 27 to 29.

36. A chip, characterized in that: The chip comprises: a processor and a communication interface, wherein the processor is used to call and run instructions from the communication interface, and when the processor executes the instructions, the method according to any one of claims 17 to 20 is implemented.

37. A chip, characterized in that: The chip includes: a processor and a communication interface, the processor is used to call and run instructions from the communication interface, and when the processor executes the instructions, the method according to any one of claims 21 to 23 is implemented.

38. A chip, characterized in that: The chip includes: a processor and a communication interface, the processor is used to call and run instructions from the communication interface, and when the processor executes the instructions, the method according to any one of claims 24 to 26 is implemented.

39. A chip, characterized in that: The chip includes: a processor and a communication interface, the processor is used to call and run instructions from the communication interface, and when the processor executes the instructions, the method according to any one of claims 27 to 29 is implemented.

Images