End-cloud collaborative rendering method and related device

Abstract

The invention discloses an end-cloud collaborative rendering method, and the method comprises the steps: carrying out the basic rendering of a first end-side scene through electronic equipment, and obtaining first basic rendering data; the electronic equipment sends a scene identifier of the first end side scene and a first high-order rendering type to a server; the server stores a first cloud side scene converted from the first end side scene for the first high-order rendering type; the electronic equipment sends the first state data to a server; after updating the first cloud side scene according to the first state data, the server executes preprocessing of a first high-order rendering type on the first cloud side scene to obtain first preprocessing data; the server sends the first preprocessing data to the electronic equipment; and the electronic equipment obtains a first image according to the first basic rendering data and the first preprocessing data, and displays the first image. Therefore, the computing power requirement and the load of the end-side GPU are reduced, the electronic equipment with relatively weak computing power of the GPU can also present a high-order rendering effect, and the user experience is effectively improved.

Description

Technical Field

[0001] The present application relates to the field of electronic technology, and in particular to a rendering method and related devices for end-cloud collaboration. Background Art

[0002] There are more and more three-dimensional (3D) applications (such as 3D games, 3D modeling software, etc.) on terminal devices such as mobile phones. Providing rendering capabilities for the above 3D applications is the most core function of terminal devices to support the operation of 3D applications. However, the graphics processing unit (GPU) (referred to as lightweight GPU for ease of description) of terminal devices (such as mobile devices) with limited size and power consumption is quite different from that of personal computer (PC)-level GPUs. For example, the single-precision computing power of Apple A15 in 2021 is 1.5T FLOPS; the single-precision computing power of NVIDA GTX 580 in 2010 has reached 1.58T FLOPS; and the computing power of NVIDIARTX 3080Ti in 2021 has reached 27.955TFLOPS. It can be seen that the computing power development of lightweight GPUs for mobile devices is about 10 years behind that of PC-level GPUs.

[0003] The GPU of the cloud-side server (i.e., PC-level GPU) has a huge computing power advantage over the lightweight GPU of mobile devices. With the development of cloud computing technology, transferring the computing power of the terminal side to the cloud side and coordinating the cloud side to process the business of the terminal side has gradually become a direction of technological evolution. At present, how to coordinate with the cloud side to achieve efficient rendering processing, so as to achieve the rendering effect of ray tracing technology on the terminal side, remains to be studied. Summary of the invention

[0004] The present application provides an end-cloud collaborative rendering method and related devices, which reduce the computing power requirements and load on the end-side GPU, and enable electronic devices with weaker GPU computing power to present high-level rendering effects, such as the rendering effects of ray tracing technology, effectively improving the user experience.

[0005] In a first aspect, the present application provides a rendering method for end-cloud collaboration, which is applied to a communication system, wherein the communication system includes a first electronic device and a server, wherein the first electronic device stores at least one end-side scene of a first application, the first end-side scene being any one of the at least one end-side scene, and the server stores a first cloud-side scene converted from the first end-side scene for a first high-order rendering type; the method includes: the first electronic device performs basic rendering on the first end-side scene to obtain first basic rendering data; the first electronic device sends a scene identifier and a first high-order rendering type of the first end-side scene to the server; the first electronic device sends first status data to the server, the first status data including data related to preprocessing of the first high-order rendering type in the status data of the first end-side scene; the server updates the first cloud-side scene according to the first status data, performs preprocessing of the first high-order rendering type on the updated first cloud-side scene, and obtains first preprocessing data; the server sends the first preprocessing data to the first electronic device; the first electronic device obtains a first image according to the first basic rendering data and the first preprocessing data, and the first image has the light and shadow effects of the first high-order rendering type; the first electronic device displays the first image.

[0006] In implementing the embodiment of the present application, a terminal-side scene is converted into a cloud-side scene for each high-order rendering type, and the server stores the cloud-side scene; for a specific high-order rendering type, the electronic device only needs to have the ability to perform basic rendering on the terminal-side scene, and the server implements high-order rendering preprocessing of the cloud-side scene corresponding to the terminal-side scene; after the basic rendering data is integrated with the preprocessing data of the high-order rendering, the final rendered image presents the light and shadow effects of the above-mentioned specific high-order rendering type. In this way, for various high-order rendering types, the computing power requirements and load on the terminal-side GPU can be reduced; electronic devices (such as mobile phones, tablets, etc.) do not need to have high-standard hardware performance to present the rendering effects of high-order rendering. In addition, since the server only performs preprocessing of high-order rendering and does not run the full APP, the consumption of cloud-side computing resources is reduced.

[0007] In one implementation, the method further includes: the first electronic device updates the scene information of the first end-side scene; the first electronic device performs basic rendering on the updated first end-side scene to obtain second basic rendering data; when the network speed between the first electronic device and the server is lower than a preset value, or the network between the first electronic device and the server is disconnected, the first electronic device determines the second image according to the second basic rendering data; the first electronic device displays the second image. By implementing the embodiment of the present application, when the network conditions are not good, the electronic device can still perform basic rendering, which can ensure the normal operation of the game APP, thereby ensuring the basic operation experience of the user.

[0008] In one implementation, the first electronic device sends a scene identifier and a first high-order rendering type of a first end-side scene to a server, including: the first electronic device sends a first request to the server, the first request includes the scene identifier and the first high-order rendering type of the first end-side scene; before the first electronic device sends the first status data to the server, the method also includes: based on the first request, the server performs preprocessing preparation, the preprocessing preparation includes loading the first cloud-side scene, and preparing a preprocessing rendering pipeline for the first high-order rendering type; the server sends confirmation information to the first electronic device, the confirmation information is used to indicate that the preprocessing preparation has been completed; the first electronic device sends the first status data to the server, including: based on the confirmation information, the first electronic device sends the first status data to the server. In implementing the embodiment of the present application, the status data is uploaded to the server only after confirming that the server has completed the preprocessing preparation; to avoid invalid upload of status data and waste of network traffic caused by the inability of the server to perform high-order rendering.

[0009] In one implementation, the state data of the scene on the first terminal side includes part or all of the following: scene identification information, light source information, character information, camera information, and scene update information.

[0010] In one implementation, the server stores a first correspondence between a first terminal-side scene, a first high-order rendering type, and a first cloud-side scene. Before the server updates the first cloud-side scene according to the first status data, the method further includes: the server determines that the cloud-side scene to be rendered is the first cloud-side scene based on the first correspondence, the scene identifier of the first terminal-side scene, and the first high-order rendering type. In implementing the embodiment of the present application, the server stores a correspondence between the first terminal-side scene, the first high-order rendering type, and the first cloud-side scene, so as to quickly locate the cloud-side scene to be rendered according to the scene identifier of the first terminal-side scene uploaded by the terminal and the first high-order rendering type.

[0011] In one implementation, the scene identifier and the first high-order rendering type of the first terminal-side scene are obtained by the first electronic device from the application package of the first application; the scene identifier of the first cloud-side scene, and the first correspondence between the first terminal-side scene, the first high-order rendering type and the first cloud-side scene are deployed on the server during the application development of the first application.

[0012] In one implementation, the server stores a second cloud-side scene converted from the first terminal-side scene for the second high-level rendering type. In the embodiment of the present application, a terminal-side scene is converted into a cloud-side scene for each high-level rendering type.

[0013] In one implementation, the server sends the first preprocessed data to the first electronic device, including: the server sends the compressed and encoded first preprocessed data to the first electronic device; before the first electronic device obtains the first image based on the first basic rendering data and the first preprocessed data, it also includes: the first electronic device decodes the received first preprocessed data to obtain the decoded first preprocessed data. In implementing the embodiment of the present application, the server feeds back the compressed and encoded preprocessed data to the first electronic device, which can reduce the amount of transmitted data and thus reduce the transmission delay.

[0014] In one implementation, the first high-order rendering type is global illumination; the first preprocessing data includes the irradiance of each pixel in the imaging picture obtained by preprocessing the global illumination, and the first preprocessing data includes the irradiance of each pixel in the imaging picture after basic rendering; the irradiance of the pixel at the first coordinate in the first image is equal to the product of the irradiance of the pixel at the first coordinate in the first imaging picture and the first basic rendering data, and the pixel value of the pixel at the first coordinate in the first image is determined based on the irradiance of the pixel at the first coordinate.

[0015] In one implementation, the system further includes a second electronic device, the first electronic device and the second electronic device are electronic devices of users participating in the same copy of the first end-side scene; the method further includes: if the first pre-processed data is pre-processed data processed independently of the viewpoint, the server sends the first pre-processed data to the second electronic device. In implementing the embodiment of the present application, if there are multiple users in the same copy of the end-side scene, and the server can share the pre-processed data in the copy that is independent of the viewpoint with all users in the copy, reducing the consumption of cloud-side computing resources.

[0016] In one implementation, the method also includes: the second electronic device performs basic rendering on the first end-side scene to obtain third basic rendering data; the second electronic device obtains a third image based on the third basic rendering data and the first pre-processed data, and the third image has the light and shadow effects of the first high-order rendering type; the second electronic device displays the third image.

[0017] In a second aspect, the present application provides a terminal-cloud collaborative rendering method, including: a first electronic device performs basic rendering on a first terminal-side scene to obtain first basic rendering data; the first electronic device stores at least one terminal-side scene, and the first terminal-side scene is any one of the at least one terminal-side scene; the first electronic device sends a scene identifier and a first high-order rendering type of the first terminal-side scene to a server; the first electronic device sends first status data to the server, and the first status data includes data related to preprocessing of the first high-order rendering type in the status data of the first terminal-side scene; the first electronic device receives the first preprocessing data sent by the server; the first preprocessing data is obtained by performing preprocessing of the first high-order rendering type on the first cloud-side scene after the server updates the first cloud-side scene according to the first status data; the server stores the first cloud-side scene converted from the first terminal-side scene to the first high-order rendering type; the first electronic device obtains a first image based on the first basic rendering data and the first preprocessing data, and the first image has a rendering effect of the first high-order rendering type; the first electronic device displays the first image.

[0018] In implementing the embodiment of the present application, a terminal-side scene is converted into a cloud-side scene for each high-order rendering type, and the server stores the cloud-side scene; for a specific high-order rendering type, the electronic device only needs to have the ability to perform basic rendering on the terminal-side scene, and the server implements high-order rendering preprocessing of the cloud-side scene corresponding to the terminal-side scene; after the basic rendering data is integrated with the preprocessing data of the high-order rendering, the final rendered image presents the light and shadow effects of the above-mentioned specific high-order rendering type. In this way, for various high-order rendering types, the computing power requirements and load on the terminal-side GPU can be reduced; electronic devices (such as mobile phones, tablets, etc.) do not need to have high-standard hardware performance to present the rendering effects of high-order rendering. In addition, since the server only performs preprocessing of high-order rendering and does not run the full APP, the consumption of cloud-side computing resources is reduced.

[0019] In one implementation, the method also includes: the first electronic device updates the scene information of the first end-side scene; the first electronic device performs basic rendering on the updated first end-side scene to obtain second basic rendering data; when the network speed between the first electronic device and the server is lower than a preset value, or the network between the first electronic device and the server is disconnected, the first electronic device determines the second image according to the second basic rendering data; and the first electronic device displays the second image.

[0020] In one implementation, a first electronic device sends a scene identifier and a first high-order rendering type of a first end-side scene to a server, including: the first electronic device sends a first request to the server, the first request includes the scene identifier and the first high-order rendering type of the first end-side scene; the first request is used to trigger the server to perform preprocessing preparation, the preprocessing preparation includes loading a first cloud-side scene, and preparing a rendering pipeline for preprocessing of the first high-order rendering type; before the first electronic device sends first status data to the server, the method also includes: the first electronic device receives confirmation information sent by the server, the confirmation information is used to indicate that the preprocessing preparation has been completed; the first electronic device sends the first status data to the server, including: based on the confirmation information, the first electronic device sends the first status data to the server.

[0021] In one implementation, the state data of the scene on the first terminal side includes part or all of the following: scene identification information, light source information, character information, camera information, and scene update information.

[0022] In one implementation, the server stores a first correspondence between a first terminal-side scene, a first high-order rendering type, and a first cloud-side scene.

[0023] In one implementation, the scene identifier and the first high-order rendering type of the first terminal-side scene are obtained by the first electronic device from the application package of the first application; the scene identifier of the first cloud-side scene, and the first correspondence between the first terminal-side scene, the first high-order rendering type and the first cloud-side scene are deployed on the server during the application development of the first application.

[0024] In one implementation, the server stores a second cloud-side scene converted from the first terminal-side scene for a second high-level rendering type.

[0025] In one implementation, the first electronic device receives the first preprocessed data sent by the server, including: the first electronic device receives the compressed and encoded first preprocessed data sent by the server; before the first electronic device obtains the first image based on the first basic rendering data and the first preprocessed data, it also includes: the first electronic device decodes the received first preprocessed data to obtain the decoded first preprocessed data.

[0026] In one implementation, the first high-order rendering type is global illumination; the first preprocessing data includes the irradiance of each pixel in the imaging picture obtained by preprocessing the global illumination, and the first basic rendering data includes the irradiance of each pixel in the imaging picture after basic rendering; the irradiance of the pixel at the first coordinate in the first image is equal to the product of the irradiance of the pixel at the first coordinate in the first preprocessing data and the first basic rendering data, and the pixel value of the pixel at the first coordinate in the first image is determined based on the irradiance of the pixel at the first coordinate.

[0027] In the third aspect, the present application provides a terminal-cloud collaborative rendering method, the method comprising: a server receiving a scene identifier and a first high-order rendering type of a first terminal-side scene sent by a first electronic device; the server receiving first status data sent by the first electronic device, the first status data including data related to preprocessing of the first high-order rendering type in the status data of the first terminal-side scene; the server updates the first cloud-side scene according to the first status data, performs preprocessing of the first high-order rendering type on the updated first cloud-side scene, and obtains first preprocessing data; the server sends the first preprocessing data to the first electronic device; the first preprocessing data is used to fuse the first basic rendering data to obtain a first image; the first image has the light and shadow effects of the first high-order rendering type; the first basic rendering data is obtained by the first electronic device performing basic rendering on the first terminal-side scene.

[0028] In implementing the embodiment of the present application, a terminal-side scene is converted into a cloud-side scene for each high-order rendering type, and the server stores the cloud-side scene; for a specific high-order rendering type, the electronic device only needs to have the ability to perform basic rendering on the terminal-side scene, and the server implements high-order rendering preprocessing of the cloud-side scene corresponding to the terminal-side scene; after the basic rendering data is integrated with the preprocessing data of the high-order rendering, the final rendered image presents the light and shadow effects of the above-mentioned specific high-order rendering type. In this way, for various high-order rendering types, the computing power requirements and load on the terminal-side GPU can be reduced; electronic devices (such as mobile phones, tablets, etc.) do not need to have high-standard hardware performance to present the rendering effects of high-order rendering. In addition, since the server only performs preprocessing of high-order rendering and does not run the full APP, the consumption of cloud-side computing resources is reduced.

[0029] In one implementation, a server receives a scene identifier and a first high-order rendering type of a first end-side scene sent by a first electronic device, including: the server receives a first request sent by the first electronic device, the first request including the scene identifier and the first high-order rendering type of the first end-side scene; before the server receives the first status data sent by the first electronic device, the method also includes: based on the first request, the server performs preprocessing preparation, the preprocessing preparation includes loading the first cloud-side scene, and preparing a preprocessing rendering pipeline for the first high-order rendering type; the server sends confirmation information to the first electronic device, the confirmation information is used to indicate that the preprocessing preparation has been completed; the first status data is sent by the first electronic device based on the confirmation information.

[0030] In one implementation, the state data of the scene on the first terminal side includes part or all of the following: scene identification information, light source information, character information, camera information, and scene update information.

[0031] In one implementation, the server stores a first correspondence between a first end-side scene, a first high-order rendering type, and a first cloud-side scene. Before the server updates the first cloud-side scene according to the first status data, the method also includes: the server determines that the cloud-side scene to be rendered is the first cloud-side scene based on the first correspondence, the scene identifier of the first end-side scene, and the first high-order rendering type.

[0032] In one implementation, the scene identifier and the first high-order rendering type of the first terminal-side scene are obtained by the first electronic device from the application package of the first application; the scene identifier of the first cloud-side scene, and the first correspondence between the first terminal-side scene, the first high-order rendering type and the first cloud-side scene are deployed on the server during the application development of the first application.

[0033] In one implementation, the server stores a second cloud-side scene converted from the first terminal-side scene for a second high-level rendering type.

[0034] In one implementation, the server sends the first pre-processed data to the first electronic device, including: the server sends the compressed and encoded first pre-processed data to the first electronic device.

[0035] In one implementation, the first high-order rendering type is global illumination; the first preprocessing data includes the irradiance of each pixel in the imaging picture obtained by preprocessing the global illumination, and the first basic rendering data includes the irradiance of each pixel in the imaging picture after basic rendering; the irradiance of the pixel at the first coordinate in the first image is equal to the product of the irradiance of the pixel at the first coordinate in the first preprocessing data and the first basic rendering data, and the pixel value of the pixel at the first coordinate in the first image is determined based on the irradiance of the pixel at the first coordinate.

[0036] In one implementation, the system also includes a second electronic device, and the first electronic device and the second electronic device are electronic devices of users participating in the same copy of the first end-side scene; the method also includes: if the first preprocessed data is preprocessed data processed independently of perspective, the server sends the first preprocessed data to the second electronic device.

[0037] In a fourth aspect, an embodiment of the present application provides an electronic device, comprising: a processor and a memory, wherein the memory is coupled to the processor, the memory is used to store computer program code, the computer program code comprises computer instructions, and when the processor reads the computer instructions from the memory, the electronic device executes the end-cloud collaborative image enhancement method described in the first aspect.

[0038] In the fifth aspect, an embodiment of the present application provides a server, comprising: a processor and a memory, wherein the memory is coupled to the processor, the memory is used to store computer program code, the computer program code comprises computer instructions, and when the processor reads the computer instructions from the memory, the server executes the end-cloud collaborative rendering method described in the second aspect.

[0039] In a sixth aspect, an embodiment of the present application provides a computer storage medium, including computer instructions. When the computer instructions are executed on an electronic device, the communication device executes the end-cloud collaborative rendering method in any possible implementation of any of the above aspects.

[0040] In a seventh aspect, an embodiment of the present application provides a computer program product. When the computer program product runs on a computer, it enables the computer to execute the end-cloud collaborative rendering method in any possible implementation of any of the above aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 A system architecture diagram of a communication system provided in an embodiment of the present application;

[0042] Figure 2A A system architecture diagram of a rendering system provided in an embodiment of the present application;

[0043] Figure 2B A system architecture diagram of another rendering system provided in an embodiment of the present application;

[0044] Figure 3A The GI-free rendering effect provided by the embodiment of the present application;

[0045] Figure 3B The rendering effect with GI provided in the embodiment of the present application;

[0046] Figure 3C A reflection-free rendering effect provided by an embodiment of the present application;

[0047] Figure 3D A rendering effect with reflection provided in an embodiment of the present application;

[0048] Figure 4 A data flow diagram provided for an embodiment of the present application;

[0049] Figure 5 A system architecture diagram of another rendering system provided in an embodiment of the present application;

[0050] Figure 6 A flowchart of a rendering method for end-cloud collaboration in the development phase is provided for an embodiment of the present application;

[0051] Figure 7 A schematic diagram of end-to-end cloud collaborative rendering is provided for an embodiment of the present application;

[0052] Figure 8 A schematic diagram of the process flow of the end-cloud collaborative rendering method in the running stage provided in an embodiment of the present application;

[0053] Fig. 9 A system architecture diagram of another rendering system provided in an embodiment of the present application;

[0054] Fig.10 A schematic diagram of the structure of a mobile device provided in an embodiment of the present application;

[0055] Fig.11 A schematic diagram of the structure of a server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0056] The technical solutions in the embodiments of the present application will be described clearly and in detail below in conjunction with the accompanying drawings. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in the text is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0057] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features, and in the description of the embodiments of the present application, unless otherwise specified, "plurality" means two or more.

[0058] The term "user interface (UI)" in the following embodiments of the present application refers to a medium interface for interaction and information exchange between an application or operating system and a user, which realizes the conversion between the internal form of information and the form acceptable to the user. The user interface is a source code written in a specific computer language such as Java and extensible markup language (XML). The interface source code is parsed and rendered on an electronic device, and finally presented as content that can be recognized by the user. The commonly used form of user interface is a graphical user interface (GUI), which refers to a user interface related to computer operation displayed in a graphical manner. It can be a visual interface element such as text, icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets, etc. displayed on the display screen of an electronic device.

[0059] 3D rendering technology is the process of projecting the model in the constructed 3D scene into a two-dimensional digital image according to the set viewpoint, light source, material and other information.

[0060] Traditional 3D rendering technologies, such as rasterization technology, divide the object model in the 3D scene into triangles, transform the three-dimensional coordinates of the triangle vertices into two-dimensional coordinates on the image through geometric transformation, and finally fill the triangles on the image with textures, thereby mapping the 3D object model to a two-dimensional screen to achieve image rendering; this technology is usually difficult to truly restore the light reflection, object shadow, refraction and other light and shadow effects in the 3D scene, so the rendered image is difficult to present a realistic 3D visual experience.

[0061] Compared with traditional 3D rendering technology, images rendered by ray tracing technology provide a more realistic 3D visual experience. Ray tracing technology simulates the process of light propagation in a 3D scene through reflection, refraction, shadow, scattering, etc., calculates the color and brightness value of each pixel, and renders a two-dimensional image with light and shadow effects that conform to the laws of real physics, thereby simulating a more realistic 3D virtual scene on electronic devices.

[0062] Ray tracing technology needs to simulate a huge number of light paths to obtain visual effects that fit the real world, which means that the demand for GPU computing power for this technology explodes with the number of simulated rays. NVIDIA first implemented real-time hardware ray tracing technology for PC-level GPUs on the RTX 20 series graphics cards in 2018; however, due to power consumption and computing power limitations, the end-side GPUs of most mobile devices are currently unable to implement hardware ray tracing technology, and large-scale commercial promotion is still under study. In the end-cloud collaborative rendering method provided in the embodiment of the present application, the mobile device collaborates with the PC-level GPU on the cloud side to render the 3D scene, and then presents the rendering effect that can be achieved by ray tracing technology on the mobile device, reducing the computing power requirements and load on the end-side GPU of the mobile device.

[0063] The following introduces the communication system applied to the end-cloud collaborative rendering method provided in the embodiment of the present application.

[0064] Figure 1 The system architecture of the communication system 10 provided in the embodiment of the present application is exemplarily shown. Figure 1 As shown, the communication system 10 includes a terminal device 100 and a cloud-side infrastructure, and the cloud-side infrastructure includes a server 200. The terminal device 100 can communicate with the cloud-side infrastructure through a communication network, and the cloud-side infrastructure can provide the terminal device 100 with a rendering service for a 3D application (Application, APP).

[0065] The above-mentioned 3D applications are applications that provide 3D picture display effects. 3D pictures are two-dimensional images rendered based on models in 3D scenes, such as 3D game apps, 3D modeling apps, 3D navigation apps, 3D home improvement apps, etc. The following embodiments take game apps as an example for illustrative explanation.

[0066] The terminal device 100 may be a mobile phone, a tablet computer, a desktop computer, a laptop computer, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a cellular phone, a personal digital assistant (PDA), an augmented reality (AR) device, a virtual reality (VR) device, an artificial intelligence (AI) device, a wearable device (such as a smart bracelet), a vehicle-mounted device, a smart home device (such as a smart TV, a smart screen, a large-screen device, etc.) and / or a smart city device. The embodiment of the present application does not impose any special restrictions on the specific type of the terminal device 100, and the terminal device 100 may also be referred to as a terminal side or an electronic device.

[0067] The terminal device 100 can be a mobile device or a non-mobile device. Usually, the size of a mobile device is relatively small, and the GPU computing power is more limited. The beneficial effect of the end-cloud collaborative rendering solution provided by this application is more significant. The subsequent embodiments take the terminal device 100 as a mobile device 100 as an example for exemplary description.

[0068] The server 200 may be a single server, or a server cluster consisting of multiple servers, or a cloud computing center. The server 200 involved in the embodiment of the present application may also be referred to as a cloud server, cloud side, or cloud end. Not limited to the server 200, the cloud infrastructure may also include more other devices, which are not specifically limited here.

[0069] The communication network may include local area networks (LAN) and / or wide area networks (WAN). The communication network may be implemented using any known network communication protocol, which may be various wired or wireless communication protocols, such as Ethernet, universal serial bus (USB), FIREWIRE, global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), Bluetooth, wireless fidelity (Wi-Fi), NFC, voice over Internet protocol (VoIP), communication protocols supporting network slicing architecture, or any other suitable communication protocols.

[0070] It should be understood that Figure 1 The system structure diagram of the communication system provided in the embodiment of the present application is only a schematic diagram, and does not constitute a specific limitation on the communication system 10. The communication system 10 may include more or less devices than shown in the figure, for example, it may also include a wireless relay device and a wireless backhaul device ( Figure 1 ), which is not shown in the figure and is not limited here.

[0071] In a rendering method for end-cloud collaboration (i.e., method 1) provided in an embodiment of the present application, all rendering work of the game APP is transferred to the cloud side, and the GPU computing power of the cloud side is used to improve the rendering effect of the mobile device. For example, Figure 2A As shown, in this method, the mobile device 100 does not need to install the game, but only needs to install the game client of the game APP, which includes an operation instruction processing module and a video decoding module; the server 200 can run the game APP, which includes a rendering module and a logic module, and all rendering work of the game APP is performed in the virtual host / container of the server 200.

[0072] Among them, the operation instruction processing module of the mobile device 100 is responsible for collecting the user's operation instruction information (such as left and right movement, perspective switching, clicking, etc.), and uploading the operation instructions to the game APP on the cloud side. The game APP on the cloud side sends the received operation instruction information to the logic module. After the logic module updates the game status according to the operation instruction information, it triggers the rendering module to render the game scene in the latest state; the game APP on the cloud side transmits the rendered game screen to the game client of the mobile device 100 in the form of a video stream. The video decoding module of the mobile device 100 decodes the video stream to obtain the game screen, and triggers the mobile device 100 to display the game screen. Optionally, relying on the powerful computing power of the PC-level GPU on the cloud side, the rendering module on the cloud side can use ray tracing technology to render the game screen corresponding to the game scene in the latest state.

[0073] Implementing the above-mentioned end-cloud collaborative rendering method can reduce the computing power requirements and load on the end-side GPU. However, the above-mentioned rendering method still has the following problems: (1) All game rendering tasks are placed on the cloud side for execution, and the computing cost of the cloud side is high. The end side is only used as a video player, which wastes the GPU resources on the end side. (2) The rendering results on the cloud side cannot be shared among multiple users, further increasing the computing cost on the cloud side. (3) The transmission bandwidth cost is high. The rendered game screen is transmitted in the form of video streaming. In the case of 1080P@60fps, the transmission bandwidth is about 10Mbps, resulting in high export bandwidth of the cloud-side server and downlink bandwidth costs of users. (4) Poor network experience. For mobile users, they often encounter poor network conditions. When the network is poor, the user's operation response delay is very high, affecting the game operation experience.

[0074] In another end-cloud collaborative rendering method (i.e., method 2) provided in an embodiment of the present application, the end side performs basic rendering of the game scene, and the cloud side performs preprocessing of high-level rendering of the game scene; the end side fuses the basic rendering data and the preprocessing data of the high-level rendering to obtain the game screen of the game scene after the high-level rendering. For example, Figure 2B As shown, in this method, the mobile device 100 can install a game APP, and the game APP includes an operation instruction processing module, a rendering module and a logic module; the server 200 includes a high-level rendering service.

[0075] The operation instruction processing module of the mobile device 100 is responsible for collecting the user's operation instruction information; and sending the operation instruction to the logic module of the mobile device 100. After the logic module updates the game state according to the operation instruction information, it triggers the rendering module of the mobile device 100 to perform basic rendering on the game scene 1 (i.e., the end-side scene 1) in the latest state, and sends the state synchronization data to the server 200. The high-order rendering service of the server 200 updates the cloud-side scene 1 corresponding to the end-side scene 1 according to the state synchronization data; performs high-order rendering preprocessing according to the updated cloud-side scene 1, and sends the preprocessing data to the mobile device 100. After the rendering module of the mobile device 100 fuses the basic rendering data and the preprocessing data of the high-order rendering, the final rendered game screen can be obtained. The game screen has a high-order rendering effect and can achieve the rendering effect that can be achieved by ray tracing technology. If there are multiple players in a copy of the same game scene, the server 200 can share the preprocessing data of the high-order rendering in the copy that is independent of the perspective with all players in the copy.

[0076] The implementation of the embodiment of the present application can have the following beneficial effects: only basic rendering and fusion are performed on the end side, and there is no need to perform ray tracing for high-level rendering, which can reduce the computing power requirements and load on the GPU on the end side; the mobile devices of game players (such as mobile phones, tablets, etc.) do not need to have high-standard hardware performance, and can also present the rendering effects that can be achieved by ray tracing technology, and play 3D games. In addition, since the cloud side only performs preprocessing of high-level rendering and does not run the full game APP, the consumption of computing resources on the cloud side is reduced; at the same time, the high-level rendering service on the cloud side can share the preprocessing data of high-level rendering of the same copy with the end-side clients corresponding to multiple game users in the same copy, further reducing the consumption of computing resources on the cloud side. The data transmitted from the cloud side to the end side can be compressed preprocessing data of high-level rendering, which requires lower transmission bandwidth than transmitting video streams (such as 720p / 1080p@30fps / 60fps, 1080p@30fps video streams); compared with the aforementioned method one, since the required transmission bandwidth is lower, the transmission delay is lower under poor network conditions, reducing the impact of weak networks on user operation experience. In addition, when network conditions are poor, the terminal side can still perform basic rendering to ensure the normal operation of the game app, thereby guaranteeing the user's basic operating experience.

[0077] The second method mentioned above is introduced in detail below. First, the technical concepts involved in the above-mentioned end-cloud collaborative rendering method are introduced.

[0078] A 3D scene (i.e., a virtual scene) may refer to a virtual three-dimensional scene environment (e.g., a city, park, forest, street, mountain, etc.) generated by a computer. It can provide a multimedia virtual world. Users can control operable virtual objects in the virtual scene through operating devices or operating interfaces, and observe virtual objects such as objects, animals, characters, and scenery in the virtual scene from the perspective of virtual objects. The mobile device 100 can construct a 3D object model based on each virtual object, and then construct a model of the entire scene. Exemplarily, a scene in a game APP may include virtual scenes of one or more game levels, or may include one or more specific areas in a game level (e.g., a house, a room, a garden, etc.).

[0079] Basic rendering and high-level rendering are relative concepts. The GPU computing power required for basic rendering is relatively low, including rendering tasks that can be completed independently by relying on the GPU computing power of the mobile device 100. The game screen of the game scene generated by basic rendering can ensure the normal operation of the game APP and the normal interaction of user operations. When the communication network between the mobile device 100 and the server 200 is poor or disconnected, and the pre-processed data of the high-level rendering fed back by the cloud side is not obtained, the game APP of the mobile device 100 can directly display the game screen after basic rendering to ensure that the user can operate and watch the game APP normally.

[0080] In the embodiment of the present application, the rendered image generated after the basic rendering is completed can be referred to as the basic rendered image, and the basic rendering data generated by the basic rendering can be the intermediate data in the process of generating the basic rendered image (for example, the irradiance of each pixel obtained by the basic rendering), or it can be the image data of the basic rendered image. The GPU computing power required for the rendering technology used in the basic rendering is what the mobile device 100 can provide, and the embodiment of the present application does not specifically limit the rendering technology. In some embodiments, the basic rendering is performed using rasterization technology on the scene to be rendered.

[0081] The GPU computing power required for high-level rendering is relatively high, including rendering tasks that rely on the GPU computing power on the cloud side to be executed. High-level rendering tasks are usually easy to decouple from basic rendering tasks. The preprocessing data of a specific high-level rendering type is used to obtain the light and shadow effects of the above-mentioned specific high-level rendering type in the picture to be rendered. The preprocessing data corresponding to different high-level rendering types may be different, and accordingly, the fused basic rendering data may also be different. For example, the above-mentioned specific high-level rendering type is GI rendering, the preprocessing data of GI includes the irradiance of each pixel after GI rendering, and the basic rendering data includes the irradiance of each pixel after basic rendering; the above-mentioned specific high-level rendering type is reflection rendering, the preprocessing data of reflection rendering includes the cube texture map (cubemap) captured by the reflection probe, and the basic rendering data includes the color of each pixel after basic rendering.

[0082] In some embodiments, high-level rendering is a rendering task related to lighting characteristics, such as global illumination (GI), ambient occlusion (AO), soft shadow, reflection, refraction, and caustics. It may also include rendering of other characteristics that are relatively computationally intensive on the cloud side and can be integrated with basic rendering on the terminal side.

[0083] Reflection refers to the phenomenon that when light propagates to different substances, it changes its propagation direction at the interface and returns to the original substance. GI is a lighting characteristic formed by direct lighting and indirect lighting. Global illumination is more in line with the real lighting in reality. Direct lighting is the lighting phenomenon formed by the light emitted by the light source directly shining on the objects in the scene, and indirect lighting is the lighting phenomenon formed in the scene by the light emitted by the light source that is reflected by the surface of the objects in the scene. For example, Figure 3A and Figure 3B The game APP shows the rendering effect of the same game screen without GI and with GI; Figure 3A As shown in the figure, when there is no GI, the game screen can also run and display normally, but the picture effect is dim and lacks the light and shadow details of indirect lighting, such as the lack of light and shadow details formed by the reflection of objects on the wall; Figure 3B As shown, when GI is used, the picture effect will be brighter and more realistic. Figure 3C and Figure 3D It shows the rendering effect of the same game screen with and without reflection; Figure 3C As shown, when there is no reflection, the game screen can also run and display normally, but the screen effect lacks the reflected light and shadow effect, for example, the mirror in the screen does not show the mirror projection content; Figure 3D As shown, when there is reflection, the mirror in the picture shows the mirror projection content.

[0084] In some embodiments, the basic rendering tasks of the game app include a minimum set of rendering tasks that ensure the normal operation of the game screen; the high-level rendering tasks of the game app include rendering tasks other than the above basic rendering tasks of the game app. In some embodiments, high-level rendering tasks other than the above minimum set may also include rendering tasks that can be achieved only by relying on the computing power of the end-side GPU. Transferring this rendering task to the cloud side can reduce the computing power requirements and load on the end-side GPU.

[0085] In some embodiments, the game APP can determine the type of high-order rendering that can be currently rendered in end-cloud collaborative rendering based on the real-time network conditions. In the embodiment of the present application, the server 200 needs to feedback the pre-processed data of the high-order rendering to the mobile device 100. The data type and data volume of the pre-processed data of different high-order rendering types may be different. For example, compared with reflection, the data volume of GI pre-processed data is lower. Under normal network conditions (for example, a round-trip delay of 50-100ms and a transmission bandwidth of 10Mbps), on the premise of ensuring the maximum transmission delay of the pre-processed data that meets the normal operation of the user, there are fewer high-order rendering types that meet the end-cloud collaborative rendering; under very good network conditions (such as RTT 10-20ms, bandwidth 100Mbps), on the premise of ensuring the maximum delay of the user's operation, there will be more high-order rendering types that meet the end-cloud collaborative rendering.

[0086] In the embodiment of the present application, the developer can determine the basic rendering tasks and high-level rendering tasks of the application (such as a game APP) based on the actual application requirements and the GPU capabilities of the mobile device, and no specific limitations are made here.

[0087] For the above-mentioned end-cloud collaborative rendering method, the present application embodiment specifically provides two implementation solutions. Among them,

[0088] In implementation scheme 1, the client side independently downloads the client-cloud collaborative plug-in SDK for the client-side 3D engine, which is used to implement the above-mentioned client-cloud collaborative rendering method. In implementation scheme 2, the native client-side 3D engine supports the above-mentioned client-cloud collaborative rendering method. The following describes the two implementation schemes in detail.

[0089] For example, Figure 4 FIG. 1 shows a data flow diagram of a mobile device 100 involved in implementing solution 1. Figure 4 As shown, the mobile device 100 includes an input output (IO) layer, a logic layer, a 3D engine layer, an operating system (OS) layer, and a hardware layer. Among them, the IO layer can obtain the user input of the game APP from the operation quality processing module of the OS layer and send it to the logic layer; the logic layer runs the logic code of the game APP (i.e., the above-mentioned logic module), and the logic layer updates the game state according to the user input; the logic layer instructs the 3D engine layer to render the game screen under the latest game state; the 3D engine layer calls the GPU through the OS layer to render the game screen.

[0090] For example, Figure 5 The system architecture diagram of a terminal-cloud collaborative rendering system involved in implementation scheme 1 is shown. Figure 5As shown, the rendering system includes a mobile device 100 and a server 200. The game APP of the mobile device 100 mainly includes the following modules: logic code, end-side 3D engine and end-cloud collaboration plug-in SDK; the server 200 mainly includes the following modules: high-level rendering service, end-cloud collaboration framework and cloud-side 3D engine.

[0091] Logic code: includes the logic code that implements the game logic of the game APP.

[0092] End-side 3D engine: The end-side 3D engine includes the engine framework, scene management, and rendering pipeline. The engine framework is used to provide game logic analysis, model analysis, and animation analysis capabilities for the end-side game app. Scene management is used to manage the scene information of one or more game scenes (i.e., end-side scenes) built by the game app; the rendering pipeline is the process of converting a three-dimensional scene model into an output in the screen pixel space, including the following functions: converting the 3D coordinates of an object into 2D coordinates in the screen pixel space; and coloring each pixel on the screen.

[0093] The end-side 3D engine is divided into two modes: development state and running state. Among them, the end-side 3D engine provides developers with an integrated development environment (IDE) in the development state, allowing developers to develop the logic code of the game APP and the rendering effect of the 3D scene; after the development is completed, the developed related files can be packaged into an application package of the game APP that can be run on mobile devices, such as an Android application package (APK). The end-side 3D engine provides the game APP with a Runtime runtime in the running state; the Runtime runtime generally has cross-platform capabilities and can support the game APP to run on mobile devices with different operating systems such as Hongmeng, Android and iOS. In the embodiment of the present application, when the game APP is running at Runtime, it can call the above-mentioned rendering pipeline to perform basic rendering on the end-side scene to be rendered and generate basic rendering data; it can also call the end-cloud collaborative plug-in SDK to request the server 200 to perform high-order rendering preprocessing; and fuse the basic rendering data and the preprocessed data of the high-order rendering, and obtain a displayable high-order rendering image based on the fused data.

[0094] End-cloud collaborative plug-in SDK: This SDK encapsulates the key capabilities of end-cloud collaborative rendering, and based on the plug-in mechanism provided by the end-side 3D engine, enables the third-party end-side 3D engine to have the ability of end-cloud collaborative rendering. The end-cloud collaborative plug-in SDK includes some or all of the following sub-functions: scene conversion, state synchronization, transmission communication, decoding, pipeline adaptation, and authentication. In the embodiment of the present application, end-cloud collaborative rendering is achieved by providing an end-cloud collaborative plug-in SDK to the third-party 3D engine, thereby avoiding invasive modifications to the third-party 3D engine.

[0095] Scene conversion: used to convert the scene to be rendered by the end-side 3D engine for end-cloud collaborative rendering (i.e., the end-side scene) into the scene required for high-level rendering preprocessing on the cloud side (i.e., the cloud-side scene). The scene information of the cloud-side scene can be consistent with or inconsistent with the scene information of the end-side scene.

[0096] State synchronization on the end side: used to synchronize state data (hereinafter referred to as state synchronization data) that affects the cloud-side preprocessing and preprocessing result distribution strategy to the cloud side. Exemplarily, the state synchronization data includes some or all of the following: scene identification information of the scene to be rendered (such as game identification (identity, ID), level ID, scene ID, copy ID, etc.), light source information (such as the type, position, posture, brightness, quantity, etc. of the light source), character information (such as the position, posture, animation, skills, quantity, etc. of the virtual character), camera information (such as the position, posture, field of view (FOV) of the camera), scene update information (such as relevant information about object movement, object animation, destruction, etc.). The camera's perspective (i.e., the game player's perspective) determines the image content of the final image after the 3D scene is rendered, and will also affect the contribution of the light source in the scene to the imaging.

[0097] Transmission communication on the client side: used to upload the client-side state synchronization data to the cloud side, and to receive the high-level rendering pre-processing data sent by the cloud side.

[0098] Decoding on the client side: used to decode the compressed and encoded pre-processed data sent from the cloud side into pre-processed data that can be consumed by the rendering pipeline on the client side.

[0099] End-side pipeline adaptation: It is used to add the consumption rendering capability of decoded pre-processed data based on the original end-side rendering pipeline according to the extension mechanism of the end-side 3D engine rendering pipeline, that is, the capability to fuse basic rendering data and pre-processed data to obtain high-level rendering effects.

[0100] End-side authentication and authorization: responsible for the authentication and authorization between the end-cloud collaboration plug-in SDK on the end side and the high-level rendering service on the cloud side.

[0101] Cloud-side 3D engine: includes major functional modules such as scene management, rendering pipeline and engine framework, providing a capability base for the end-cloud collaborative framework and high-level rendering services.

[0102] End-cloud collaboration framework: includes some or all of the following sub-functions: state synchronization, transmission communication, decoding, session management, and authentication.

[0103] State synchronization on the cloud side: used to receive the state synchronization data of the end-side scene (e.g., end-side scene 1) sent by the end-side, and update the scene information of the corresponding cloud-side scene (e.g., cloud-side scene 1 corresponding to end-side scene 1) in the scene management module of the cloud-side 3D engine.

[0104] Cloud-side encoding: used to compress and encode pre-processed data of high-level rendering services to reduce the amount of transmitted data.

[0105] Transmission communication on the cloud side: used to receive status synchronization data transmitted by the end side, and to send compressed and encoded pre-processed data to the end side.

[0106] Session management: used to assign clients connected to the end side to the corresponding high-level rendering services. For example, the assignment is made based on the game ID, level / scene ID, copy ID, and other information uploaded by the end side. It can be understood that the server 200 can establish sessions with multiple end-side clients and the high-level rendering services corresponding to the sessions, so that high-level rendering services can be provided to multiple clients at the same time.

[0107] Cloud-side authentication: used for authentication between the high-level rendering service on the cloud side and the end-to-end cloud collaboration plug-in SDK on the end side.

[0108] High-level rendering service: Based on the cloud-side 3D engine and the end-cloud collaborative framework, it provides high-level rendering services, that is, high-level rendering preprocessing, such as GI preprocessing, AO preprocessing, reflection preprocessing, soft shadow preprocessing and other high-level rendering services.

[0109] In some embodiments, in order to save cloud computing power and facilitate development and debugging, during the development stage of the game APP, Figure 5 The modules of the mobile device 100 and the server 200 involved in the rendering system shown can be simulated, developed and tested on the same device, that is, the cloud-side 3D engine and the terminal-side 3D engine in the development state can also be set in the same device, which can be a mobile device or a cloud-side server, and is not specifically limited here. Preferably, the device can be a mobile device.

[0110] Based on the foregoing embodiments, an embodiment of the present application provides an end-cloud collaborative rendering method, which is applied to an end-cloud collaborative rendering system. The system includes a game APP of a mobile device 100, and the game APP includes an end-side 3D engine and an end-cloud collaborative plug-in SDK. Figure 6 A method flow chart of a terminal-cloud collaborative rendering method in the development stage is shown, and the method flow includes part or all of steps S101 to S125.

[0111] S101. The IDE of the end-side 3D engine determines an end-side scene 1 and a high-level rendering type 1 for end-cloud collaborative rendering.

[0112] In an embodiment of the present application, during the APP development phase, the development state of the end-side 3D engine provides an IDE for developers; developers select the end-side scene (for example, end-side scene 1) to be rendered in end-cloud collaboration in the IDE and the high-order rendering type for the above-mentioned end-side scene. For game apps, the end-side scene can be one or more levels in a game, or it can be a partial space in a level (for example, the interior of a building). The high-order rendering type is the rendering type corresponding to the high-order rendering task described in the previous embodiment. Exemplarily, high-order rendering types include GI, AO, soft shadows, reflections, refractions, and caustics. The embodiments of the present application may also include other high-order rendering types, which are not specifically limited here. The subsequent embodiments mainly use GI and reflection as examples for illustrative explanation.

[0113] S102. The end-side 3D engine sends the scene information and high-level rendering type 1 of the end-side scene 1 to the end-cloud collaborative plug-in SDK.

[0114] In some embodiments, the scene information of a terminal scene includes a description of a 3D object model (e.g., a building, a person, or other object) in the scene (e.g., the name, geometry, texture, material, roughness, and transparency of the object model), and may also include a description of the background environment, light source information, scene identification information, character information, camera information, etc. in the scene. There are many ways to represent the geometric structure of a 3D object model. For example, a triangular facet mesh (Mesh) can be used to represent the surface of a 3D object, that is, the surface of a 3D object is approximately represented by multiple Meshes. The Mesh information of a Mesh may include useful properties and functions such as the vertex coordinates, normals, texture coordinates, and triangle drawing sequence of the Mesh.

[0115] In an embodiment of the present application, some high-order rendering types can obtain the lighting information of each point in the scene by deploying probes in the scene. In one implementation, the terminal side can deploy probes for implementing specific high-order rendering types (such as GI) in the 3D scene according to a preset deployment rule (for example, uniformly distributed according to a preset density within a preset spatial range of the 3D scene), and each probe can capture, store and update the lighting information received at its location. When it is necessary to shade a shading point (such as shading point 1) of the 3D scene, the probes adjacent to the shading point 1 can be queried, and the shading point 1 can be shaded according to the lighting information of the location of the adjacent probe, and then the pixels corresponding to the shading point 1 in the imaging picture can be rendered.

[0116] In some embodiments, the scene information of the end-side scene 1 may also include relevant information of the probes required for high-order rendering deployed in the scene, such as the deployment rules of the probes, the spatial range of deployment, the number, the historical fusion coefficient, the deviation value, and other information. The embodiments of the present application do not specifically limit the relevant information of the probes. In the embodiments of the present application, the probes deployed to implement GI may be referred to as GI probes, and the probes deployed to implement reflection may be referred to as reflection probes.

[0117] S103. The end-cloud collaboration plug-in SDK converts the scene information of the end-side scene 1 into the scene information of the cloud-side scene 1 according to the high-level rendering type 1 and the preset conversion rules.

[0118] In the embodiment of the present application, the information content of the scene information of the terminal side scene 1 and the cloud side scene 1 may be consistent or inconsistent.

[0119] In some embodiments, after the cloud collaboration plug-in SDK obtains the scene information of the terminal side scene 1, it can simplify the information content of the scene information of the terminal side scene 1 according to preset conversion rules, and only retain the scene information required for the preprocessing of the high-order rendering type 1. The simplified scene information indicates the cloud side scene 1 corresponding to the terminal side scene 1.

[0120] Exemplarily, in one implementation, if the developer selects high-level rendering types such as GI, AO, and soft shadow, the scene information of the converted cloud-side scene 1 only needs to retain the attribute information (such as Mesh information and color information) of the object surface of each 3D object model in the terminal-side scene 1; even compared with the accuracy of the Mesh information of each model in the scene information of the terminal-side scene 1, the accuracy of the Mesh information in the scene information of the cloud-side scene 1 can be lower. Reducing the accuracy of the Mesh information is conducive to reducing the complexity of subsequent high-level rendering preprocessing. If the high-level rendering types such as reflection and refraction are selected, the scene information of the converted cloud-side scene 1 and the terminal-side scene 1 needs to maintain a higher similarity, for example, the information content of the scene information of the cloud-side scene 1 and the terminal-side scene 1 is consistent.

[0121] In some embodiments, the data formats of the scene information of the terminal side scene 1 and the cloud side scene 1 may be consistent or inconsistent. The data format of the scene information indicates what types of information the scene information includes, how to represent each type of information, and how to arrange each type of information in the scene information.

[0122] In some embodiments, the cloud-side 3D engine and the terminal-side 3D engine may use the same 3D engine (i.e., a 3D engine of the same version from the same manufacturer), or different 3D engines. The data formats of scene information managed by different 3D engines are usually different. When the cloud-side 3D engine and the terminal-side 3D engine are different 3D engines, it is necessary to convert the scene information of the terminal-side scene 1 into the scene information of the cloud-side scene 1 based on the data format of the scene information corresponding to the high-order rendering type 1 managed by the cloud-side 3D engine. The conversion process may also involve simplification of the above-mentioned information content.

[0123] In some embodiments, when the cloud-side 3D engine and the terminal-side 3D engine are the same 3D engine, the data format of the scene information of the terminal-side scene 1 and the cloud-side scene 1 is the same, and the scene information of the terminal-side scene 1 and the cloud-side scene 1 can be kept consistent without conversion.

[0124] In an embodiment of the present application, the developer can manually convert the end-side scene 1 into the cloud-side scene 1 according to preset conversion rules through the end-cloud collaboration plug-in SDK; or use the developed automatic conversion tool to automatically convert the end-side scene 1 into the cloud-side scene 1 according to preset conversion rules.

[0125] Step S103 is optional. In some embodiments, there is no need to execute S103, the terminal side scenario and the cloud side scenario remain completely consistent, and the cloud side scenario 1 involved in the subsequent embodiments is equivalent to the terminal side scenario 1.

[0126] It can be understood that in the embodiment of the present application, a terminal scene is converted into a cloud scene for each high-level rendering type. For example, for high-level rendering type 1 (such as GI), terminal scene 1 can be converted into cloud scene 1; for high-level rendering type 2 (such as reflection), terminal scene 1 can be converted into cloud scene 2.

[0127] S104. The end-cloud collaboration plug-in SDK sends the scene information of the cloud-side scene 1 and the high-level rendering type 1 to the IDE of the cloud-side 3D engine.

[0128] S105. The IDE of the cloud-side 3D engine adjusts the scene information of the cloud-side scene 1.

[0129] In some embodiments, conversion errors may occur during the conversion of the terminal scene 1 to the cloud scene 1, resulting in the scene conversion not strictly complying with the above-mentioned preset conversion rules. For example, due to human errors when the developer manually performs scene conversion, or a runtime bug when the automatic conversion tool automatically performs scene conversion, resulting in conversion errors. After importing the scene information of the cloud-side scene 1 into the IDE of the cloud-side 3D engine, the developer can check and adjust the scene information of the cloud-side scene 1 through the IDE of the cloud-side 3D engine. Exemplarily, the above adjustments include: repairing the material, geometry, size or position of the 3D object model with conversion problems, repairing the spatial range and number of GI probes, etc.

[0130] Step S105 is optional. In some embodiments, step S105 does not need to be performed.

[0131] S106 . The IDE of the terminal-side 3D engine performs basic rendering on the terminal-side scene 1 according to the status data 1 of the terminal-side scene 1 , and obtains basic rendering data 1 .

[0132] The state data of the 3D scene includes data that can be adjusted in the scene information when the 3D scene is running. The state data indicates the real-time state of the 3D scene. During the development phase, the IDE of the end-side 3D engine can simulate and generate various state data of the end-side scene 1 (for example, state data 1) to debug the rendering effect of the end-side scene 1 during the development phase.

[0133] In some embodiments, the IDE of the end-side 3D engine updates the scene information of the end-side scene 1 according to the state data 1; the IDE of the end-side 3D engine uses rasterization technology to perform basic rendering on the updated end-side scene 1 to obtain basic rendering data 1. In the embodiment of the present application, when the network is poor, the basic rendered game screen can be obtained according to the basic rendering data 1, and the game screen can ensure the normal operation of the game APP and the normal interaction of user operations. For example, see Figure 7In one implementation, the end-side rendering pipeline includes a basic rendering channel and a conversion channel; the basic rendering channel uses rasterization technology to render the end-side scene 1, and the output basic rendering data 1 includes the irradiance of each pixel in the imaging picture of the end-side scene 1; the conversion channel determines the pixel value (for example, RGB color information) of each pixel in the imaging picture according to the above basic rendering data 1, and outputs the basic rendering image as the rendering picture to be displayed.

[0134] S107. The IDE of the end-side 3D engine sends the state synchronization data 1 of the end-side scene 1 to the end-cloud collaboration plug-in SDK.

[0135] S108 . The end-cloud collaboration plug-in SDK sends the status synchronization data 1 of the end-side scene 1 to the end-cloud collaboration framework of the server 200 .

[0136] S109. The end-cloud collaboration framework of the server 200 sends the state synchronization data 1 of the end-side scene 1 to the IDE of the cloud-side 3D engine.

[0137] In some embodiments, the state synchronization data 1 includes state data required for implementing high-level rendering type 1 of the cloud-side scene 1 in the state data 1. The state synchronization data 1 may include part or all of the data in the state data 1. The end-cloud collaboration plug-in SDK synchronizes the state synchronization data 1 to the IDE of the cloud-side 3D engine, and the cloud-side 3D engine IDE can update the cloud-side scene 1 to the latest state according to the state synchronization data 1.

[0138] S110. The IDE of the cloud-side 3D engine updates the cloud-side scene 1 according to the state synchronization data 1, pre-processes the updated cloud-side scene 1 with the high-order rendering type 1, and obtains the pre-processed data 1 with the high-order rendering type 1.

[0139] High-order rendering type 1 can be the rendering of any one of the lighting characteristics such as GI, AO, soft shadows, reflection, refraction and caustics. The embodiments of the present application do not specifically limit the rendering algorithms of lighting characteristics such as GI, AO, soft shadows, reflection, refraction and caustics. For example, the rendering algorithms for implementing GI include but are not limited to: ray tracing, path tracing, dynamic diffuse global illumination (DDGI), spherical harmonic lighting, voxel-based global illumination, and point-based global illumination, etc.

[0140] In an embodiment of the present application, the IDE of the cloud-side 3D engine can synchronize data 1 according to the status of the terminal-side scene 1, update the scene information of the cloud-side scene 1 corresponding to the terminal-side scene 1, and use the rendering algorithm 1 to perform preprocessing of the high-order rendering type 1 (such as GI) on the updated cloud-side scene 1 to obtain preprocessing data; the preprocessing data can indicate the light and shadow effects of the high-order rendering type 1 in the cloud-side scene 1.

[0141] Exemplarily, taking the high-level rendering type 1 as GI and the rendering algorithm 1 as DDGI as an example, the preprocessing of the high-level rendering type 1 is exemplified. The DDGI algorithm uses GI probes to store the lighting information of the scene, and dynamically updates it using ray tracing, so as to achieve real-time dynamic diffuse global illumination effects. The DDGI algorithm packages a group of probes into a DDGI Volume (i.e., a cubic area in 3D space). You only need to drag the Volume into the scene to be rendered, and the Volume will automatically place probes in it; the shading points in the Volume will automatically capture lighting information through the surrounding probes.

[0142] In some embodiments, the scene information of the cloud side scene 1 includes relevant information of the GI probe (such as deployment rules, deployment location and data, etc.); see Figure 7 Based on the scene information of the cloud-side scene 1, the IDE of the cloud-side 3D engine uses the DDGI rendering channel of the cloud-side rendering pipeline to deploy and run GI probes in the cloud-side scene 1. The GI probes obtain and store the lighting information of their locations and generate Probe layer data; the Probe layer data includes the lighting information of each GI probe, and the lighting information of the GI probe includes irradiance. The shading point corresponding to each pixel in the imaging picture in the cloud-side scene 1 is determined according to the camera viewing angle. For any shading point in the Volume in the cloud-side scene 1 (for example, shading point 1), the DDGI rendering pipeline obtains the lighting information of the 8 GI probes around the shading point 1; based on the lighting information of the above 8 GI probes, the irradiance is interpolated to obtain the irradiance of the shading point 1. The preprocessing data of GI includes the irradiance of the shading point corresponding to each pixel in the imaging picture.

[0143] The probe deployed by DDGI stores spherical information. The DDGI algorithm encodes spherical data into a two-dimensional texture map through octahedral mapping. The smallest unit of the texture map is a texel, and a texel corresponds to one or more pixels. The lighting information of the GI probe can include the irradiance received from the hemisphere in the direction of the texel (w), the distance r(w) from the nearest object seen from the texel direction to the probe, and the square of the distance r 2 (w). The irradiance is encoded as a three-dimensional vector texture, r(w) and r 2(w) are encoded together as a two-dimensional vector texture (the x component stores r(w), and the y component stores r 2 (w)).

[0144] In one implementation, probe 1 is any one of the above-mentioned 8 GI probes. The above-mentioned interpolation processing of irradiance is performed according to the illumination information of the above-mentioned 8 GI probes to obtain the irradiance of shading point 1, including: obtaining three weight coefficients of probe 1, namely, trilinear difference coefficient, directional coefficient and Chebyshev coefficient, and taking the normalized value of the product of the above three coefficients as the weight of probe 1; based on the weight of each probe, weighted processing is performed on the irradiance of the above-mentioned 8 GI probes to obtain the irradiance of shading point 1. Among them, the trilinear difference coefficient indicates the distance between probe 1 and shading point 1. If the coefficient is large, the weight of probe 1 is reduced; the directional coefficient indicates the angle between the direction of shading point 1 pointing to probe 1 and the surface normal of shading point 1. If the coefficient is too large, the weight of probe 1 is reduced; the Chebyshev coefficient indicates the probability that there is an occluder between probe 1 and shading point 1. If the coefficient is large, the weight of probe 1 is reduced; the Chebyshev coefficient is based on the above-mentioned distance r(w) and the distance square r 2 (w) Determined.

[0145] In some embodiments, in addition to the illumination information of probe 1, the Probe layer data also includes classification information (classification) and relocation information (relocation) of probe 1; the classification information is used to indicate whether the state (state) of probe 1 is a valid probe, and then determine whether to use the illumination information of probe 1 according to the state of probe 1; the relocation information (relocation) is used to indicate whether to adjust and how to adjust the position of probe 1.

[0146] Exemplarily, taking the high-order rendering type 1 as reflection and the rendering algorithm 1 as a reflection capture algorithm based on a reflection probe as an example, the preprocessing of the high-order rendering type 1 is exemplarily described. The IDE of the cloud-side 3D engine uses the cloud-side rendering pipeline to deploy and run the reflection probe in the cloud-side scene 1 to obtain the lighting information of the reflection probe. The lighting information of the reflection probe includes the cubemap of the location of the reflection probe. The cubemap is a synthesis of the reflection texture map obtained by the reflection probe along the six directions of the front, back, left, right, up and down of the camera. Cubemap can be used to implement environment mapping, and environment mapping can simulate the environment around the probe. The preprocessing data of reflection includes the cubemap of all reflection probes in the cloud-side scene 1, or one or more cubemaps to which the position of the camera uploaded by the end side belongs, or the cubemap after the one or more cubemaps are fused, or the cubemap of the reflection probe closest to the camera.

[0147] S111. The IDE of the cloud-side 3D engine sends pre-processed data 1 of high-level rendering type 1 to the end-cloud collaborative plug-in SDK.

[0148] In some embodiments, in step S110, the IDE of the cloud-side 3D engine performs preprocessing of the high-level rendering type 1, and the obtained preprocessed data includes preprocessed data of all viewing angles of the camera. The IDE of the cloud-side 3D engine crops the preprocessed data according to the viewing angle of the camera in the state synchronization data 1, obtains the preprocessed data 1 within the viewing angle of the camera, that is, the preprocessed data related to the end-side user of the mobile device 100, and sends the preprocessed data 1 to the end-cloud collaboration plug-in SDK.

[0149] S112. The end-cloud collaborative plug-in SDK sends pre-processed data 1 of high-level rendering type 1 to the IDE of the end-side 3D engine.

[0150] In some embodiments, the IDE of the cloud-side 3D engine uses a preset encoding algorithm to compress and encode the preprocessed data 1, and sends the encoded preprocessed data to the end-cloud collaborative plug-in. The end-cloud collaborative plug-in SDK uses a decoding algorithm corresponding to the preset encoding algorithm to decode the received preprocessed data, obtain the decoded preprocessed data 1, and send it to the IDE of the end-side 3D engine. The embodiments of the present application do not specifically limit the above-mentioned preset encoding algorithm.

[0151] S113 , the IDE of the terminal-side 3D engine fuses the basic rendering data 1 and the pre-processed data 1 of the high-order rendering type 1 to obtain an image 1 having a rendering effect of the high-order rendering type 1 .

[0152] It can be understood that since the scene information of the end-side scene 1 includes the scene information of the cloud-side scene 1, and for the same state data (such as camera information), the end-side scene 1 and the cloud-side scene 1 remain synchronized; therefore, the pixels in the imaging picture indicated by the basic rendering data 1 generated by the basic rendering and the pixels in the imaging picture indicated by the preprocessing data 1 of the high-order rendering can correspond one to one.

[0153] In some embodiments, the high-level rendering type 1 is GI, the rendering algorithm is DDGI, the basic rendering data 1 includes the irradiance of each pixel in the imaging picture after raster rendering, and the pre-processing data 1 includes the irradiance of each pixel in the imaging picture after DDGI rendering; see Figure 7, the module of the end-cloud collaborative plug-in SDK adds a fusion channel in the end-side rendering pipeline for consuming preprocessed data according to the extension mechanism of the end-side 3D engine rendering pipeline; the basic rendering channel outputs the basic rendering data 1 to the fusion channel. The IDE of the end-side 3D engine inputs the preprocessed data 1 received from the DDGI rendering channel on the cloud side into the DDGI rendering channel on the end side, and the DDGI rendering channel outputs the preprocessed data 1 to the fusion channel; the fusion channel outputs the fused rendering data, and the fused rendering data includes the fused irradiance, and the irradiance of the pixel at the fused coordinate 1 is equal to the product of the irradiance of the pixel at the coordinate 1 in the basic rendering data 1 and the irradiance of the pixel at the coordinate 1 in the preprocessed data 1. The above-mentioned fused rendering data is input into the conversion channel, and the conversion channel can determine the pixel value of each pixel according to the fused rendering data, thereby outputting the rendered picture (i.e., image 1).

[0154] See also Figure 7 When the network is poor, the DDGI rendering channel on the client side does not receive the pre-processing data 1 fed back by the cloud. After the basic rendering data 1 is input into the fusion channel, the fusion channel outputs the basic rendering data 1 to the conversion channel. The conversion channel can determine the pixel value of each pixel based on the basic rendering data 1, thereby outputting the basic rendering image as the rendering picture to be displayed. The basic rendering image does not have the light and shadow effects of the high-level rendering type 1. The game APP can display the above basic rendering image to ensure the normal operation of the game APP.

[0155] In some embodiments, the high-level rendering type 1 is reflection, the rendering algorithm is a reflection capture algorithm based on a reflection probe, and the basic rendering data (and the basic rendering image) include the pixel information (e.g., color) of each pixel in the imaging picture that has been raster rendered. The pre-processed data 1 for reflection includes the cubemap of the reflection probe in the cloud side scene 1; the pixel information (e.g., color) of each pixel in the imaging picture after reflection rendering is determined according to the cubemap of the reflection probe; the pixel information of the pixels at the same coordinates in the imaging picture of the basic rendering and the imaging picture of the reflection rendering are fused as the pixel information of the pixels at the same coordinates of the image 1.

[0156] In some embodiments, the pixel information (e.g., color) of each pixel in the imaged picture after reflection rendering is determined according to the cubemap of the reflection probe, including: the reflection preprocessing data 1 includes the cubemap of all reflection probes in the cloud side scene 1; one or more cubemaps to which the position of the camera belongs in the cubemap of all the reflection probes are determined according to the camera position; cubemap1 is determined after the one or more cubemaps are fused; the intersection point P of the reflection vector R after the light emitted from the camera is reflected by the ground and the bounding sphere containing cubemap1 is determined; the vector CP from the center point C of cubemap1 to the intersection point P is used as the new reflection vector R'; the cubemap1 is sampled using the reflection vector R', that is, the pixel information (e.g., color) of the corresponding pixel in the imaged picture is determined according to the texture of the intersection point of the reflection vector R' and cubemap1.

[0157] The reflection preprocessing data 1 may also include one or more cubemaps to which the position of the camera uploaded by the terminal belongs, or a cubemap after the one or more cubemaps are fused, or a cubemap of the reflection probe closest to the camera. How to calculate the pixel information of each pixel in the image after reflection rendering can be described in the above embodiment, which will not be repeated here.

[0158] S114. The developer adjusts the pre-processing related parameter 1 in the scene information of the cloud-side scene 1 through the IDE of the cloud-side 3D engine according to the high-level rendering effect presented by the image 1.

[0159] In the embodiment of the present application, if you are not satisfied with the high-level rendering effect presented in image 1, you can adjust the cloud-side scene preprocessing related parameters (such as parameter 1) on the cloud side. For example, when the developer observes that the rendering effect of virtual object 1 in image 1 does not meet the expected effect, adjust the material of the virtual object; when the developer observes that the lighting effect of GI in image 1 is not realistic enough, adjust the position information of GI probes, parameter information of GI calculation, etc.

[0160] S115. The IDE of the cloud-side 3D engine performs preprocessing of the high-order rendering type 1 according to the cloud-side scene 1 after adjusting the parameter 1, and obtains the updated preprocessing data 2.

[0161] S116. The IDE of the cloud-side 3D engine sends the adjusted parameters 1 and updated pre-processed data 2 to the end-cloud collaborative plug-in.

[0162] S117. The end-cloud collaborative plug-in sends the adjusted parameter 1 and updated pre-processed data 2 to the IDE of the end-side 3D engine.

[0163] The adjusted parameters 1 and updated pre-processed data 2 may be sent simultaneously or separately.

[0164] S118. The IDE of the terminal-side 3D engine updates parameter 1 in the scene information of the terminal-side scene 1, so that parameter 1 in the scene information of the terminal-side scene 1 and the cloud-side scene 1 remain synchronized; basic rendering is performed according to the terminal-side scene 1 after the updated parameter 1, and basic rendering data 2 is obtained.

[0165] S119, the IDE of the terminal-side 3D engine fuses the basic rendering data 2 and the pre-processing data 2 to obtain an image 2 having a rendering effect of a high-order rendering type 1.

[0166] S120. The IDE of the terminal-side 3D engine adjusts parameter 2 of the terminal-side scene 1 and / or the processing logic of consuming pre-processed data according to the rendering effect of image 2.

[0167] In the embodiment of the present application, when the high-level rendering effect presented by image 2 is not satisfactory, the relevant parameters of the end-side scene (such as parameter 2) can also be adjusted on the end side; when the rendering delay of image 2 is large, the processing logic of consuming preprocessed data can also be adjusted, such as adjusting the encoding / decoding algorithm of the preprocessed data to increase the encoding and decoding rate of the preprocessed data and reduce the rendering delay of the end-cloud collaborative rendering.

[0168] S121. The IDE of the terminal-side 3D engine performs basic rendering according to the terminal-side scene 1 after adjusting the parameter 2, and obtains basic rendering data 3.

[0169] S122 , the rendering pipeline of the IDE of the terminal-side 3D engine fuses the basic rendering data 3 and the pre-processed data 2 of the high-order rendering type 1 to obtain an image 3 having a rendering effect of the high-order rendering type 1 .

[0170] In some embodiments, when the scene information of the cloud-side scene 1 does not include parameter 2, step S122 is executed after step S121, that is, the pre-processed data 2 is fused with the basic rendering data 3 to obtain the rendered picture to be displayed. When the scene information of the cloud-side scene 1 includes parameter 2, after adjusting the parameter 2 of the terminal-side scene 1, the parameter 2 of the cloud-side scene 1 is also adjusted synchronously, and the pre-processed data of the high-order rendering is updated to pre-processed data 3 according to the adjusted cloud-side scene 1, and the basic rendering data 3 is fused with the pre-processed data 3 to obtain the rendered picture to be displayed. In one implementation, step S121 also includes A1 to A7.

[0171] A1. The IDE of the end-side 3D engine sends the adjusted parameter 2 to the end-cloud collaboration plug-in SDK, so that the parameter 2 in the scene information of the end-side scene 1 and the cloud-side scene 1 remain synchronized.

[0172] A2. The end-cloud collaboration plug-in SDK sends the adjusted parameter 2 to the end-cloud collaboration framework of the server 200.

[0173] A3. The end-cloud collaboration framework of the server 200 sends the adjusted parameter 2 to the IDE of the cloud-side 3D engine.

[0174] A4. The IDE of the cloud-side 3D engine updates parameter 2 in the scene information of the terminal-side scene 1, so that parameter 1 in the scene information of the terminal-side scene 1 and the cloud-side scene 1 remain synchronized; the updated cloud-side scene 1 is preprocessed with the high-order rendering type 1 to obtain preprocessed data 3 of the high-order rendering type 1.

[0175] A5. The IDE of the cloud-side 3D engine sends pre-processed data 3 of high-level rendering type 1 to the end-cloud collaborative plug-in.

[0176] A6. The end-cloud collaborative plug-in sends pre-processed data 3 of high-level rendering type 1 to the IDE of the end-side 3D engine.

[0177] A7. After the rendering pipeline of the IDE of the terminal 3D engine fuses the basic rendering data 3 and the pre-processed data 3 of the high-order rendering type 1, an image 4 having a rendering effect of the high-order rendering type 1 is obtained.

[0178] Some or all of the above steps S114 to S122 are optional. In some embodiments, the developer only triggers parameter adjustment on the cloud side, that is, executes steps S114 to S119, without executing S120 to S122. In some embodiments, the developer only triggers parameter adjustment on the client side, that is, executes steps S120 to S122, without executing steps S114 to S119.

[0179] The embodiment of the present application does not specifically limit the execution order of steps S114 to S119 and S120 to S122. In some embodiments, the developer may also first trigger parameter adjustment on the client side (ie, steps S114 to S119), and then trigger parameter adjustment on the cloud side (S120 to S122).

[0180] In some embodiments, after step S122, if the rendering effect of the adjusted high-level rendering image still does not meet the rendering requirements of the developer, steps S114 to S119 and steps S120 to S122 are repeated, that is, the cloud side and the terminal side are iteratively adjusted until the rendering effect meets the rendering requirements of the developer.

[0181] In some embodiments, in step S101, high-level rendering type 1 (such as GI) and end-side scene 1 are selected, and steps S106 to S122 are executed for different state data of end-side scene 1; end-side scene 1 is selected, and steps S101 to S122 are executed for different high-level rendering types; steps S101 to S122 are executed for different end-side scenes. In this way, the double-side scenes involved in the end-cloud collaborative rendering scheme are tested and adjusted, so that for any high-level rendering type, any end-side scene, and any state of the end-side scene, the end-cloud collaborative rendering scheme provided in the embodiment of the present application can achieve better high-level rendering effects.

[0182] S123, the end-cloud collaboration plug-in SDK packages the adjusted end-side scenarios and related files into a game APP.

[0183] S124. The IDE of the cloud-side 3D engine packages the adjusted cloud-side scene and related files, and uploads them to the cloud-side high-level rendering service.

[0184] Referring to the above steps S101 to S122, after the developer has debugged the rendering effects of all game scenes (i.e., the terminal-side scenes) of the game APP, the developer packages the project files of the adjusted terminal-side scenes into the terminal-side APP, such as packaging the scene information of the adjusted terminal-side scene 1, the scene ID of the terminal-side scene 1, and the high-order rendering type 1 to the application package of the game APP. The developer packages the adjusted cloud-side scene and uploads it to the cloud-side high-order rendering service, such as packaging the scene information of the cloud-side scene 1, the scene ID of the cloud-side scene 1, the correspondence between the scene ID of the terminal-side scene 1 and the high-order rendering type 1, etc. The scene management module in the cloud-side high-order rendering service is responsible for managing the uploaded cloud-side scenes, and records the correspondence between the scene IDs of various terminal-side scenes, various high-order rendering types, and the scene IDs of the cloud-side scenes.

[0185] S125 . The developer associates the cloud-side high-level rendering service with the authentication information of the game APP of the mobile device 100 . The authentication information is used to verify the legitimacy of the mobile device 100 .

[0186] In the embodiment of the present application, the authentication method and authentication information between the cloud-side high-level rendering service and the terminal-side APP are not specifically limited.

[0187] In some embodiments, the developer associates the authentication information of the cloud-side high-level rendering service with the game APP of the mobile device 100, including: the developer registers the high-level rendering service of the game APP on the Portal website, obtains a key pair, namely, public key 1 and private key 1; the game APP on the terminal side stores public key 1, and the terminal-cloud collaboration framework on the cloud side stores private key 1. The authentication information on the terminal side is public key 1, which is used to encrypt the session and verify the digital signature; the authentication information on the cloud side is private key 1 corresponding to public key 1, and private key 1 can decrypt the data encrypted by public key 1.

[0188] The game APP of the mobile device also includes a logic module. Figure 8 A method flow chart of a terminal-cloud collaborative rendering method in the running stage is shown, and the above method flow includes part or all of steps S201 to S215.

[0189] S201. The logic module of the mobile device 100 sends a rendering request to the Runtime of the end-side 3D engine. The rendering request includes the scene ID of the end-side scene 1 to be rendered and the high-order rendering type 1. The rendering request is used to request rendering based on the latest state of the end-side scene 1 to obtain an image with the rendering effect of the high-order rendering type 1.

[0190] In some embodiments, the game screen of the game scene of the game APP is refreshed at a preset refresh rate; when the logic module detects that the game screen of the current game scene is updated, it sends a rendering request to the Runtime of the end-side 3D engine to render the updated game screen; the updated game screen is displayed in a new refresh cycle. For example, when it is detected that the user switches the perspective of the game, the game APP will update the game screen according to the switched perspective. The mobile device 100 can obtain the scene information of the end-side scene 1 corresponding to the current game screen, the scene ID of the end-side scene 1, and the high-order rendering type 1 from the application package of the game APP.

[0191] S202. The end-side 3D engine Rumtime performs basic rendering based on the current scene information of the end-side scene 1 to obtain basic rendering data 4.

[0192] For details, please refer to the related description of the aforementioned step S106, which will not be repeated here.

[0193] S203. The Runtime of the end-side 3D engine sends a preprocessing request for high-order rendering type 1 to the end-cloud collaborative plug-in SDK. The preprocessing request includes the scene ID of the end-side scene 1 and the high-order rendering type 1. The preprocessing request is used to indicate that the preprocessing of high-order rendering type 1 is performed on the end-side scene 1 and the cloud-side scene corresponding to the high-order rendering type 1.

[0194] S204. The end-cloud collaboration plug-in SDK sends a pre-processing request to the end-cloud collaboration framework.

[0195] S205. The end-cloud collaboration framework performs authentication and authorization based on the received pre-processing request; after the authentication and authorization is passed, the pre-processing request is sent to the high-level rendering service.

[0196] In some embodiments, the Runtime of the end-side 3D engine sends a pre-processing request encrypted by public key 1 to the end-cloud collaboration plug-in SDK; the end-cloud collaboration framework uses private key 1 to decrypt the received pre-processing request; if the decryption is successful, the authentication is passed, and the decrypted pre-processing request is sent to the high-level rendering service. The embodiment of the present application does not specifically limit the authentication method between the high-level rendering service of the server 200 and the game APP of the mobile device 100.

[0197] S206. The high-level rendering service performs preprocessing preparation, including loading cloud-side scene 1 and preparing a rendering pipeline of high-level rendering type 1. The cloud-side scene 1 is determined according to the scene ID of the terminal-side scene 1 in the preprocessing request and the high-level rendering type 1.

[0198] In some embodiments, the high-level rendering service in step S124 is packaged with the scene ID of the terminal scene, the correspondence between the high-level rendering type and the scene ID of the cloud scene, for example, the scene ID of the terminal scene 1, the correspondence between the high-level rendering type 1 and the scene ID of the cloud scene 1, and the scene information of the cloud scene; according to the above correspondence, it can be determined that the cloud-side scene 1 is the cloud-side scene after the terminal scene 1 is converted to the high-level rendering type 1, that is, the scene to be rendered.

[0199] In some embodiments, the preprocessing request may further include the game ID of the game to which the terminal scene 1 belongs. The high-level rendering service may record the scene IDs of the terminal scenes of each game according to the game ID classification, so that the high-level rendering service can quickly locate the terminal scene 1 and the cloud scene 1 corresponding to the terminal scene 1.

[0200] S207. The high-level rendering service sends a confirmation message to the end-cloud collaborative plug-in SDK, where the confirmation message is used to indicate that the preprocessing preparation has been completed.

[0201] S208. The end-cloud collaborative plug-in SDK sends status synchronization data 2 related to preprocessing of high-level rendering type 1 to the high-level rendering service.

[0202] Specifically, the state synchronization data 2 may refer to the related description of the state synchronization data 1, which will not be repeated here.

[0203] S209 . The high-level rendering service updates the cloud-side scene 1 according to the received status synchronization data 2 , and performs preprocessing of the high-level rendering type 1 based on the scene information of the updated cloud-side scene 1 to obtain preprocessing data 4 .

[0204] For details, please refer to the relevant description of the aforementioned step S110, which will not be repeated here. In the embodiment of the present application, the high-level rendering service only performs preprocessing of the high-level rendering and does not run the full game APP; compared with the aforementioned method 1, the consumption of cloud-side computing resources is reduced.

[0205] S210. The high-level rendering service sends pre-processed data 4 related to the end-side user to the end-cloud collaborative framework.

[0206] In some embodiments, in step S209, the high-level rendering service performs the preprocessing of the high-level rendering type 1, and can obtain preprocessed data within all viewing angles of the camera. The high-level rendering service crops the preprocessed data according to the current viewing angle of the camera in the state synchronization data 2 uploaded by the mobile device 100, obtains the preprocessed data 4 within the current viewing angle of the camera, that is, the preprocessed data related to the end-side user of the mobile device 100, and sends the preprocessed data 4 to the end-cloud collaborative framework in step S210. In some embodiments, the preprocessed data 4 generated in step S209 is the preprocessed data within the current viewing angle of the camera, and no cropping is required.

[0207] S211, the end-cloud collaborative framework compresses and encodes the pre-processed data 4 to obtain the encoded data 1.

[0208] S212. The end-cloud collaboration framework sends the encoded data 1 to the end-cloud collaboration plug-in SDK.

[0209] S213. The end-cloud collaborative plug-in SDK decodes the encoded data 1 to obtain pre-processed data 4.

[0210] In some embodiments, the IDE of the cloud-side 3D engine uses a preset encoding algorithm to compress and encode the preprocessed data 4 to obtain the encoded data 1; then, the encoded data 1 is encapsulated using a preset communication protocol and sent to the end-cloud collaborative plug-in SDK. The end-cloud collaborative plug-in decapsulates the received data to obtain the encoded data 1; the end-cloud collaborative plug-in decodes the received encoded data 1 using a preset encoding algorithm to obtain the decoded preprocessed data 4. The embodiments of the present application do not specifically limit the above-mentioned preset encoding algorithm and preset communication protocol. For example, the preset encoding algorithm is the HEVC algorithm, and the preset communication protocol is the UDP protocol.

[0211] In the embodiment of the present application, the data transmitted from the cloud side to the terminal side is pre-processed data after compression encoding, and the transmission bandwidth required is lower than that of the video stream data transmitted by the aforementioned method one.

[0212] S214, the end-cloud collaborative plug-in SDK sends pre-processed data 4 to the Runtime of the end-side 3D engine.

[0213] S215 , the Runtime of the terminal-side 3D engine fuses the basic rendering data 4 and the pre-processing data 4 to obtain an image 4 having a rendering effect of the high-order rendering type 1 .

[0214] For details, please refer to the related description of the aforementioned step S113, which will not be repeated here.

[0215] In some embodiments, in step S201, the above-mentioned rendering request and preprocessing request may also include multiple high-order rendering types. The server 200 may perform preprocessing on the cloud-side scene 1 according to each of the multiple high-order rendering types in the preprocessing request, and feedback the preprocessing data of each high-order rendering type to the mobile device 100; the mobile device 100 may fuse the basic rendering data and the preprocessing data of the above-mentioned multiple high-order rendering types to generate an image with the light and shadow effects of the above-mentioned multiple high-order rendering types.

[0216] In the embodiment of the present application, the end-side game APP only needs to have the ability to perform basic rendering of 3D scenes. The game APP implements high-level rendering of 3D game scenes by calling the end-cloud collaborative plug-in SDK; after the basic rendering data is integrated with the pre-processed data of the high-level rendering, the final rendered image presents the light and shadow effects of the high-level rendering, such as the light and shadow effects that can be achieved by ray tracing. In this way, 3D applications (such as game APPs) running on mobile devices with weak GPU capabilities have high-level rendering capabilities (such as ray tracing capabilities), which effectively improves the user experience.

[0217] Exemplarily, the terminal scene 1 to be rendered is a multi-player game scene of a game APP, and multiple game players simultaneously participate in the same copy of the game scene, such as user 1 of mobile device 100, and user 2 of mobile device 300. If the lighting and shadow effects of the high-order rendering type 1 (such as GI) of the current scene are independent of the user's perspective, the pre-processed data of the high-order rendering type 1 can be shared with other players of the game for other players to achieve high-order rendering effects on the terminal side. It can be understood that for the latest state of the terminal scene 1, the server 200 does not need to perform pre-processing for each user for the high-order rendering type that is independent of the perspective, but only needs to perform pre-processing once and share it with other users. In this way, the consumption of cloud-side computing resources is further reduced in the terminal-cloud collaborative rendering scheme.

[0218] In some embodiments, the pre-processing request may also include the copy ID of the game copy currently running in the terminal scene 1. Based on the game ID, scene ID and copy ID, other players in the same copy as the terminal user of the mobile device 100 may be queried.

[0219] In some embodiments, the preprocessing of the high-level rendering type 1 of the terminal scene 1 in the latest state can be perspective-independent processing or perspective-dependent processing. Perspective-independent processing refers to processing that is independent of the current perspective of the terminal user (i.e., the perspective of the aforementioned camera); for example, in the latest state of the game scene, even if the perspectives of user 1 and user 2 in the game scene are different, the light and shadow effects of the high-level rendering type 1 in the game screen seen by user 1 and user 2 are the same, so the preprocessing of the high-level rendering type 1 is also the same. Perspective-dependent processing refers to processing related to the current perspective of the terminal user of the mobile device 100 (i.e., the perspective of the aforementioned camera); for example, in the latest state of the game scene, although user 1 and user 2 are in the same copy of the same game scene, due to different user perspectives, user 1 and user 2 see different light and shadow effects of the high-level rendering type 1 in the game screen, so the high-level rendering of the high-level rendering type 1 is also different.

[0220] In some embodiments, the method further includes: the Runtime of the end-side 3D engine of the mobile device 200 performs basic rendering of the end-side scene 1 to obtain basic rendering data 5; the high-level rendering service of the server 200 determines whether the preprocessing of the high-level rendering type 1 of the cloud-side scene 1 is view-independent processing; if so, the preprocessing data of the high-level rendering type 1 can be shared with other users who also participate in the end-side scene 1, such as user 2, that is, the end-cloud collaboration framework sends the encoded preprocessing data 4 to the end-cloud collaboration plug-in SDK of the mobile device 200. The end-cloud collaboration plug-in SDK of the mobile device 200 sends the decoded preprocessing data 4 to the Runtime of the end-side 3D engine, and the Runtime of the end-side 3D engine fuses the basic rendering data 5 and the preprocessing data 4 to obtain an image 5 with the rendering effect of the high-level rendering type 1.

[0221] In the second implementation scheme provided in the embodiment of the present application, the native end-side 3D engine supports the above-mentioned end-cloud collaborative rendering method.

[0222] For example, Fig. 9 The following is a schematic diagram of the system architecture of a rendering system for end-cloud collaboration involved in implementing Solution 2. Figure 5 The system architecture of deploying the end-cloud collaboration plug-in SDK is shown. Fig. 9 The main differences of the end-cloud collaborative rendering system shown are:

[0223] (1) Figure 5The first implementation scheme shown is to provide the end-to-end 3D engine with end-to-end collaborative rendering capabilities through the end-to-end collaborative plug-in SDK; the end-to-end collaborative plug-in SDK can be obtained through the application store download and browser download, and developers can clearly perceive the existence of the SDK. Fig. 9 The second implementation scheme shown is that the end-side 3D engine natively supports the above-mentioned end-cloud collaborative rendering method; the functions in the end-cloud collaborative plug-in SDK can be built into the end-side 3D engine in the form of modules / pre-set plug-ins, and developers do not need to download and install the function through the application store or other channels.

[0224] (2) Figure 5 In the implementation scheme 1 shown, the 3D engine on the device side is usually different from the 3D engine on the cloud side, so it is necessary to provide scene conversion capabilities in the device-cloud collaboration plug-in SDK. Fig. 9 In the implementation scheme 2 shown, when the cloud-side 3D engine is consistent with the terminal-side 3D engine, the scene conversion function module in the terminal-cloud collaboration module may no longer be needed. It should be noted that even if there is no scene conversion in the implementation scheme 2, the cloud-side scene and the terminal-side scene may not be completely consistent. For example, the cloud-side scene reduces the accuracy of the mesh information.

[0225] (3) Figure 5 In the implementation scheme 1 shown, it is necessary to add the consumption rendering capability of pre-processed data on the basis of the original rendering pipeline, so it is necessary to provide pipeline adaptation capability in the end-cloud collaboration plug-in SDK. For example, see Figure 7 The pipeline adaptation function module adds a fusion channel to the end-side rendering pipeline to consume pre-processed data according to the extension mechanism of the end-side 3D engine rendering pipeline. Fig. 9 In the implementation scheme 2 shown in the figure, the end-cloud collaboration module of the end-side 3D engine can provide a native end-cloud collaborative rendering pipeline, which itself can have the ability to consume pre-processed data; in this way, there is no need Figure 7 The functional modules of the pipeline adaptation are shown.

[0226] In the second implementation scheme of the present application, the method flow in the development phase and the method flow in the operation phase are similar to those of the aforementioned implementation scheme one and will not be repeated here.

[0227] In the embodiment of the present application, the first electronic device may be the aforementioned mobile device 100, and the second electronic device may be the aforementioned mobile device 300; the first request may be the aforementioned pre-processing request; the first application may be the aforementioned 3D application (e.g., a game APP); the first terminal-side scene may be the aforementioned terminal-side scene 1, the first high-order rendering type may be the aforementioned high-order rendering type 1, and the first cloud-side scene may be the aforementioned cloud-side scene 1; the second high-order rendering type may be the aforementioned high-order rendering type 2, and the second cloud-side scene may be the aforementioned cloud-side scene 2; the first basic rendering data may be the aforementioned basic rendering data 4, the first state data may be the aforementioned state synchronization data 2, and the first pre-processing data may be the aforementioned pre-processing data 4; the first image may be the aforementioned image 4. The first coordinate may be the aforementioned coordinate 1; the third basic rendering data may be the aforementioned basic rendering data 5, and the third image may be the aforementioned image 5

[0228] The structure of a mobile device 100 provided in an embodiment of the present application is introduced below. Fig.10 A schematic structural diagram of a mobile device 100 is shown.

[0229] The mobile device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0230] It is understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the mobile device 100. In other embodiments of the present application, the mobile device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0231] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0232] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0233] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0234] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0235] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple groups of I2C buses. The processor 110 may be coupled to the touch sensor 180K, the charger, the flash, the camera 193, etc. through different I2C bus interfaces. For example, the processor 110 may be coupled to the touch sensor 180K through the I2C interface, so that the processor 110 communicates with the touch sensor 180K through the I2C bus interface, thereby realizing the touch function of the mobile device 100.

[0236] The I2S interface can be used for audio communication. In some embodiments, the processor 110 can include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to achieve communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 can transmit an audio signal to the wireless communication module 160 via the I2S interface to achieve the function of answering a call through a Bluetooth headset.

[0237] The PCM interface can also be used for audio communication, sampling, quantizing and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 can be coupled via a PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 via the PCM interface to realize the function of answering calls via a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0238] The UART interface is a universal serial data bus for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is generally used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 through the UART interface to implement the Bluetooth function. In some embodiments, the audio module 170 can transmit an audio signal to the wireless communication module 160 through the UART interface to implement the function of playing music through a Bluetooth headset.

[0239] The MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to implement the shooting function of the mobile device 100. The processor 110 and the display screen 194 communicate via the DSI interface to implement the display function of the mobile device 100.

[0240] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 with the camera 193, the display 194, the wireless communication module 160, the audio module 170, the sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0241] The USB interface 130 is an interface that complies with the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the mobile device 100, and can also be used to transfer data between the mobile device 100 and peripheral devices. It can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other electronic devices, such as AR devices, etc.

[0242] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present invention is only a schematic illustration and does not constitute a structural limitation on the mobile device 100. In other embodiments of the present application, the mobile device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0243] The charging management module 140 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through a wireless charging coil of the mobile device 100. While the charging management module 140 is charging the battery 142, it may also power the electronic device through the power management module 141.

[0244] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the display screen 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0245] The wireless communication function of the mobile device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0246] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in mobile device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0247] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the mobile device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0248] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0249] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the mobile device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, demodulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0250] In some embodiments, the antenna 1 of the mobile device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the mobile device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0251] The mobile device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0252] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the mobile device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0253] The mobile device 100 can implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor.

[0254] The ISP is used to process the data fed back by the camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also perform algorithm optimization on the noise and brightness of the image. The ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0255] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the mobile device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0256] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the mobile device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0257] Video codecs are used to compress or decompress digital videos. The mobile device 100 may support one or more video codecs. Thus, the mobile device 100 may play or record videos in a variety of coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0258] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of mobile device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0259] The internal memory 121 may include one or more random access memories (RAM) and one or more non-volatile memories (NVM).

[0260] Random access memory may include static random-access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM, for example, the fifth generation DDR SDRAM is generally referred to as DDR5 SDRAM), etc.; non-volatile memory may include disk storage devices and flash memory.

[0261] Flash memory can be divided into NOR FLASH, NAND FLASH, 3D NAND FLASH, etc. according to the operating principle; single-level cell (SLC), multi-level cell (MLC), triple-level cell (TLC), quad-level cell (QLC), etc. according to the storage unit potential level; universal flash storage (UFS), embedded multi media card (eMMC), etc. according to the storage specification.

[0262] The random access memory can be directly read and written by the processor 110, and can be used to store executable programs (such as machine instructions) of the operating system or other running programs, and can also be used to store user and application data.

[0263] The non-volatile memory may also store executable programs and user and application data, etc., and may be loaded into the random access memory in advance for direct reading and writing by the processor 110 .

[0264] The external memory interface 120 can be used to connect to an external non-volatile memory to expand the storage capacity of the mobile device 100. The external non-volatile memory communicates with the processor 110 through the external memory interface 120 to implement a data storage function. For example, files such as music and videos are stored in the external non-volatile memory.

[0265] The mobile device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0266] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

[0267] The speaker 170A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The mobile device 100 can listen to music or listen to a hands-free call through the speaker 170A.

[0268] The receiver 170B, also called a "earpiece", is used to convert audio electrical signals into sound signals. When the mobile device 100 receives a call or voice message, the voice can be received by placing the receiver 170B close to the ear.

[0269] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by approaching the microphone 170C with his mouth to input the sound signal into the microphone 170C.

[0270] The earphone interface 170D is used to connect a wired earphone and can be a USB interface 130 or a 3.5 mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0271] The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be disposed on the display screen 194 .

[0272] The gyro sensor 180B may be used to determine the motion posture of the mobile device 100. In some embodiments, the angular velocity of the mobile device 100 around three axes (ie, x, y, and z axes) may be determined by the gyro sensor 180B.

[0273] The air pressure sensor 180C is used to measure air pressure. In some embodiments, the mobile device 100 calculates the altitude through the air pressure value measured by the air pressure sensor 180C to assist positioning and navigation.

[0274] The magnetic sensor 180D includes a Hall sensor, and the mobile device 100 can detect the opening and closing of the flip leather case using the magnetic sensor 180D.

[0275] The acceleration sensor 180E can detect the magnitude of the acceleration of the mobile device 100 in various directions (generally three axes).

[0276] The distance sensor 180F is used to measure the distance. The mobile device 100 can measure the distance by infrared or laser.

[0277] The proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector such as a photodiode.

[0278] The ambient light sensor 180L is used to sense the brightness of the ambient light. The mobile device 100 can adaptively adjust the brightness of the display screen 194 according to the sensed brightness of the ambient light.

[0279] The fingerprint sensor 180H is used to collect fingerprints. The mobile device 100 can use the collected fingerprint characteristics to implement fingerprint unlocking, access application locks, fingerprint photography, fingerprint call answering, etc.

[0280] The temperature sensor 180J is used to detect temperature. In some embodiments, the mobile device 100 uses the temperature detected by the temperature sensor 180J to execute a temperature processing strategy.

[0281] The touch sensor 180K is also called a "touch control device". The touch sensor 180K can be set on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, also called a "touch control screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the mobile device 100, which is different from the position of the display screen 194.

[0282] The bone conduction sensor 180M can obtain a vibration signal. In some embodiments, the bone conduction sensor 180M can obtain a vibration signal of a vibrating bone block of a human vocal part.

[0283] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The mobile device 100 may receive key input and generate key signal input related to user settings and function control of the mobile device 100.

[0284] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback.

[0285] Indicator 192 may be an indicator light, which may be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

[0286] The SIM card interface 195 is used to connect a SIM card.

[0287] The following introduces the structure of a server 200 provided in an embodiment of the present application. Fig.11 The structure of a server 200 provided in an embodiment of the present application is exemplified.

[0288] like Fig.11As shown, the server 200 may include: one or more processors 1001, a memory 1002, a communication interface 1003, a transmitter 1005, a receiver 1006, a coupler 1007, and an antenna 1008. These components may be connected via a bus 1004 or other means. Fig.11 Take bus connection as an example.

[0289] The communication interface 1003 can be used for the server 200 to communicate with other communication devices, such as the terminal device 100. Specifically, the communication interface 1003 can be a 3G communication interface, a 4G communication interface, a 5G communication interface, or a communication interface of a future new air interface. Not limited to a wireless communication interface, the server 200 can also be configured with a wired communication interface 1003, such as a local access network (LAN) interface. The transmitter 1005 can be used to transmit and process the signal output by the processor 1001. The receiver 1006 can be used to receive and process the mobile communication signal received by the antenna 1008.

[0290] In some embodiments of the present application, the transmitter 1005 and the receiver 1006 can be regarded as a wireless modem. In the server 200, the number of the transmitter 1005 and the receiver 1006 can be one or more. The antenna 1008 can be used to convert the electromagnetic energy in the transmission line into electromagnetic waves in the free space, or convert the electromagnetic waves in the free space into electromagnetic energy in the transmission line. The coupler 1007 is used to divide the mobile communication signal received by the antenna 1008 into multiple paths and distribute them to multiple receivers 1006.

[0291] The memory 1002 is coupled to the processor 1001 and is used to store various software programs and / or multiple sets of instructions. Specifically, the memory 1002 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more disk storage devices, flash memory devices or other non-volatile solid-state storage devices. The memory 1002 may store a network communication program, which may be used to communicate with one or more additional devices, one or more terminal devices, or one or more network devices.

[0292] In some embodiments of the present application, the memory 1002 may be used to store an implementation program of the application distribution method provided by one or more embodiments of the present application on the server 200 side. For the implementation of the application distribution method provided by one or more embodiments of the present application, please refer to the above embodiments.

[0293] The processor 1001 can be used to read and execute computer-readable instructions. Specifically, the processor 1001 can be used to call a program stored in the memory 1002, such as an implementation program of the application distribution method provided in one or more embodiments of the present application on the server 200 side, and execute the instructions contained in the program.

[0294] It should be noted that Fig.11 The server 200 shown is only one implementation of the embodiment of the present application. In actual applications, the server 200 may also include more or fewer components, which is not limited here.

[0295] For more details about the functions and working principles of the server 200, please refer to the relevant contents in the above embodiments, which will not be repeated here.

[0296] The various implementation modes of the present application can be combined arbitrarily to achieve different technical effects.

[0297] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state disk (SSD)), etc.

[0298] Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.

[0299] In short, the above description is only an embodiment of the technical solution of the present invention, and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made according to the disclosure of the present invention shall be included in the protection scope of the present invention.

Claims

1. A rendering method for end-cloud collaboration, characterized in that: include: The first electronic device performs basic rendering on the first terminal side scene to obtain first basic rendering data; The first electronic device stores at least one terminal side scenario, and the first terminal side scenario is any one of the at least one terminal side scenario; The first electronic device sends a scene identifier and a first high-level rendering type of the first terminal-side scene to the server; The first electronic device sends first status data to the server, where the first status data includes data related to preprocessing of the first high-order rendering type in the status data of the first terminal side scene; The first electronic device receives the first pre-processed data sent by the server; The first pre-processed data is obtained by performing pre-processing of the first high-level rendering type on the first cloud-side scene after the server updates the first cloud-side scene according to the first state data; the server stores the first cloud-side scene converted from the first terminal-side scene for the first high-level rendering type; The first electronic device acquires a first image according to the first basic rendering data and the first pre-processed data, wherein the first image has a rendering effect of the first high-order rendering type; The first electronic device displays the first image.

2. The method according to claim 1, characterized in that The method further comprises: The first electronic device updates the scene information of the first terminal side scene; The first electronic device performs basic rendering on the updated first terminal-side scene to obtain second basic rendering data; When the network speed between the first electronic device and the server is lower than a preset value or the network between the first electronic device and the server is disconnected, the first electronic device determines a second image according to the second basic rendering data; The first electronic device displays the second image.

3. The method according to claim 1, characterized in that The first electronic device sends the scene identifier and the first high-order rendering type of the first terminal-side scene to the server, including: The first electronic device sends a first request to the server, the first request including a scene identifier of the first terminal-side scene and a first high-order rendering type; the first request is used to trigger the server to perform preprocessing preparation, the preprocessing preparation including loading a first cloud-side scene and preparing a rendering pipeline for preprocessing of the first high-order rendering type; Before the first electronic device sends the first status data to the server, the method further includes: The first electronic device receives confirmation information sent by the server, where the confirmation information is used to indicate that the preprocessing preparation has been completed; The first electronic device sending first status data to the server includes: Based on the determination information, the first electronic device sends first status data to the server.

4. The method according to any one of claims 1 to 3, characterized in that: The state data of the scene on the first end side includes part or all of the following: scene identification information, light source information, character information, camera information and scene update information.

5. The method according to any one of claims 1 to 4, characterized in that: The server stores a first correspondence between the first terminal-side scene, the first high-level rendering type, and the first cloud-side scene.

6. The method according to any one of claims 1 to 5, characterized in that: The scene identifier of the first terminal side scene and the first high-level rendering type are obtained by the first electronic device from the application package of the first application; The scene identifier of the first cloud-side scene, and the first corresponding relationship between the first terminal-side scene, the first high-order rendering type, and the first cloud-side scene are deployed on the server during the application development of the first application.

7. The method according to any one of claims 1 to 6, characterized in that: The server stores a second cloud-side scene converted from the first terminal-side scene for a second high-level rendering type.

8. The method according to any one of claims 1 to 7, characterized in that: The first electronic device receives the first pre-processed data sent by the server, including: The first electronic device receives the first pre-processed data after compression encoding sent by the server; Before the first electronic device acquires the first image according to the first basic rendering data and the first pre-processing data, the method further includes: The first electronic device decodes the received first pre-processed data to obtain the decoded first pre-processed data.

9. The method according to any one of claims 1 to 8, characterized in that: The first high-level rendering type is global illumination; The first pre-processed data includes the irradiance of each pixel in the imaging picture obtained by pre-processing the global illumination, and the first basic rendering data includes the irradiance of each pixel in the imaging picture after basic rendering; The irradiance of the pixel at the first coordinate in the first image is equal to the product of the irradiance of the pixel at the first coordinate in the first preprocessing data and the first basic rendering data, and the pixel value of the pixel at the first coordinate in the first image is determined according to the irradiance of the pixel at the first coordinate.

10. A rendering method for end-cloud collaboration, characterized in that: The method comprises: The server receives a scene identifier and a first high-level rendering type of the first terminal-side scene sent by the first electronic device; The server receives first status data sent by the first electronic device, where the first status data includes data related to preprocessing of the first high-order rendering type in status data of the first terminal-side scene; The server updates the first cloud-side scene according to the first state data, performs preprocessing of the first high-order rendering type on the updated first cloud-side scene, and obtains first preprocessing data; The server sends the first pre-processed data to the first electronic device; The first pre-processed data is used to fuse the first basic rendering data to obtain a first image; the first image has the light and shadow effects of the first high-order rendering type; the first basic rendering data is obtained by the first electronic device performing basic rendering on the first end-side scene.

11. The method according to claim 10, characterized in that The server receives the scene identifier and the first high-order rendering type of the first terminal-side scene sent by the first electronic device, including: The server receives a first request sent by the first electronic device, where the first request includes a scene identifier of the first terminal-side scene and a first high-level rendering type; Before the server receives the first status data sent by the first electronic device, the method further includes: Based on the first request, the server performs preprocessing preparation, wherein the preprocessing preparation includes loading a first cloud-side scene and preparing a preprocessing rendering pipeline for the first high-order rendering type; The server sends confirmation information to the first electronic device, where the confirmation information is used to indicate that the preprocessing preparation has been completed; The first status data is sent by the first electronic device based on the determination information.

12. The method according to claim 10 or 11, characterized in that: The state data of the scene on the first end side includes part or all of the following: scene identification information, light source information, character information, camera information and scene update information.

13. The method according to any one of claims 10 to 12, characterized in that: The server stores a first correspondence between the first terminal-side scene, the first high-level rendering type, and the first cloud-side scene. Before the server updates the first cloud-side scene according to the first status data, the method further includes: The server determines, based on the first corresponding relationship, the scene identifier of the first terminal-side scene, and the first high-order rendering type, that the cloud-side scene to be rendered is the first cloud-side scene.

14. The method according to any one of claims 11 to 13, characterized in that: The scene identifier of the first terminal side scene and the first high-level rendering type are obtained by the first electronic device from the application package of the first application; The scene identifier of the first cloud-side scene, and the first corresponding relationship between the first terminal-side scene, the first high-order rendering type, and the first cloud-side scene are deployed on the server during the application development of the first application.

15. The method according to any one of claims 10 to 14, characterized in that The server stores a second cloud-side scene converted from the first terminal-side scene for a second high-level rendering type.

16. The method according to any one of claims 10 to 15, characterized in that The server sending the first pre-processed data to the first electronic device includes: The server sends the first pre-processed data after compression encoding to the first electronic device.

17. The method according to any one of claims 10 to 16, characterized in that: The first high-level rendering type is global illumination; The first pre-processed data includes the irradiance of each pixel in the imaging picture obtained by pre-processing the global illumination, and the first basic rendering data includes the irradiance of each pixel in the imaging picture after basic rendering; The irradiance of the pixel at the first coordinate in the first image is equal to the product of the irradiance of the pixel at the first coordinate in the first preprocessing data and the first basic rendering data, and the pixel value of the pixel at the first coordinate in the first image is determined according to the irradiance of the pixel at the first coordinate.

18. The method according to any one of claims 10 to 16, characterized in that The method further comprises: If the first preprocessed data is perspective-independent preprocessed data, the server sends the first preprocessed data to the second electronic device; the first electronic device and the second electronic device are electronic devices of users participating in the same copy of the first end-side scene.

19. An electronic device, characterized in that: include: One or more processors, one or more memories; the one or more memories are coupled to the one or more processors; The one or more memories are used to store a computer program, and when the computer program runs on the processor, the electronic device executes the method according to any one of claims 1 to 9.

20. A server, characterized in that: include: One or more processors, one or more memories; the one or more memories are coupled to the one or more processors; The one or more memories are used to store a computer program, which, when executed on the processor, enables the server to perform the method according to any one of claims 10 to 18.

21. A computer readable medium, characterized in that A computer program is stored, and the computer program can be executed by a processor to implement the method according to any one of claims 1 to 18.

Images