End-cloud collaborative rendering method and related device

Abstract

The invention discloses an end-cloud collaborative rendering method, and the method comprises the steps: carrying out the basic rendering of a first end-side scene through electronic equipment, and obtaining first basic rendering data; the electronic equipment sends a scene identifier of the first end side scene and a first high-order rendering type to a server; the server stores a first cloud side scene converted from the first end side scene for the first high-order rendering type; the electronic equipment sends the first state data to a server; after updating the first cloud side scene according to the first state data, the server executes preprocessing of a first high-order rendering type on the first cloud side scene to obtain first preprocessing data; the server sends the first preprocessing data to the electronic equipment; and the electronic equipment obtains a first image according to the first basic rendering data and the first preprocessing data, and displays the first image. Therefore, the computing power requirement and the load of the end-side GPU are reduced, the electronic equipment with relatively weak computing power of the GPU can also present a high-order rendering effect, and the user experience is effectively improved.

Description

End-cloud collaborative rendering method and related device

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on July 27, 2023, with application number 202310940383.X, and priority to the Chinese patent application entitled “End-cloud collaborative rendering method and related device”, all contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of electronic technology, and in particular to a rendering method and related devices for end-cloud collaboration. Background Art

[0003] There are an increasing number of three-dimensional (3D) applications (such as 3D games and 3D modeling software) on mobile devices and other terminal devices. Providing rendering capabilities for these 3D applications is the most core function of terminal devices to support the operation of 3D applications. However, the graphics processing unit (GPU) (for ease of description, referred to as lightweight GPU) of terminal devices with limited size and power consumption (such as mobile devices) is significantly behind that of personal computer (PC)-level GPUs. For example, the single-precision computing power of Apple A15 in 2021 was 1.5T FLOPS; while the single-precision computing power of NVIDA GTX 580 in 2010 had reached 1.58T FLOPS; and the computing power of NVIDIA RTX 3080Ti in 2021 reached 27.955TFLOPS. This shows that the computing power development of lightweight GPUs for mobile devices is about 10 years behind that of PC-level GPUs.

[0004] Cloud-side server GPUs (i.e., PC-class GPUs) offer a significant computing power advantage over lightweight GPUs in mobile devices. With the development of cloud computing, shifting computing power from devices to the cloud and collaborating with the cloud to process device-side services is becoming a key technology trend. Currently, research is underway to achieve efficient rendering processing in collaboration with the cloud, thereby achieving the same rendering effects as ray tracing technology on the device side.

[0005] Summary of the Invention

[0006] This application provides an end-cloud collaborative rendering method and related devices, which reduce the computing power requirements and load on the end-side GPU, and enable electronic devices with weaker GPU computing power to present high-level rendering effects, such as the rendering effects of ray tracing technology, effectively improving the user experience.

[0007] In a first aspect, the present application provides a rendering method for end-cloud collaboration, which is applied to a communication system, wherein the communication system includes a first electronic device and a server, the first electronic device stores at least one end-side scene of a first application, the first end-side scene is any one of the at least one end-side scene, and the server stores a first cloud-side scene converted from the first end-side scene for a first high-order rendering type; the method includes: the first electronic device performs basic rendering on the first end-side scene to obtain first basic rendering data; the first electronic device sends a scene identifier and a first high-order rendering type of the first end-side scene to the server; the first electronic device sends first status data to the server, the first status data including data related to preprocessing of the first high-order rendering type in the status data of the first end-side scene; the server updates the first cloud-side scene according to the first status data, performs preprocessing of the first high-order rendering type on the updated first cloud-side scene, and obtains first preprocessing data; the server sends the first preprocessing data to the first electronic device; the first electronic device obtains a first image according to the first basic rendering data and the first preprocessing data, and the first image has the light and shadow effects of the first high-order rendering type; the first electronic device displays the first image.

[0008] In implementing the embodiment of the present application, a terminal-side scene is converted into a cloud-side scene for each high-order rendering type, and the server stores the cloud-side scene; for a specific high-order rendering type, the electronic device only needs to have the ability to perform basic rendering on the terminal-side scene, and the server implements the high-order rendering preprocessing of the cloud-side scene corresponding to the terminal-side scene; after the basic rendering data is integrated with the preprocessing data of the high-order rendering, the final rendered image presents the light and shadow effects of the above-mentioned specific high-order rendering type. In this way, for various high-order rendering types, the computing power requirements and load on the terminal-side GPU can be reduced; electronic devices (such as mobile phones, tablets, etc.) do not need to have high-standard hardware performance to present the rendering effects of high-order rendering. In addition, since the server only performs preprocessing of high-order rendering and does not run the full APP, the consumption of cloud-side computing resources is reduced.

[0009] In one implementation, the method further includes: the first electronic device updates scene information of the first end-side scene; the first electronic device performs basic rendering on the updated first end-side scene to obtain second basic rendering data; when the network speed between the first electronic device and the server is lower than a preset value, or when the network between the first electronic device and the server is disconnected, the first electronic device determines a second image based on the second basic rendering data; and the first electronic device displays the second image. By implementing the embodiment of the present application, even when network conditions are poor, the electronic device can still perform basic rendering, thereby ensuring the normal operation of the game app and thus guaranteeing the user's basic operating experience.

[0010] In one implementation, a first electronic device sends a scene identifier and a first high-order rendering type of a first end-side scene to a server, including: the first electronic device sends a first request to the server, the first request including the scene identifier and the first high-order rendering type of the first end-side scene; before the first electronic device sends first status data to the server, the method further includes: based on the first request, the server performs preprocessing preparation, the preprocessing preparation including loading the first cloud-side scene and preparing a rendering pipeline for preprocessing of the first high-order rendering type; the server sends confirmation information to the first electronic device, the confirmation information is used to indicate that the preprocessing preparation has been completed; the first electronic device sends the first status data to the server, including: based on the confirmation information, the first electronic device sends the first status data to the server. In implementing the embodiment of the present application, the status data is uploaded to the server only after confirming that the server has completed the preprocessing preparation; thus avoiding invalid upload of status data and waste of network traffic caused by the server's inability to perform high-order rendering.

[0011] In one implementation, the state data of the scene on the first terminal side includes part or all of the following: scene identification information, light source information, character information, camera information, and scene update information.

[0012] In one implementation, the server stores a first correspondence between a first terminal-side scene, a first high-order rendering type, and a first cloud-side scene. Before the server updates the first cloud-side scene according to the first status data, the method further includes: the server determines that the cloud-side scene to be rendered is the first cloud-side scene based on the first correspondence, as well as the scene identifier and the first high-order rendering type of the first terminal-side scene. In implementing the embodiment of the present application, the server stores a correspondence between the first terminal-side scene, the first high-order rendering type, and the first cloud-side scene, so as to quickly locate the cloud-side scene to be rendered based on the scene identifier and the first high-order rendering type of the first terminal-side scene uploaded by the terminal.

[0013] In one implementation, the scene identifier and the first high-order rendering type of the first end-side scene are obtained by the first electronic device from the application package of the first application; the scene identifier of the first cloud-side scene, and the first correspondence between the first end-side scene, the first high-order rendering type and the first cloud-side scene are deployed on the server during the application development of the first application.

[0014] In one implementation, the server stores a second cloud-side scene converted from the first device-side scene for the second high-level rendering type. In this embodiment of the present application, one device-side scene is converted into one cloud-side scene for each high-level rendering type.

[0015] In one implementation, the server sending the first preprocessed data to the first electronic device includes: the server sending the compressed and encoded first preprocessed data to the first electronic device; and before the first electronic device obtains the first image based on the first basic rendering data and the first preprocessed data, the first electronic device decoding the received first preprocessed data to obtain the decoded first preprocessed data. In implementing this embodiment of the present application, the server feeding back the compressed and encoded preprocessed data to the first electronic device can reduce the amount of transmitted data and thereby reduce transmission latency.

[0016] In one implementation, the first high-order rendering type is global illumination; the first preprocessing data includes the irradiance of each pixel in the imaging picture obtained by preprocessing the global illumination, and the first preprocessing data includes the irradiance of each pixel in the imaging picture after basic rendering; the irradiance of the pixel at the first coordinate in the first image is equal to the product of the irradiance of the pixel at the first coordinate in the first imaging picture and the first basic rendering data, and the pixel value of the pixel at the first coordinate in the first image is determined based on the irradiance of the pixel at the first coordinate.

[0017] In one implementation, the system further includes a second electronic device, the first electronic device and the second electronic device being electronic devices of users participating in the same replica of the first end-side scene; the method further includes: if the first pre-processed data is perspective-independent pre-processed data, the server sending the first pre-processed data to the second electronic device. In implementing this embodiment of the present application, if multiple users are in the same replica of the end-side scene, the server can share the perspective-independent pre-processed data in the replica with all users in the replica, thereby reducing the consumption of cloud-side computing resources.

[0018] In one implementation, the method also includes: the second electronic device performs basic rendering on the first end-side scene to obtain third basic rendering data; the second electronic device obtains a third image based on the third basic rendering data and the first pre-processed data, and the third image has the light and shadow effects of the first high-order rendering type; the second electronic device displays the third image.

[0019] In the second aspect, the present application provides an end-cloud collaborative rendering method, including: a first electronic device performs basic rendering on a first end-side scene to obtain first basic rendering data; the first electronic device stores at least one end-side scene, and the first end-side scene is any one of the at least one end-side scene; the first electronic device sends a scene identifier and a first high-order rendering type of the first end-side scene to a server; the first electronic device sends first status data to the server, and the first status data includes data related to preprocessing of the first high-order rendering type in the status data of the first end-side scene; the first electronic device receives the first preprocessing data sent by the server; the first preprocessing data is obtained by performing preprocessing of the first high-order rendering type on the first cloud-side scene after the server updates the first cloud-side scene according to the first status data; the server stores the first cloud-side scene converted from the first end-side scene for the first high-order rendering type; the first electronic device obtains a first image based on the first basic rendering data and the first preprocessing data, and the first image has a rendering effect of the first high-order rendering type; the first electronic device displays the first image.

[0020] In implementing the embodiment of the present application, a terminal-side scene is converted into a cloud-side scene for each high-order rendering type, and the server stores the cloud-side scene; for a specific high-order rendering type, the electronic device only needs to have the ability to perform basic rendering on the terminal-side scene, and the server implements the high-order rendering preprocessing of the cloud-side scene corresponding to the terminal-side scene; after the basic rendering data is integrated with the preprocessing data of the high-order rendering, the final rendered image presents the light and shadow effects of the above-mentioned specific high-order rendering type. In this way, for various high-order rendering types, the computing power requirements and load on the terminal-side GPU can be reduced; electronic devices (such as mobile phones, tablets, etc.) do not need to have high-standard hardware performance to present the rendering effects of high-order rendering. In addition, since the server only performs preprocessing of high-order rendering and does not run the full APP, the consumption of cloud-side computing resources is reduced.

[0021] In one implementation, the method also includes: the first electronic device updates the scene information of the first end-side scene; the first electronic device performs basic rendering on the updated first end-side scene to obtain second basic rendering data; when the network speed between the first electronic device and the server is lower than a preset value, or the network between the first electronic device and the server is disconnected, the first electronic device determines the second image based on the second basic rendering data; and the first electronic device displays the second image.

[0022] In one implementation, the first electronic device sends a scene identifier and a first high-order rendering type of a first end-side scene to a server, including: the first electronic device sends a first request to the server, the first request including the scene identifier and the first high-order rendering type of the first end-side scene; the first request is used to trigger the server to perform preprocessing preparation, the preprocessing preparation including loading the first cloud-side scene and preparing a rendering pipeline for preprocessing of the first high-order rendering type; before the first electronic device sends the first status data to the server, the method also includes: the first electronic device receives confirmation information sent by the server, the confirmation information is used to indicate that the preprocessing preparation has been completed; the first electronic device sends the first status data to the server, including: based on the confirmation information, the first electronic device sends the first status data to the server.

[0023] In one implementation, the state data of the scene on the first terminal side includes part or all of the following: scene identification information, light source information, character information, camera information, and scene update information.

[0024] In one implementation, the server stores a first correspondence between a first terminal-side scene, a first high-level rendering type, and a first cloud-side scene.

[0025] In one implementation, the scene identifier and the first high-order rendering type of the first end-side scene are obtained by the first electronic device from the application package of the first application; the scene identifier of the first cloud-side scene, and the first correspondence between the first end-side scene, the first high-order rendering type and the first cloud-side scene are deployed on the server during the application development of the first application.

[0026] In one implementation, the server stores a second cloud-side scene converted from the first terminal-side scene for a second high-level rendering type.

[0027] In one implementation, the first electronic device receives the first preprocessed data sent by the server, including: the first electronic device receives the compressed and encoded first preprocessed data sent by the server; before the first electronic device obtains the first image based on the first basic rendering data and the first preprocessed data, it also includes: the first electronic device decodes the received first preprocessed data to obtain the decoded first preprocessed data.

[0028] In one implementation, the first high-order rendering type is global illumination; the first preprocessing data includes the irradiance of each pixel in the imaging picture obtained by preprocessing the global illumination, and the first basic rendering data includes the irradiance of each pixel in the imaging picture after basic rendering; the irradiance of the pixel at the first coordinate in the first image is equal to the product of the first preprocessing data and the irradiance of the pixel at the first coordinate in the first basic rendering data, and the pixel value of the pixel at the first coordinate in the first image is determined based on the irradiance of the pixel at the first coordinate.

[0029] In the third aspect, the present application provides a rendering method for end-cloud collaboration, the method comprising: a server receiving a scene identifier and a first high-order rendering type of a first end-side scene sent by a first electronic device; the server receiving first status data sent by the first electronic device, the first status data including data related to pre-processing of the first high-order rendering type in the status data of the first end-side scene; the server updates the first cloud-side scene according to the first status data, performs pre-processing of the first high-order rendering type on the updated first cloud-side scene, and obtains first pre-processed data; the server sends the first pre-processed data to the first electronic device; the first pre-processed data is used to fuse the first basic rendering data to obtain a first image; the first image has the light and shadow effects of the first high-order rendering type; the first basic rendering data is obtained by the first electronic device performing basic rendering on the first end-side scene.

[0030] In implementing the embodiment of the present application, a terminal-side scene is converted into a cloud-side scene for each high-order rendering type, and the server stores the cloud-side scene; for a specific high-order rendering type, the electronic device only needs to have the ability to perform basic rendering on the terminal-side scene, and the server implements the high-order rendering preprocessing of the cloud-side scene corresponding to the terminal-side scene; after the basic rendering data is integrated with the preprocessing data of the high-order rendering, the final rendered image presents the light and shadow effects of the above-mentioned specific high-order rendering type. In this way, for various high-order rendering types, the computing power requirements and load on the terminal-side GPU can be reduced; electronic devices (such as mobile phones, tablets, etc.) do not need to have high-standard hardware performance to present the rendering effects of high-order rendering. In addition, since the server only performs preprocessing of high-order rendering and does not run the full APP, the consumption of cloud-side computing resources is reduced.

[0031] In one implementation, the server receives a scene identifier and a first high-order rendering type of a first end-side scene sent by a first electronic device, including: the server receives a first request sent by the first electronic device, the first request including the scene identifier and the first high-order rendering type of the first end-side scene; before the server receives the first status data sent by the first electronic device, the method also includes: based on the first request, the server performs preprocessing preparation, the preprocessing preparation includes loading the first cloud-side scene, and preparing a preprocessing rendering pipeline for the first high-order rendering type; the server sends confirmation information to the first electronic device, the confirmation information is used to indicate that the preprocessing preparation has been completed; the first status data is sent by the first electronic device based on the confirmation information.

[0032] In one implementation, the state data of the scene on the first terminal side includes part or all of the following: scene identification information, light source information, character information, camera information, and scene update information.

[0033] In one implementation, the server stores a first correspondence between a first end-side scene, a first high-order rendering type, and a first cloud-side scene. Before the server updates the first cloud-side scene according to the first status data, the method also includes: the server determines that the cloud-side scene to be rendered is the first cloud-side scene based on the first correspondence, the scene identifier of the first end-side scene, and the first high-order rendering type.

[0034] In one implementation, the scene identifier and the first high-order rendering type of the first end-side scene are obtained by the first electronic device from the application package of the first application; the scene identifier of the first cloud-side scene, and the first correspondence between the first end-side scene, the first high-order rendering type and the first cloud-side scene are deployed on the server during the application development of the first application.

[0035] In one implementation, the server stores a second cloud-side scene converted from the first terminal-side scene for a second high-level rendering type.

[0036] In one implementation, the server sends the first pre-processed data to the first electronic device, including: the server sends the compressed and encoded first pre-processed data to the first electronic device.

[0037] In one implementation, the first high-order rendering type is global illumination; the first preprocessing data includes the irradiance of each pixel in the imaging picture obtained by preprocessing the global illumination, and the first basic rendering data includes the irradiance of each pixel in the imaging picture after basic rendering; the irradiance of the pixel at the first coordinate in the first image is equal to the product of the first preprocessing data and the irradiance of the pixel at the first coordinate in the first basic rendering data, and the pixel value of the pixel at the first coordinate in the first image is determined based on the irradiance of the pixel at the first coordinate.

[0038] In one implementation, the system also includes a second electronic device, and the first electronic device and the second electronic device are electronic devices of users participating in the same copy of the first end-side scene; the method also includes: if the first preprocessed data is perspective-independent preprocessed data, the server sends the first preprocessed data to the second electronic device.

[0039] In a fourth aspect, an embodiment of the present application provides an electronic device, comprising: a processor and a memory, wherein the memory is coupled to the processor, the memory is used to store computer program code, and the computer program code includes computer instructions. When the processor reads the computer instructions from the memory, the electronic device executes the end-cloud collaborative image enhancement method described in the first aspect.

[0040] In the fifth aspect, an embodiment of the present application provides a server, which includes: a processor and a memory, the memory being coupled to the processor, the memory being used to store computer program code, the computer program code including computer instructions, and when the processor reads the computer instructions from the memory, the server executes the end-cloud collaborative rendering method described in the second aspect.

[0041] In a sixth aspect, an embodiment of the present application provides a computer storage medium comprising computer instructions, which, when executed on an electronic device, enables a communication device to execute a rendering method for end-cloud collaboration in any possible implementation of any of the above aspects.

[0042] In a seventh aspect, an embodiment of the present application provides a computer program product. When the computer program product is run on a computer, it enables the computer to execute the end-cloud collaborative rendering method in any possible implementation of any of the above aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] FIG1 is a system architecture diagram of a communication system provided in an embodiment of the present application;

[0044] FIG2A is a system architecture diagram of a rendering system provided in an embodiment of the present application;

[0045] FIG2B is a system architecture diagram of another rendering system provided in an embodiment of the present application;

[0046] FIG3A shows the rendering effect without GI provided by an embodiment of the present application;

[0047] FIG3B shows a rendering effect with GI provided by an embodiment of the present application;

[0048] FIG3C shows a reflection-free rendering effect provided by an embodiment of the present application;

[0049] FIG3D shows a rendering effect with reflection provided by an embodiment of the present application;

[0050] FIG4 is a data flow diagram provided in an embodiment of the present application;

[0051] FIG5 is a system architecture diagram of another rendering system provided in an embodiment of the present application;

[0052] FIG6 is a flow chart of a rendering method for end-cloud collaboration in the development phase according to an embodiment of the present application;

[0053] FIG7 is a schematic diagram of end-cloud collaborative rendering provided by an embodiment of the present application;

[0054] FIG8 is a schematic diagram of a process flow of a rendering method for end-cloud collaboration provided in an embodiment of the present application during the operation phase;

[0055] FIG9 is a system architecture diagram of another rendering system provided in an embodiment of the present application;

[0056] FIG10 is a schematic diagram of the structure of a mobile device provided in an embodiment of the present application;

[0057] FIG11 is a schematic diagram of the structure of the server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0058] The following is a clear and detailed description of the technical solutions in the embodiments of the present application in conjunction with the accompanying drawings. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in the text is only a description of the association relationship between related objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0059] In the following, the terms "first" and "second" are used for descriptive purposes only and should not be understood to imply or suggest relative importance or implicitly indicate the number of the technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of this application, unless otherwise specified, "plurality" means two or more.

[0060] The term "user interface (UI)" in the following embodiments of this application refers to a medium interface for interaction and information exchange between an application or operating system and a user, which realizes the conversion between the internal form of information and the form acceptable to the user. The user interface is a source code written in a specific computer language such as Java and extensible markup language (XML). The interface source code is parsed and rendered on an electronic device and finally presented as content that the user can recognize. The commonly used form of user interface is graphical user interface (GUI), which refers to a user interface related to computer operations that is displayed in a graphical manner. It can be a visual interface element such as text, icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets, etc. displayed on the display screen of an electronic device.

[0061] 3D rendering technology is the process of projecting the model in a constructed 3D scene into a two-dimensional digital image according to the set viewpoint, light source, material and other information.

[0062] Traditional 3D rendering technologies, such as rasterization, divide object models in a 3D scene into triangles, transform the three-dimensional coordinates of the triangle vertices into two-dimensional coordinates on the image through geometric transformations, and finally fill the triangles on the image with textures, thereby mapping the 3D object model onto a two-dimensional screen to achieve image rendering. This technology usually has difficulty in truly reproducing light and shadow effects such as light reflection, object shadows, and refraction in 3D scenes, so the rendered image cannot present a realistic 3D visual experience.

[0063] Compared to traditional 3D rendering techniques, images rendered with ray tracing technology provide a more realistic 3D visual experience. Ray tracing simulates the propagation of light in a 3D scene through reflection, refraction, shadowing, and scattering, calculating the color and brightness of each pixel. The resulting two-dimensional image's lighting and shadow effects adhere to the laws of real-world physics, resulting in a more realistic 3D virtual scene on electronic devices.

[0064] Ray tracing technology requires the simulation of a huge number of light paths to obtain visual effects that fit the real world, which means that the demand for GPU computing power for this technology explodes with the number of simulated rays. NVIDIA first implemented real-time hardware ray tracing technology for PC-level GPUs on the RTX 20 series graphics cards in 2018; however, due to power consumption and computing power limitations, the end-side GPUs of most mobile devices are currently unable to implement hardware ray tracing technology, and large-scale commercial promotion is still under study. In the end-cloud collaborative rendering method provided in the embodiment of the present application, the mobile device collaborates with the PC-level GPU on the cloud side to render the 3D scene, and then presents the rendering effect that can be achieved by ray tracing technology on the mobile device, reducing the computing power requirements and load on the end-side GPU of the mobile device.

[0065] The following introduces the communication system used in the end-cloud collaborative rendering method provided in the embodiment of the present application.

[0066] FIG1 exemplarily illustrates the system architecture of a communication system 10 provided in an embodiment of the present application. As shown in FIG1 , the communication system 10 includes a terminal device 100 and cloud-side infrastructure, wherein the cloud-side infrastructure includes a server 200. The terminal device 100 can communicate with the cloud-side infrastructure via a communication network, and the cloud-side infrastructure can provide 3D application (APP) rendering services for the terminal device 100.

[0067] The aforementioned 3D applications are applications that provide 3D display effects. 3D images are two-dimensional images rendered based on models in a 3D scene. Examples include 3D gaming apps, 3D modeling apps, 3D navigation apps, and 3D home improvement apps. The following embodiments use gaming apps as an example for illustrative purposes.

[0068] The terminal device 100 may be a mobile phone, a tablet computer, a desktop computer, a laptop computer, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a cellular phone, a personal digital assistant (PDA), an augmented reality (AR) device, a virtual reality (VR) device, an artificial intelligence (AI) device, a wearable device (such as a smart bracelet), an in-vehicle device, a smart home device (such as a smart TV, a smart screen, a large-screen device, etc.), and / or a smart city device. The embodiments of the present application do not impose any special restrictions on the specific type of the terminal device 100, and the terminal device 100 may also be referred to as a terminal side or an electronic device.

[0069] The terminal device 100 can be a mobile device or a non-mobile device. Mobile devices are generally relatively small, and thus their GPU computing power is more limited. The beneficial effects of the end-cloud collaborative rendering solution provided in this application are more significant. The subsequent embodiments will be described using the mobile device 100 as an example.

[0070] The server 200 may be a single server, a server cluster consisting of multiple servers, or a cloud computing center. The server 200 involved in the embodiments of the present application may also be referred to as a cloud server, cloud side, or cloud end. The cloud infrastructure is not limited to the server 200 and may include many other devices, which are not specifically limited here.

[0071] The communication network may include local area networks (LANs) and / or wide area networks (WANs). The communication network may be implemented using any known network communication protocol, which may be various wired or wireless communication protocols, such as Ethernet, universal serial bus (USB), FireWire, global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), Bluetooth, wireless fidelity (Wi-Fi), NFC, voice over Internet protocol (VoIP), a communication protocol supporting a network slicing architecture, or any other suitable communication protocol.

[0072] It should be understood that Figure 1 is merely a schematic diagram of the system structure of the communication system provided in an embodiment of the present application, and does not constitute a specific limitation on the communication system 10. The communication system 10 may include more or fewer devices than shown in the figure. For example, it may also include wireless relay equipment and wireless backhaul equipment (not shown in Figure 1), which are not limited here.

[0073] In a rendering method for end-cloud collaboration (i.e., method one) provided in an embodiment of the present application, all rendering work of the game APP is transferred to the cloud side, and the GPU computing power of the cloud side is used to improve the rendering effect of the mobile device. For example, as shown in Figure 2A, in this method, the mobile device 100 does not need to install the game, but only needs to install the game client of the game APP, which includes an operation instruction processing module and a video decoding module; the server 200 can run the game APP, which includes a rendering module and a logic module, and all rendering work of the game APP is executed in the virtual host / container of the server 200.

[0074] Among them, the operation instruction processing module of the mobile device 100 is responsible for collecting the user's operation instruction information (such as left and right movement, perspective switching, click, etc.), and uploading the operation instructions to the game APP on the cloud side. The game APP on the cloud side sends the received operation instruction information to the logic module. After the logic module updates the game status according to the operation instruction information, it triggers the rendering module to render the game scene in the latest state; the game APP on the cloud side transmits the rendered game screen to the game client of the mobile device 100 in the form of a video stream. The video decoding module of the mobile device 100 decodes the video stream to obtain the game screen, and triggers the mobile device 100 to display the game screen. Optionally, relying on the powerful computing power of the PC-level GPU on the cloud side, the rendering module on the cloud side can use ray tracing technology to render the game screen corresponding to the game scene in the latest state.

[0075] Implementing the above-mentioned end-cloud collaborative rendering method can reduce the computing power requirements and load on the end-side GPU. However, the above-mentioned rendering method still has the following problems: (1) All game rendering tasks are placed on the cloud side for execution, and the cloud side computing cost is high. The end side is only used as a video player, which wastes the GPU resources on the end side. (2) The rendering results on the cloud side cannot be shared among multiple users, further increasing the computing cost on the cloud side. (3) The transmission bandwidth cost is high. When the rendered game screen is transmitted in the form of video streaming, the transmission bandwidth is about 10Mbps in the case of 1080P@60fps, resulting in high export bandwidth of the cloud side server and the downlink bandwidth cost of the user. (4) Poor experience in weak network. For mobile users, they often encounter poor network conditions; when the network is poor, the user's operation response delay is very high, affecting the game operation experience.

[0076] In another end-cloud collaborative rendering method (i.e., method 2) provided in an embodiment of the present application, the end-side performs basic rendering of the game scene, and the cloud-side performs preprocessing of high-level rendering of the game scene; the end-side fuses the basic rendering data and the preprocessing data of the high-level rendering to obtain the game screen of the game scene after high-level rendering. For example, as shown in Figure 2B, in this method, the mobile device 100 can install a game APP, which includes an operation instruction processing module, a rendering module, and a logic module; the server 200 includes a high-level rendering service.

[0077] The operation instruction processing module of the mobile device 100 is responsible for collecting the user's operation instruction information; and sending the operation instruction to the logic module of the mobile device 100. After the logic module updates the game status according to the operation instruction information, it triggers the rendering module of the mobile device 100 to perform basic rendering on the game scene 1 (i.e., the end-side scene 1) in the latest state, and sends the state synchronization data to the server 200. The high-level rendering service of the server 200 updates the cloud-side scene 1 corresponding to the end-side scene 1 according to the state synchronization data; pre-processes the high-level rendering according to the updated cloud-side scene 1, and sends the pre-processed data to the mobile device 100. After the rendering module of the mobile device 100 fuses the basic rendering data and the pre-processed data of the high-level rendering, it can obtain the final rendered game screen, which has a high-level rendering effect and can achieve the rendering effect that can be achieved by ray tracing technology. If there are multiple players in a copy of the same game scene, the server 200 can share the pre-processed data of the high-level rendering in the copy that is independent of the perspective with all players in the copy.

[0078] The implementation of the embodiment of the present application can have the following beneficial effects: only basic rendering and fusion are performed on the end side, and there is no need to perform ray tracing for high-level rendering, which can reduce the computing power requirements and load on the end-side GPU; the mobile devices of game players (such as mobile phones, tablets, etc.) do not need to have high-standard hardware performance, and can also present the rendering effects that can be achieved by ray tracing technology, and play 3D games smoothly. In addition, since the cloud side only performs pre-processing of high-level rendering and does not run the full game APP, the consumption of cloud-side computing resources is reduced; at the same time, the high-level rendering service on the cloud side can share the pre-processed data of high-level rendering for the same copy with the end-side clients corresponding to multiple game users in the same copy, further reducing the consumption of cloud-side computing resources. The data transmitted by the cloud side to the end side can be compressed pre-processed data of high-level rendering, which requires lower transmission bandwidth than transmitting video streams (such as 720p / 1080p@30fps / 60fps, 1080p@30fps video streams); compared with the aforementioned method one, since the required transmission bandwidth is lower, the transmission delay is lower under poor network conditions, reducing the impact of weak network on user operation experience. In addition, even when network conditions are poor, the client can still perform basic rendering to ensure the normal operation of the game app, thereby guaranteeing the user's basic operating experience.

[0079] The following is a detailed introduction to the second method. First, the technical concepts involved in the above-mentioned end-cloud collaborative rendering method are introduced.

[0080] A 3D scene (i.e., a virtual scene) may refer to a virtual three-dimensional scene environment (e.g., a city, park, woods, streets, mountains, etc.) generated by a computer. It may provide a multimedia virtual world, where users may control operable virtual objects in the virtual scene through an operating device or an operating interface, and observe virtual objects such as objects, animals, people, and scenery in the virtual scene from the perspective of the virtual objects. The mobile device 100 may construct a 3D object model based on each virtual object, and then construct a model of the entire scene. For example, a scene in a game APP may include a virtual scene of one or more game levels, or may include one or more specific areas in a game level (e.g., a house, a room, a garden, etc.).

[0081] Basic rendering and high-level rendering are relative concepts. The GPU computing power required for basic rendering is relatively low, including rendering tasks that can be completed independently by relying on the GPU computing power of the mobile device 100. The game screen of the game scene generated by basic rendering can ensure the normal operation of the game APP and the normal interaction of user operations. When the communication network between the mobile device 100 and the server 200 is poor or disconnected, and the pre-processed data of the high-level rendering fed back by the cloud side is not obtained, the game APP of the mobile device 100 can directly display the game screen after basic rendering to ensure the normal operation and viewing of the game APP by the user.

[0082] In the embodiment of the present application, the rendered image generated after the basic rendering is completed can be referred to as the basic rendered image, and the basic rendering data generated by the basic rendering can be the intermediate data in the process of generating the basic rendered image (for example, the irradiance of each pixel obtained by the basic rendering), or it can be the image data of the basic rendered image. The GPU computing power required for the rendering technology used in the basic rendering is what the mobile device 100 can provide, and the embodiment of the present application does not specifically limit the rendering technology. In some embodiments, the basic rendering is performed on the scene to be rendered using rasterization technology.

[0083] The GPU computing power required for high-level rendering is relatively high, including rendering tasks that rely on the GPU computing power on the cloud side to be executed. High-level rendering tasks are usually easy to decouple from basic rendering tasks. The pre-processing data of a specific high-level rendering type is used to obtain the light and shadow effects of the above-mentioned specific high-level rendering type in the picture to be rendered. The pre-processing data corresponding to different high-level rendering types may be different, and accordingly, the fused basic rendering data may also be different. For example, if the above-mentioned specific high-level rendering type is GI rendering, the pre-processing data of GI includes the irradiance of each pixel after GI rendering, and the basic rendering data includes the irradiance of each pixel after basic rendering; if the above-mentioned specific high-level rendering type is reflection rendering, the pre-processing data of reflection rendering includes the cube texture map (cubemap) captured by the reflection probe, and the basic rendering data includes the color of each pixel after basic rendering.

[0084] In some embodiments, high-level rendering is a rendering task related to lighting characteristics, such as global illumination (GI), ambient occlusion (AO), soft shadow, reflection, refraction, and caustics. It can also include the rendering of other characteristics that are relatively intensive on the cloud side and can be integrated with the basic rendering on the terminal side.

[0085] Among them, reflection refers to the phenomenon that when light propagates through different substances, it changes its propagation direction at the interface and returns to the original substance. GI is a lighting characteristic formed by direct lighting and indirect lighting. Global illumination is more consistent with real lighting in reality. Direct lighting is the lighting phenomenon formed by the light emitted by the light source directly shining on the objects in the scene, and indirect lighting is the lighting phenomenon formed in the scene by the light generated by the light emitted by the light source after being reflected by the surface of the objects in the scene. For example, Figures 3A and 3B show the rendering effects of the same game screen displayed by the game APP without GI and with GI; as shown in Figure 3A, when there is no GI, the game screen can also run and display normally, but the picture effect is dim and lacks the light and shadow details of indirect lighting, such as the light and shadow details formed by the reflection of the wall on the object; as shown in Figure 3B, when there is GI, the picture effect will be brighter and more realistic. For example, Figures 3C and 3D show the rendering effects of the same game screen with and without reflection; as shown in Figure 3C, when there is no reflection, the game screen can also run and display normally, but the screen effect lacks the reflected light and shadow effect, for example, the mirror in the screen does not display the mirror projection content; as shown in Figure 3D, when there is reflection, the mirror in the screen displays the mirror projection content.

[0086] In some embodiments, a gaming app's basic rendering tasks include a minimum set of rendering tasks required to ensure the proper functioning of the game screen; a gaming app's high-level rendering tasks include rendering tasks beyond these basic rendering tasks. In some embodiments, these high-level rendering tasks beyond this minimum set may also include rendering tasks that can be implemented solely with the computing power of the client-side GPU. Offloading these rendering tasks to the cloud can reduce the computing power requirements and load on the client-side GPU.

[0087] In some embodiments, the game APP can determine the current high-order rendering type that can be collaboratively rendered by the end-cloud according to the real-time network conditions. In the embodiment of the present application, the server 200 needs to feedback the pre-processed data of the high-order rendering to the mobile device 100. The data type and data volume of the pre-processed data of different high-order rendering types may be different. For example, compared with reflection, the data volume of the pre-processed data of GI is lower. Under normal network conditions (for example, a round-trip delay of 50-100ms and a transmission bandwidth of 10Mbps), under the premise of ensuring the maximum transmission delay of the pre-processed data that meets the normal operation of the user, there are fewer high-order rendering types that meet the end-cloud collaborative rendering; under very good network conditions (such as RTT 10-20ms and bandwidth 100Mbps), under the premise of ensuring the maximum delay of the user's operation, there will be more high-order rendering types that meet the end-cloud collaborative rendering.

[0088] In the embodiment of the present application, developers can determine the basic rendering tasks and high-level rendering tasks of applications (such as game apps) based on actual application requirements and the GPU capabilities of mobile devices. No specific limitations are made here.

[0089] For the above-mentioned end-cloud collaborative rendering method, the present application embodiment specifically provides two implementation solutions.

[0090] In implementation solution one, the client independently downloads the client-cloud collaboration plug-in SDK for the client-side 3D engine. This plug-in implements the aforementioned client-cloud collaborative rendering method. In implementation solution two, the native client-side 3D engine supports the aforementioned client-cloud collaborative rendering method. The following details both implementation solutions.

[0091] For example, FIG4 shows a data flow diagram of a mobile device 100 involved in implementation scheme 1. As shown in FIG4 , the mobile device 100 includes an input / output (IO) layer, a logic layer, a 3D engine layer, an operating system (OS) layer, and a hardware layer. Among them, the IO layer can obtain the user input of the game APP from the operation quality processing module of the OS layer and send it to the logic layer; the logic layer runs the logic code of the game APP (i.e., the above-mentioned logic module), and the logic layer updates the game status according to the user input; the logic layer instructs the 3D engine layer to render the game screen in the latest game state; the 3D engine layer calls the GPU through the OS layer to render the game screen.

[0092] For example, Figure 5 shows a schematic diagram of the system architecture of a device-cloud collaborative rendering system involved in implementing Solution 1. As shown in Figure 5, the rendering system includes a mobile device 100 and a server 200. The game app on mobile device 100 primarily includes the following modules: logic code, a device-side 3D engine, and a device-cloud collaborative plug-in SDK; while the server 200 primarily includes the following modules: a high-level rendering service, a device-cloud collaborative framework, and a cloud-side 3D engine.

[0093] Logic code: includes the logic code that implements the game logic of the game app.

[0094] End-side 3D engine: The end-side 3D engine includes major modules such as the engine framework, scene management, and rendering pipeline. The engine framework is used to provide game logic parsing, model parsing, and animation parsing capabilities for end-side game apps. Scene management is used to manage the scene information of one or more game scenes (i.e., end-side scenes) built by the game app; the rendering pipeline is the process of converting a three-dimensional scene model into an output in screen pixel space, including the following functions: converting the 3D coordinates of an object into 2D coordinates in screen pixel space; and shading each pixel on the screen.

[0095] The end-side 3D engine is divided into two modes: development state and running state. Among them, the end-side 3D engine provides developers with an integrated development environment (IDE) in the development state, allowing developers to develop the logic code of the game APP and the rendering effect of the 3D scene; after the development is completed, the developed related files can be packaged into an application package of the game APP that can be run on mobile devices, such as an Android application package (APK). The end-side 3D engine provides the game APP with a Runtime runtime in the running state; the Runtime runtime generally has cross-platform capabilities and can support the game APP to run on mobile devices with different operating systems such as Hongmeng, Android and iOS. In the embodiment of the present application, when the game APP is running at Runtime, it can call the above-mentioned rendering pipeline to perform basic rendering on the end-side scene to be rendered and generate basic rendering data; it can also call the end-cloud collaborative plug-in SDK to request the server 200 to perform high-order rendering preprocessing; and fuse the basic rendering data and the preprocessed data of the high-order rendering, and obtain a displayable high-order rendering image based on the fused data.

[0096] End-cloud collaborative plug-in SDK: This SDK encapsulates the key capabilities of end-cloud collaborative rendering, and based on the plug-in mechanism provided by the end-side 3D engine, enables third-party end-side 3D engines to have the ability of end-cloud collaborative rendering. The end-cloud collaborative plug-in SDK includes some or all of the following sub-functions: scene conversion, state synchronization, transmission communication, decoding, pipeline adaptation, and authentication. In the embodiment of the present application, end-cloud collaborative rendering is achieved by providing an end-cloud collaborative plug-in SDK to the third-party 3D engine, thereby avoiding intrusive modifications to the third-party 3D engine.

[0097] Scene conversion: This function converts the scene to be rendered by the device-side 3D engine for end-cloud collaborative rendering (i.e., the device-side scene) into the scene required for high-level rendering preprocessing on the cloud side (i.e., the cloud-side scene). The scene information of the cloud-side scene can be consistent with or inconsistent with the scene information of the device-side scene.

[0098] State synchronization on the end side: used to synchronize state data (hereinafter referred to as state synchronization data) that affects the cloud-side preprocessing and preprocessing result distribution strategy to the cloud side. Exemplarily, the state synchronization data includes some or all of the following: scene identification information of the scene to be rendered (such as game identification (identity, ID), level ID, scene ID, copy ID, etc.), light source information (such as the type, position, posture, brightness, quantity, etc. of the light source), character information (such as the position, posture, animation, skills, quantity, etc. of the virtual character), camera information (such as the position, posture, field of view (FOV) of the camera), scene update information (such as information related to object movement, object animation, destruction, etc.). The camera's perspective (i.e., the game player's perspective) determines the final image content after the 3D scene is rendered, and will also affect the contribution of the light source in the scene to the imaging.

[0099] Device-side transmission communication: used to upload device-side status synchronization data to the cloud side and receive high-level rendering pre-processing data sent by the cloud side.

[0100] Decoding on the client side: used to decode the compressed and encoded pre-processed data sent from the cloud side into pre-processed data that can be consumed by the rendering pipeline on the client side.

[0101] End-side pipeline adaptation: Based on the extension mechanism of the end-side 3D engine rendering pipeline, it adds the ability to consume rendering data after decoding and pre-processing on the basis of the original end-side rendering pipeline. In other words, it is able to fuse basic rendering data and pre-processing data to obtain advanced rendering effects.

[0102] Device-side authentication and authorization: Responsible for the authentication and authorization between the device-side cloud collaboration plug-in SDK and the high-level rendering service on the cloud side.

[0103] Cloud-side 3D engine: Contains major functional modules such as scene management, rendering pipeline, and engine framework, providing a capability foundation for the end-cloud collaboration framework and high-level rendering services.

[0104] End-cloud collaboration framework: includes some or all of the following sub-functions: state synchronization, transmission communication, decoding, session management, and authentication.

[0105] State synchronization on the cloud side: used to receive the state synchronization data of the end-side scene (for example, end-side scene 1) sent by the end-side, and update the scene information of the corresponding cloud-side scene (for example, cloud-side scene 1 corresponding to end-side scene 1) in the scene management module of the cloud-side 3D engine.

[0106] Cloud-side encoding: used to compress and encode pre-processed data for high-level rendering services to reduce the amount of transmitted data.

[0107] Cloud-side transmission communication: used to receive status synchronization data transmitted by the client side and send compressed and encoded pre-processed data to the client side.

[0108] Session management: This is used to assign clients connected to the end-side to corresponding high-level rendering services. For example, this is done based on information uploaded by the end-side, such as the game ID, level / scene ID, and instance ID. It is understood that server 200 can establish sessions with multiple end-side clients and the corresponding high-level rendering services for each session, thereby providing high-level rendering services to multiple clients simultaneously.

[0109] Cloud-side authentication: used for authentication between the cloud-side high-level rendering service and the end-to-end cloud collaboration plug-in SDK.

[0110] High-level rendering service: Based on the cloud-side 3D engine and the end-cloud collaborative framework, it provides high-level rendering services, namely, high-level rendering preprocessing, such as GI preprocessing, AO preprocessing, reflection preprocessing, soft shadow preprocessing and other high-level rendering services.

[0111] In some embodiments, to save cloud-side computing power and facilitate development and debugging, during the development phase of a game app, the modules of the mobile device 100 and server 200 involved in the rendering system shown in FIG5 can all be simulated, developed, and tested on the same device. That is, the cloud-side 3D engine and the client-side 3D engine in the development phase can also be set up on the same device, which can be a mobile device or a cloud-side server, without specific limitation. Preferably, the device can be a mobile device.

[0112] Based on the aforementioned embodiments, embodiments of the present application provide a method for end-cloud collaborative rendering. This method is applied to an end-cloud collaborative rendering system, which includes a gaming app on a mobile device 100, the gaming app including an end-side 3D engine and an end-cloud collaborative plug-in SDK. Figure 6 shows a flowchart of a method for end-cloud collaborative rendering during the development phase, which includes some or all of steps S101 through S125.

[0113] S101. The IDE of the end-side 3D engine determines end-side scene 1 and high-level rendering type 1 for end-cloud collaborative rendering.

[0114] In an embodiment of the present application, during the APP development phase, the development state of the end-side 3D engine provides an IDE for developers; developers select the end-side scene (for example, end-side scene 1) to be rendered in end-cloud collaboratively in the IDE and the high-order rendering type for the above-mentioned end-side scene. For game APPs, the end-side scene can be one or more levels in a game, or it can be a partial space in a level (for example, the interior of a building). The high-order rendering type is the rendering type corresponding to the high-order rendering task described in the aforementioned embodiment. Exemplarily, high-order rendering types include GI, AO, soft shadows, reflections, refractions, and caustics. The embodiment of the present application may also include other high-order rendering types, which are not specifically limited here. The subsequent embodiments mainly use GI and reflection as examples for illustrative explanation.

[0115] S102. The end-side 3D engine sends the scene information and high-level rendering type 1 of the end-side scene 1 to the end-cloud collaboration plug-in SDK.

[0116] In some embodiments, the scene information of a client-side scene includes a description of a 3D object model (e.g., a building, a person, or other object) in the scene (e.g., the object model's name, geometry, texture, material, roughness, and transparency), and may also include a description of the scene's background environment, light source information, scene identification information, character information, camera information, and other information. The geometric structure of a 3D object model can be represented in a variety of ways. For example, a triangular mesh can be used, where the surface of the 3D object is approximated by multiple meshes. The mesh information of a mesh can include useful properties and functions such as the mesh's vertex coordinates, normals, texture coordinates, and triangle drawing sequence.

[0117] In an embodiment of the present application, some high-order rendering types can obtain the lighting information of each point in the scene by deploying probes in the scene. In one implementation method, the terminal side can deploy probes for implementing specific high-order rendering types (such as GI) in the 3D scene according to preset deployment rules (for example, uniformly distributed according to a preset density within a preset spatial range of the 3D scene), and each probe can capture, store and update the lighting information received at its location. When it is necessary to shade a shading point (for example, shading point 1) in the 3D scene, the probes adjacent to the shading point 1 can be queried, and the shading point 1 can be shaded according to the lighting information of the location where the adjacent probes are located, and then the pixels corresponding to the shading point 1 in the imaging picture can be rendered.

[0118] In some embodiments, the scene information of the end-side scene 1 may also include information about the probes required for high-level rendering deployed in the scene, such as the probe deployment rules, deployment spatial range, quantity, historical fusion coefficient, deviation value, and other information. The embodiments of the present application do not specifically limit the relevant information of the probes. In the embodiments of the present application, the probes deployed to implement GI can be referred to as GI probes, and the probes deployed to implement reflection can be referred to as reflection probes.

[0119] S103. The end-cloud collaboration plug-in SDK converts the scene information of the end-side scene 1 into the scene information of the cloud-side scene 1 according to the preset conversion rules based on the high-level rendering type 1.

[0120] In the embodiment of the present application, the information content of the scene information of the terminal side scene 1 and the cloud side scene 1 may be consistent or inconsistent.

[0121] In some embodiments, after the cloud collaboration plug-in SDK obtains the scene information of the end-side scene 1, it can simplify the information content of the scene information of the end-side scene 1 according to preset conversion rules, and only retain the scene information required for the preprocessing of high-order rendering type 1. The simplified scene information indicates the cloud-side scene 1 corresponding to the end-side scene 1.

[0122] For example, in one implementation, if the developer selects high-level rendering types such as GI, AO, and soft shadows, the scene information of the converted cloud-side scene 1 only needs to retain the attribute information of the object surface of each 3D object model in the end-side scene 1 (such as Mesh information and color information); even compared with the accuracy of the Mesh information of each model in the scene information of the end-side scene 1, the accuracy of the Mesh information in the scene information of the cloud-side scene 1 can be lower. Reducing the accuracy of the Mesh information is conducive to reducing the complexity of subsequent high-level rendering preprocessing. If the high-level rendering types such as reflection and refraction are selected, the scene information of the converted cloud-side scene 1 and the end-side scene 1 needs to maintain a higher similarity, for example, the information content of the scene information of the cloud-side scene 1 and the end-side scene 1 remains consistent.

[0123] In some embodiments, the data formats of the scene information of the device-side scene 1 and the cloud-side scene 1 may be consistent or inconsistent. The data format of the scene information indicates what types of information the scene information includes, how each type of information is represented, and how the various types of information are arranged in the scene information.

[0124] In some embodiments, the cloud-side 3D engine and the terminal-side 3D engine may use the same 3D engine (i.e., a 3D engine of the same version from the same manufacturer), or different 3D engines. The data formats of the scene information managed by different 3D engines are usually different. When the cloud-side 3D engine and the terminal-side 3D engine are different 3D engines, it is necessary to convert the scene information of the terminal-side scene 1 into the scene information of the cloud-side scene 1 based on the data format of the scene information corresponding to the high-order rendering type 1 managed by the cloud-side 3D engine. The conversion process may also involve simplification of the above-mentioned information content.

[0125] In some embodiments, when the cloud-side 3D engine and the end-side 3D engine are the same 3D engine, the data format of the scene information of the end-side scene 1 and the cloud-side scene 1 is the same, and the scene information of the end-side scene 1 and the cloud-side scene 1 can be kept consistent without conversion.

[0126] In an embodiment of the present application, the developer can manually convert the end-side scene 1 into the cloud-side scene 1 according to the preset conversion rules through the end-cloud collaboration plug-in SDK; or use the developed automatic conversion tool to automatically convert the end-side scene 1 into the cloud-side scene 1 according to the preset conversion rules.

[0127] Step S103 is optional. In some embodiments, S103 does not need to be executed, and the device-side scenario and the cloud-side scenario remain completely consistent. The cloud-side scenario 1 involved in subsequent embodiments is equivalent to the device-side scenario 1.

[0128] It is understood that in this embodiment of the present application, a device-side scene is converted into a cloud-side scene for each high-level rendering type. For example, for high-level rendering type 1 (such as GI), device-side scene 1 can be converted into cloud-side scene 1; for high-level rendering type 2 (such as reflection), device-side scene 1 can be converted into cloud-side scene 2.

[0129] S104. The end-cloud collaboration plug-in SDK sends the scene information of the cloud-side scene 1 and the high-level rendering type 1 to the IDE of the cloud-side 3D engine.

[0130] S105. The IDE of the cloud-side 3D engine adjusts the scene information of the cloud-side scene 1.

[0131] In some embodiments, during the conversion of the terminal-side scene 1 to the cloud-side scene 1, conversion errors may occur, resulting in the scene conversion not strictly complying with the above-mentioned preset conversion rules. For example, due to human errors when the developer manually performs the scene conversion, or a runtime bug occurs when the automatic conversion tool automatically performs the scene conversion, resulting in a conversion error. After importing the scene information of the cloud-side scene 1 into the IDE of the cloud-side 3D engine, the developer can check and adjust the scene information of the cloud-side scene 1 through the IDE of the cloud-side 3D engine. Exemplarily, the above-mentioned adjustments include: repairing the material, geometry, size or position of the 3D object model with conversion problems, repairing the spatial range and number of GI probes, etc.

[0132] Step S105 is optional. In some embodiments, step S105 does not need to be performed.

[0133] S106 : The IDE of the end-side 3D engine performs basic rendering on the end-side scene 1 according to the state data 1 of the end-side scene 1 to obtain basic rendering data 1 .

[0134] The 3D scene's state data includes data that can be adjusted during the 3D scene's runtime. This state data indicates the real-time state of the 3D scene. During the development phase, the client-side 3D engine's IDE can simulate and generate various state data for client-side scene 1 (for example, state data 1) to debug the rendering effects of client-side scene 1.

[0135] In some embodiments, the IDE of the end-side 3D engine updates the scene information of the end-side scene 1 according to the status data 1; the IDE of the end-side 3D engine performs basic rendering on the updated end-side scene 1 using rasterization technology to obtain basic rendering data 1. In an embodiment of the present application, when the network is poor, the basic rendered game screen can be obtained according to the basic rendering data 1, and the game screen can ensure the normal operation of the game APP and the normal interaction of user operations. Exemplarily, referring to Figure 7, in one implementation, the end-side rendering pipeline includes a basic rendering channel and a conversion channel; the basic rendering channel uses rasterization technology to render the end-side scene 1, and the output basic rendering data 1 includes the irradiance of each pixel in the imaging picture of the end-side scene 1; the conversion channel determines the pixel value (for example, RGB color information) of each pixel in the imaging picture according to the above-mentioned basic rendering data 1, and outputs a basic rendering image as the rendering picture to be displayed.

[0136] S107. The IDE of the end-side 3D engine sends the state synchronization data 1 of the end-side scene 1 to the end-cloud collaboration plug-in SDK.

[0137] S108 . The end-cloud collaboration plug-in SDK sends the state synchronization data 1 of the end-side scenario 1 to the end-cloud collaboration framework of the server 200 .

[0138] S109 . The end-cloud collaboration framework of the server 200 sends the state synchronization data 1 of the end-side scene 1 to the IDE of the cloud-side 3D engine.

[0139] In some embodiments, state synchronization data 1 includes state data required to implement high-level rendering type 1 of cloud-side scene 1 in state data 1. State synchronization data 1 may include some or all of the data in state data 1. The end-cloud collaboration plug-in SDK synchronizes state synchronization data 1 to the IDE of the cloud-side 3D engine. The cloud-side 3D engine IDE can update the cloud-side scene 1 to the latest state based on state synchronization data 1.

[0140] S110. The IDE of the cloud-side 3D engine updates the cloud-side scene 1 according to the status synchronization data 1, pre-processes the updated cloud-side scene 1 with the high-order rendering type 1, and obtains the pre-processed data 1 of the high-order rendering type 1.

[0141] High-level rendering type 1 can be the rendering of any one of the lighting characteristics such as GI, AO, soft shadows, reflection, refraction and caustics. The embodiment of the present application does not specifically limit the rendering algorithms of lighting characteristics such as GI, AO, soft shadows, reflection, refraction and caustics. For example, the rendering algorithms for implementing GI include but are not limited to: ray tracing, path tracing, dynamic diffuse global illumination (DDGI), spherical harmonic lighting, voxel-based global illumination, and point-based global illumination, etc.

[0142] In an embodiment of the present application, the IDE of the cloud-side 3D engine can synchronize data 1 according to the status of the end-side scene 1, update the scene information of the cloud-side scene 1 corresponding to the end-side scene 1, and use the rendering algorithm 1 to perform preprocessing of the high-order rendering type 1 (such as GI) on the updated cloud-side scene 1 to obtain preprocessing data; the preprocessing data can indicate the light and shadow effects of the high-order rendering type 1 in the cloud-side scene 1.

[0143] For example, taking the high-level rendering type 1 as GI and the rendering algorithm 1 as DDGI as an example, the preprocessing of the high-level rendering type 1 is explained exemplarily. The DDGI algorithm uses GI probes to store the lighting information of the scene and dynamically updates it using ray tracing, so as to achieve real-time dynamic diffuse global illumination effects. The DDGI algorithm packages a group of probes into a DDGI Volume (that is, a cubic area in 3D space). You only need to drag the Volume into the scene to be rendered, and the Volume will automatically place probes in it; the shading points in the Volume will automatically capture lighting information through the surrounding probes.

[0144] In some embodiments, the scene information of the cloud-side scene 1 includes relevant information of the GI probe (such as deployment rules, deployment location and data, etc.); referring to FIG7 , based on the scene information of the cloud-side scene 1, the IDE of the cloud-side 3D engine uses the DDGI rendering channel of the cloud-side rendering pipeline to deploy and run the GI probe in the cloud-side scene 1. The GI probe obtains and stores the lighting information of the location and generates Probe layer data; the Probe layer data includes the lighting information of each GI probe, and the lighting information of the GI probe includes irradiance. The shading point corresponding to each pixel in the imaging picture in the cloud-side scene 1 is determined according to the camera viewing angle. For any shading point in the volume of the cloud-side scene 1 (for example, shading point 1), the DDGI rendering pipeline obtains the lighting information of the 8 GI probes around the shading point 1; based on the lighting information of the above 8 GI probes, the irradiance is interpolated to obtain the irradiance of the shading point 1. The pre-processed data of GI includes the irradiance of the shading point corresponding to each pixel in the imaging picture.

[0145] The probe deployed by DDGI stores spherical information. The DDGI algorithm encodes spherical data into a two-dimensional texture map through octahedral mapping. The smallest unit of the texture map is the texel, and one texel corresponds to one or more pixels. The lighting information of the GI probe can include the irradiance received from the hemisphere in the direction of the texel (w), the distance r(w) from the nearest object to the probe as seen from the texel direction, and the square of the distance r 2 (w). The irradiance is encoded as a three-dimensional vector texture, r(w) and r 2 (w) are encoded together as a two-dimensional vector texture (the x component stores r(w), the y component stores r 2 (w)).

[0146] In one implementation, probe 1 is any one of the above-mentioned 8 GI probes. The above-mentioned interpolation processing of irradiance is performed based on the lighting information of the above-mentioned 8 GI probes to obtain the irradiance of shading point 1, including: obtaining three weight coefficients of probe 1, namely the trilinear difference coefficient, the directional coefficient and the Chebyshev coefficient, and taking the normalized value of the product of the above three coefficients as the weight of probe 1; based on the weight of each probe, the irradiance of the above-mentioned 8 GI probes is weighted to obtain the irradiance of shading point 1. Among them, the trilinear difference coefficient indicates the distance between probe 1 and shading point 1. If the coefficient is large, the weight of probe 1 is reduced; the directional coefficient indicates the angle between the direction of shading point 1 pointing to probe 1 and the surface normal of shading point 1. If the coefficient is too large, the weight of probe 1 is reduced; the Chebyshev coefficient indicates the probability that there is an occluder between probe 1 and shading point 1. If the coefficient is large, the weight of probe 1 is reduced; the Chebyshev coefficient indicates the probability that there is an occluder between probe 1 and shading point 1. If the coefficient is large, the weight of probe 1 is reduced; the Chebyshev coefficient is based on the above-mentioned distance r(w) and the distance square r 2 (w) Determined.

[0147] In some embodiments, in addition to the illumination information of probe 1, the Probe layer data also includes classification information (classification) and relocation information (relocation) of probe 1; the classification information is used to indicate whether the state of probe 1 is a valid probe, and then determine whether to use the illumination information of probe 1 based on the state of probe 1; the relocation information (relocation) is used to indicate whether and how to adjust the position of probe 1.

[0148] Exemplarily, taking the high-order rendering type 1 as reflection and the rendering algorithm 1 as a reflection capture algorithm based on a reflection probe as an example, the preprocessing of the high-order rendering type 1 is exemplarily described. The IDE of the cloud-side 3D engine uses the cloud-side rendering pipeline to deploy and run the reflection probe in the cloud-side scene 1 to obtain the lighting information of the reflection probe. The lighting information of the reflection probe includes the cube texture (cubemap) of the location of the reflection probe. The cubemap is a synthesis of the reflection texture maps obtained by the reflection probe along the six directions of the front, back, left, right, up and down of the camera. The cubemap can be used to implement environment mapping, and the environment mapping can simulate the environment around the probe. The preprocessing data of the reflection includes the cubemap of all reflection probes in the cloud-side scene 1, or one or more cubemaps belonging to the position of the camera uploaded by the end side, or the cubemap after the one or more cubemaps are fused, or the cubemap of the reflection probe closest to the camera.

[0149] S111. The IDE of the cloud-side 3D engine sends pre-processed data 1 of high-level rendering type 1 to the end-cloud collaboration plug-in SDK.

[0150] In some embodiments, in step S110, the IDE of the cloud-side 3D engine performs preprocessing of the high-level rendering type 1, and the obtained preprocessed data includes preprocessed data of all viewing angles of the camera. The IDE of the cloud-side 3D engine crops the preprocessed data according to the viewing angle of the camera in the state synchronization data 1, obtains the preprocessed data 1 within the viewing angle of the camera, that is, the preprocessed data related to the end-side user of the mobile device 100, and sends the preprocessed data 1 to the end-cloud collaboration plug-in SDK.

[0151] S112. The end-cloud collaboration plug-in SDK sends pre-processed data 1 of high-level rendering type 1 to the IDE of the end-side 3D engine.

[0152] In some embodiments, the cloud-side 3D engine's IDE compresses and encodes preprocessed data 1 using a preset encoding algorithm and sends the encoded preprocessed data to the end-cloud collaboration plug-in. The end-cloud collaboration plug-in SDK decodes the received preprocessed data using a decoding algorithm corresponding to the preset encoding algorithm, obtains decoded preprocessed data 1, and sends it to the end-side 3D engine's IDE. This embodiment of the present application does not specifically limit the preset encoding algorithm.

[0153] S113 : The IDE of the terminal-side 3D engine fuses the basic rendering data 1 and the pre-processed data 1 of the high-order rendering type 1 to obtain an image 1 having a rendering effect of the high-order rendering type 1.

[0154] It can be understood that since the scene information of the end-side scene 1 includes the scene information of the cloud-side scene 1, and for the same state data (such as camera information), the end-side scene 1 and the cloud-side scene 1 remain synchronized; therefore, the pixels in the imaging picture indicated by the basic rendering data 1 generated by the basic rendering and the pixels in the imaging picture indicated by the pre-processing data 1 of the high-level rendering can correspond one to one.

[0155] In some embodiments, high-level rendering type 1 is GI, the rendering algorithm is DDGI, basic rendering data 1 includes the irradiance of each pixel in the imaged image after raster rendering, and pre-processed data 1 includes the irradiance of each pixel in the imaged image after DDGI rendering. Referring to FIG7 , the module of the end-cloud collaborative plug-in SDK adds a fusion channel to the end-side rendering pipeline based on the extension mechanism of the end-side 3D engine rendering pipeline to consume pre-processed data. The basic rendering channel outputs basic rendering data 1 to the fusion channel. The end-side 3D engine IDE inputs pre-processed data 1 received from the cloud-side DDGI rendering channel into the end-side DDGI rendering channel, which then outputs pre-processed data 1 to the fusion channel. The fusion channel outputs fused rendering data, which includes fused irradiance. The fused irradiance of the pixel at coordinate 1 is equal to the product of the irradiance of the pixel at coordinate 1 in the basic rendering data 1 and the irradiance of the pixel at coordinate 1 in the pre-processed data 1. The fused rendering data is input into the conversion channel, and the conversion channel can determine the pixel value of each pixel according to the fused rendering data, thereby outputting a rendering image (ie, image 1).

[0156] As shown in Figure 7, when the network is poor and the client-side DDGI rendering pipeline has not received pre-processed data 1 from the cloud, the basic rendering data 1 is input into the fusion pipeline, which then outputs the basic rendering data 1 to the conversion pipeline. The conversion pipeline uses this basic rendering data 1 to determine the pixel values ​​of each pixel and output a basic rendered image as the rendered image to be displayed. This basic rendered image lacks the lighting and shadow effects of high-level rendering type 1. However, the game app can display this basic rendered image to ensure normal operation.

[0157] In some embodiments, high-level rendering type 1 is reflection, and the rendering algorithm is based on a reflection probe reflection capture algorithm. The base rendering data (and base rendered image) includes pixel information (e.g., color) for each pixel in the imaged scene after raster rendering. Preprocessed reflection data 1 includes a cubemap of reflection probes in cloud-side scene 1; pixel information (e.g., color) for each pixel in the imaged scene after reflection rendering is determined based on the cubemap of the reflection probes; and pixel information for pixels at the same coordinate in the base rendered imaged scene and the reflection rendered imaged scene is fused to provide the pixel information for the pixel at the same coordinate in image 1.

[0158] In some embodiments, pixel information (e.g., color) of each pixel in the image after reflection rendering is determined based on the cubemap of the reflection probe, including: reflection pre-processed data 1 including the cubemaps of all reflection probes in the cloud-side scene 1; determining one or more cubemaps to which the camera position belongs in the cubemaps of all the reflection probes based on the camera position; determining cubemap1 after the one or more cubemaps are fused; determining the intersection point P of the reflection vector R after the light emitted from the camera is reflected by the ground and the bounding sphere containing cubemap1; using the vector CP from the center point C of cubemap1 to the intersection point P as a new reflection vector R'; and using the reflection vector R' to sample the cubemap1, that is, determining the pixel information (e.g., color) of the corresponding pixel in the image based on the texture of the intersection point of the reflection vector R' and cubemap1.

[0159] The pre-processed reflection data 1 may also include one or more cubemaps corresponding to the camera's location uploaded by the client, or a fused cubemap of one or more cubemaps, or a cubemap of the reflection probe closest to the camera. The specific method for calculating the pixel information for each pixel in the image after reflection rendering can be found in the previous embodiment and will not be further described here.

[0160] S114. The developer adjusts pre-processing-related parameter 1 in the scene information of the cloud-side scene 1 through the IDE of the cloud-side 3D engine according to the high-level rendering effect presented by the image 1.

[0161] In this embodiment of the application, if you are not satisfied with the high-level rendering effect presented in Image 1, you can adjust the cloud-side scene pre-processing related parameters (such as Parameter 1). For example, if the developer observes that the rendering effect of virtual object 1 in Image 1 does not meet expectations, you can adjust the material of the virtual object; if the developer observes that the lighting effect of GI in Image 1 is not realistic enough, you can adjust the position information of the GI probe, the parameter information of the GI calculation, etc.

[0162] S115. The IDE of the cloud-side 3D engine performs preprocessing of the high-order rendering type 1 according to the cloud-side scene 1 after adjusting the parameter 1, and obtains the updated preprocessing data 2.

[0163] S116. The IDE of the cloud-side 3D engine sends the adjusted parameter 1 and the updated pre-processed data 2 to the end-cloud collaboration plug-in.

[0164] S117. The end-cloud collaboration plug-in sends the adjusted parameter 1 and the updated pre-processed data 2 to the IDE of the end-side 3D engine.

[0165] The adjusted parameters 1 and the updated pre-processed data 2 may be sent simultaneously or separately.

[0166] S118. The IDE of the end-side 3D engine updates parameter 1 in the scene information of the end-side scene 1, so that parameter 1 in the scene information of the end-side scene 1 and the cloud-side scene 1 remain synchronized; basic rendering is performed according to the end-side scene 1 after the updated parameter 1, and basic rendering data 2 is obtained.

[0167] S119. The IDE of the terminal-side 3D engine fuses the basic rendering data 2 and the pre-processed data 2 to obtain an image 2 having a rendering effect of the high-order rendering type 1.

[0168] S120 . The IDE of the end-side 3D engine adjusts parameter 2 of the end-side scene 1 and / or the processing logic of consuming pre-processed data according to the rendering effect of image 2 .

[0169] In an embodiment of the present application, when the high-level rendering effect presented by Image 2 is not satisfactory, the relevant parameters of the end-side scene (such as Parameter 2) can be adjusted on the end-side; when the rendering delay of Image 2 is large, the processing logic of the consumption pre-processed data can also be adjusted, such as adjusting the encoding / decoding algorithm of the pre-processed data to increase the encoding and decoding rate of the pre-processed data and reduce the rendering delay of the end-cloud collaborative rendering.

[0170] S121. The IDE of the end-side 3D engine performs basic rendering according to the end-side scene 1 after adjusting the parameter 2, and obtains basic rendering data 3.

[0171] S122 : The rendering pipeline of the IDE of the terminal-side 3D engine fuses the basic rendering data 3 with the pre-processed data 2 of the high-level rendering type 1 to obtain an image 3 having a rendering effect of the high-level rendering type 1.

[0172] In some embodiments, if the scene information of the cloud-side scene 1 does not include parameter 2, step S121 is followed by step S122, where the pre-processed data 2 is fused with the basic rendering data 3 to obtain the rendered image to be displayed. If the scene information of the cloud-side scene 1 includes parameter 2, after adjusting parameter 2 of the client-side scene 1, parameter 2 of the cloud-side scene 1 must also be adjusted synchronously. The pre-processed data for the high-level rendering is updated to pre-processed data 3 based on the adjusted cloud-side scene 1. The basic rendering data 3 is fused with the pre-processed data 3 to obtain the rendered image to be displayed. In one implementation, step S121 is followed by steps A1 to A7.

[0173] A1. The IDE of the device-side 3D engine sends the adjusted parameter 2 to the device-cloud collaboration plug-in SDK, so that parameter 2 in the scene information of the device-side scene 1 and the cloud-side scene 1 remain synchronized.

[0174] A2. The end-cloud collaboration plug-in SDK sends the adjusted parameter 2 to the end-cloud collaboration framework of the server 200.

[0175] A3. The end-cloud collaboration framework of server 200 sends the adjusted parameter 2 to the IDE of the cloud-side 3D engine.

[0176] A4. The IDE of the cloud-side 3D engine updates parameter 2 in the scene information of the terminal-side scene 1, so that parameter 1 in the scene information of the terminal-side scene 1 and the cloud-side scene 1 remain synchronized; preprocessing of the updated cloud-side scene 1 using high-level rendering type 1 is performed to obtain preprocessing data 3 of the high-level rendering type 1.

[0177] A5. The IDE of the cloud-side 3D engine sends pre-processed data 3 of high-level rendering type 1 to the end-cloud collaboration plug-in.

[0178] A6. The end-cloud collaboration plug-in sends pre-processed data 3 of high-level rendering type 1 to the IDE of the end-side 3D engine.

[0179] A7. The rendering pipeline of the IDE of the terminal-side 3D engine fuses the basic rendering data 3 with the pre-processed data 3 of the high-level rendering type 1 to obtain an image 4 having the rendering effect of the high-level rendering type 1.

[0180] Some or all of steps S114 to S122 are optional. In some embodiments, developers trigger parameter adjustments only on the cloud side, executing steps S114 to S119, without executing steps S120 to S122. In some embodiments, developers trigger parameter adjustments only on the client side, executing steps S120 to S122, without executing steps S114 to S119.

[0181] The present embodiment does not specifically limit the execution order of steps S114 to S119 and S120 to S122. In some embodiments, the developer may first trigger parameter adjustment on the client side (i.e., steps S114 to S119) and then trigger parameter adjustment on the cloud side (S120 to S122).

[0182] In some embodiments, after step S122, if the rendering effect of the adjusted high-level rendering image still does not meet the developer's rendering requirements, steps S114 to S119 and steps S120 to S122 are repeated, that is, the cloud side and the terminal side are iteratively adjusted until the rendering effect meets the developer's rendering requirements.

[0183] In some embodiments, in step S101, high-level rendering type 1 (e.g., GI) and end-side scene 1 are selected, and steps S106 to S122 are executed for different state data of end-side scene 1; end-side scene 1 is selected, and steps S101 to S122 are executed for different high-level rendering types; and steps S101 to S122 are executed for different end-side scenes. In this way, the end-cloud collaborative rendering solution involved in the end-cloud collaborative rendering solution is tested and adjusted, so that for any high-level rendering type, any end-side scene, and any state of the end-side scene, the end-cloud collaborative rendering solution provided in the embodiment of the present application can achieve better high-level rendering effects.

[0184] S123 and the end-cloud collaboration plug-in SDK package the adjusted end-side scenarios and related files into a game app.

[0185] S124. The IDE of the cloud-side 3D engine packages the adjusted cloud-side scene and related files and uploads them to the cloud-side high-level rendering service.

[0186] Referring to the above steps S101 to S122, after the developer has debugged the rendering effects of all game scenes (i.e., terminal-side scenes) of the game APP, the developer packages the project files of the adjusted terminal-side scenes into the terminal-side APP, such as packaging the scene information of the adjusted terminal-side scene 1, the scene ID of the terminal-side scene 1, and the high-order rendering type 1 to the application package of the game APP. The developer packages the adjusted cloud-side scene and uploads it to the cloud-side high-order rendering service, such as packaging the scene information of the cloud-side scene 1, the scene ID of the cloud-side scene 1, the correspondence between the scene ID of the terminal-side scene 1 and the high-order rendering type 1, etc. The scene management module in the cloud-side high-order rendering service is responsible for managing the uploaded cloud-side scenes, and records the correspondence between the scene IDs of various terminal-side scenes, various high-order rendering types, and the scene IDs of the cloud-side scenes.

[0187] S125 . The developer associates the cloud-side high-level rendering service with the authentication information of the game APP of the mobile device 100 . The authentication information is used to verify the legitimacy of the mobile device 100 .

[0188] In the embodiments of the present application, there is no specific limitation on the authentication method and authentication information between the cloud-side high-level rendering service and the terminal-side APP.

[0189] In some embodiments, a developer associates the cloud-side high-level rendering service with the authentication information of the game app on the mobile device 100. This includes: the developer registers the game app's high-level rendering service on the portal website and obtains a key pair, namely public key 1 and private key 1; the game app on the device stores public key 1, and the cloud-side end-cloud collaboration framework stores private key 1. The authentication information on the device side is public key 1, which is used to encrypt the session and verify the digital signature; the authentication information on the cloud side is private key 1 corresponding to public key 1, which is used to decrypt data encrypted by public key 1.

[0190] The game APP of the mobile device also includes a logic module. FIG8 shows a flow chart of a method of end-cloud collaborative rendering method in the running stage. The above method flow includes part or all of steps S201 to S215.

[0191] S201. The logic module of the mobile device 100 sends a rendering request to the runtime of the end-side 3D engine. The rendering request includes the scene ID of the end-side scene 1 to be rendered and the high-order rendering type 1. The rendering request is used to request rendering based on the latest state of the end-side scene 1 to obtain an image with the rendering effect of the high-order rendering type 1.

[0192] In some embodiments, the game screen of the game scene of the game APP is refreshed according to a preset refresh rate; when the logic module detects that the game screen of the current game scene is updated, it sends a rendering request to the runtime of the end-side 3D engine to render the updated game screen; the updated game screen is displayed in a new refresh cycle. For example, when it is detected that the user switches the perspective of the game, the game APP will update the game screen according to the switched perspective. The mobile device 100 can obtain the scene information of the end-side scene 1 corresponding to the current game screen, the scene ID of the end-side scene 1, and the high-level rendering type 1 from the application package of the game APP.

[0193] S202 : The end-side 3D engine Rumtime performs basic rendering based on the current scene information of the end-side scene 1 to obtain basic rendering data 4 .

[0194] For details, please refer to the related description of the aforementioned step S106, which will not be repeated here.

[0195] S203. The runtime of the end-side 3D engine sends a preprocessing request for high-order rendering type 1 to the end-cloud collaborative plug-in SDK. The preprocessing request includes the scene ID of the end-side scene 1 and high-order rendering type 1. The preprocessing request is used to instruct to perform preprocessing of high-order rendering type 1 on the cloud-side scene corresponding to the end-side scene 1 and the high-order rendering type 1.

[0196] S204. The end-cloud collaboration plug-in SDK sends a pre-processing request to the end-cloud collaboration framework.

[0197] S205. The end-cloud collaboration framework performs authentication and authorization based on the received pre-processing request; after authentication and authorization are passed, the pre-processing request is sent to the high-level rendering service.

[0198] In some embodiments, the runtime of the end-side 3D engine sends a pre-processing request encrypted with public key 1 to the end-cloud collaboration plug-in SDK. The end-cloud collaboration framework decrypts the received pre-processing request using private key 1. If the decryption is successful, authentication is successful, and the decrypted pre-processing request is sent to the high-level rendering service. This embodiment of the application does not specifically limit the authentication method between the high-level rendering service of the server 200 and the game app of the mobile device 100.

[0199] S206. The high-level rendering service performs preprocessing preparation, including loading cloud-side scene 1 and preparing a rendering pipeline of high-level rendering type 1. Cloud-side scene 1 is determined based on the scene ID and high-level rendering type 1 of the terminal-side scene 1 in the preprocessing request.

[0200] In some embodiments, the high-level rendering service in step S124 is packaged with the scene ID of the terminal-side scene, the correspondence between the high-level rendering type and the scene ID of the cloud-side scene, for example, the correspondence between the scene ID of the terminal-side scene 1, the high-level rendering type 1 and the scene ID of the cloud-side scene 1, as well as the scene information of the cloud-side scene; based on the above correspondence, it can be determined that the cloud-side scene 1 is the cloud-side scene after the terminal-side scene 1 is converted to the high-level rendering type 1, that is, the scene to be rendered.

[0201] In some embodiments, the pre-processing request may further include the game ID of the game to which the client-side scene 1 belongs. The high-level rendering service may record the scene IDs of the client-side scenes of each game by game ID, so that the high-level rendering service can quickly locate the client-side scene 1 and the cloud-side scene 1 corresponding to the client-side scene 1.

[0202] S207. The high-level rendering service sends a confirmation message to the end-cloud collaboration plug-in SDK, where the confirmation message is used to indicate that the pre-processing preparation has been completed.

[0203] S208. The end-cloud collaboration plug-in SDK sends state synchronization data 2 related to the pre-processing of the high-level rendering type 1 to the high-level rendering service.

[0204] Specifically, the state synchronization data 2 may refer to the related description of the state synchronization data 1, which will not be repeated here.

[0205] S209 . The high-level rendering service updates the cloud-side scene 1 according to the received state synchronization data 2 , and performs high-level rendering type 1 preprocessing based on the scene information of the updated cloud-side scene 1 to obtain preprocessing data 4 .

[0206] For details, please refer to the description of step S110 above, which will not be repeated here. In the embodiment of the present application, the high-level rendering service only performs pre-processing of high-level rendering and does not run the full game APP; compared with the above method 1, the consumption of cloud-side computing resources is reduced.

[0207] S210. The high-level rendering service sends pre-processed data 4 related to the end-side user to the end-cloud collaboration framework.

[0208] In some embodiments, in step S209, the high-level rendering service performs preprocessing of the high-level rendering type 1, and can obtain preprocessed data within all viewing angles of the camera. The high-level rendering service crops the preprocessed data based on the current viewing angle of the camera in the status synchronization data 2 uploaded by the mobile device 100, obtains preprocessed data 4 within the current viewing angle of the camera, that is, preprocessed data related to the end-side user of the mobile device 100, and sends the preprocessed data 4 to the end-cloud collaboration framework in step S210. In some embodiments, the preprocessed data 4 generated in step S209 is the preprocessed data within the current viewing angle of the camera and does not need to be cropped.

[0209] S211. The end-cloud collaborative framework compresses and encodes the pre-processed data 4 to obtain the encoded data 1.

[0210] S212. The end-cloud collaboration framework sends the encoded data 1 to the end-cloud collaboration plug-in SDK.

[0211] S213. The end-cloud collaborative plug-in SDK decodes the encoded data 1 to obtain pre-processed data 4.

[0212] In some embodiments, the IDE of the cloud-side 3D engine uses a preset encoding algorithm to compress and encode the pre-processed data 4 to obtain the encoded data 1; then, the encoded data 1 is encapsulated using a preset communication protocol and sent to the end-cloud collaboration plug-in SDK. The end-cloud collaboration plug-in decapsulates the received data to obtain the encoded data 1; the end-cloud collaboration plug-in decodes the received encoded data 1 using a preset encoding algorithm to obtain the decoded pre-processed data 4. The embodiments of the present application do not specifically limit the above-mentioned preset encoding algorithm and preset communication protocol. For example, the preset encoding algorithm is the HEVC algorithm, and the preset communication protocol is the UDP protocol.

[0213] In the embodiment of the present application, the data transmitted from the cloud side to the terminal side is pre-processed data after compression encoding, and the transmission bandwidth required is lower than that of the video stream data transmitted by the aforementioned method 1.

[0214] S214. The end-cloud collaborative plug-in SDK sends pre-processed data 4 to the runtime of the end-side 3D engine.

[0215] S215 , the runtime of the terminal-side 3D engine fuses the basic rendering data 4 and the pre-processed data 4 to obtain an image 4 having a rendering effect of the high-order rendering type 1.

[0216] For details, please refer to the relevant description of the aforementioned step S113, which will not be repeated here.

[0217] In some embodiments, in step S201, the above-mentioned rendering request and preprocessing request may also include multiple high-level rendering types. The server 200 may perform preprocessing on the cloud-side scene 1 according to each of the multiple high-level rendering types in the preprocessing request, and feedback the preprocessing data of each high-level rendering type to the mobile device 100; the mobile device 100 may fuse the basic rendering data and the preprocessing data of the above-mentioned multiple high-level rendering types to generate an image with the light and shadow effects of the above-mentioned multiple high-level rendering types.

[0218] In the embodiment of this application, the end-side game app only needs to have the ability to perform basic rendering of the 3D scene. The game app implements high-level rendering of the 3D game scene by calling the end-cloud collaborative plug-in SDK. After the basic rendering data is integrated with the pre-processed data of the high-level rendering, the final rendered image presents the light and shadow effects of the high-level rendering, such as the light and shadow effects achieved by ray tracing. In this way, 3D applications (such as game apps) running on mobile devices with weak GPU capabilities are equipped with high-level rendering capabilities (such as ray tracing capabilities), effectively improving the user experience.

[0219] Exemplarily, the end-side scene 1 to be rendered is a multi-player game scene of a game APP, and multiple game players participate in the same copy of the game scene at the same time, such as user 1 of mobile device 100, and user 2 of mobile device 300. If the lighting and shadow effects of the high-order rendering type 1 (such as GI) of the current scene are independent of the user's perspective, the pre-processed data of the high-order rendering type 1 can be shared with other players of the game so that other players can achieve high-order rendering effects on the end side. It can be understood that for the latest state of the end-side scene 1, the server 200 does not need to perform pre-processing for each user for the high-order rendering type that is independent of the perspective, but only needs to perform pre-processing once and share it with other users. In this way, the consumption of cloud-side computing resources is further reduced in the end-cloud collaborative rendering solution.

[0220] In some embodiments, the pre-processing request may also include the copy ID of the game copy currently running in the client scene 1. Based on the game ID, scene ID, and copy ID, other players in the same copy as the client user of the mobile device 100 may be queried.

[0221] In some embodiments, the preprocessing of the high-level rendering type 1 of the terminal-side scene 1 in the latest state can be perspective-independent processing or perspective-dependent processing. Perspective-independent processing refers to processing that is independent of the current perspective of the terminal-side user (i.e., the perspective of the aforementioned camera); for example, in the latest state of the game scene, even if the perspectives of user 1 and user 2 in the game scene are different, the light and shadow effects of the high-level rendering type 1 in the game screen seen by user 1 and user 2 are the same, so the preprocessing of the high-level rendering type 1 is also the same. Perspective-dependent processing refers to processing that is related to the current perspective of the terminal-side user of the mobile device 100 (i.e., the perspective of the aforementioned camera); for example, in the latest state of the game scene, although user 1 and user 2 are in the same copy of the same game scene, due to different user perspectives, user 1 and user 2 see different light and shadow effects of the high-level rendering type 1 in the game screen, so the high-level rendering of the high-level rendering type 1 is also different.

[0222] In some embodiments, the method further includes: the runtime of the device-side 3D engine of the mobile device 200 performs basic rendering of the device-side scene 1 to obtain basic rendering data 5; the high-level rendering service of the server 200 determines whether the preprocessing of the high-level rendering type 1 of the cloud-side scene 1 is view-independent processing; if so, the preprocessed data of the high-level rendering type 1 can be shared with other users who are also participating in the device-side scene 1, such as user 2, that is, the device-cloud collaboration framework sends the encoded preprocessed data 4 to the device-cloud collaboration plug-in SDK of the mobile device 200. The device-cloud collaboration plug-in SDK of the mobile device 200 sends the decoded preprocessed data 4 to the runtime of the device-side 3D engine, and the runtime of the device-side 3D engine fuses the basic rendering data 5 with the preprocessed data 4 to obtain an image 5 with the rendering effect of the high-level rendering type 1.

[0223] In the second implementation scheme provided in the embodiment of the present application, the native end-side 3D engine supports the above-mentioned end-cloud collaborative rendering method.

[0224] For example, Figure 9 shows a schematic diagram of the system architecture of a cloud-end collaborative rendering system involved in implementing Solution 2. Compared to the system architecture of deploying the cloud-end collaborative plug-in SDK shown in Figure 5, the main differences of the cloud-end collaborative rendering system shown in Figure 9 are:

[0225] (1) Implementation Plan 1, shown in FIG5, provides the end-cloud collaborative rendering capability for the end-side 3D engine through the end-cloud collaborative plug-in SDK. The end-cloud collaborative plug-in SDK can be obtained through downloading from the app store and browser, and developers can clearly perceive the existence of the SDK. Implementation Plan 2, shown in FIG9, is that the end-side 3D engine natively supports the above-mentioned end-cloud collaborative rendering method. The functions in the end-cloud collaborative plug-in SDK can be built into the end-side 3D engine through modules / pre-installed plug-ins, and developers do not need to download and install the functions through the app store or other channels.

[0226] (2) In the implementation scheme 1 shown in Figure 5, the end-side 3D engine is usually different from the cloud-side 3D engine, so it is necessary to provide scene conversion capabilities in the end-cloud collaboration plug-in SDK. In the implementation scheme 2 shown in Figure 9, when the cloud-side 3D engine is consistent with the end-side 3D engine, the scene conversion function module in the end-cloud collaboration module may no longer be required. It should be noted that even if there is no scene conversion in the implementation scheme 2, the cloud-side scene and the end-side scene may not be completely consistent. For example, the cloud-side scene reduces the accuracy of the mesh information.

[0227] (3) In the implementation scheme 1 shown in FIG5, it is necessary to add the consumption rendering capability of pre-processed data on the basis of the original rendering pipeline, so it is necessary to provide the pipeline adaptation capability in the end-cloud collaborative plug-in SDK. For example, referring to FIG7, the pipeline adaptation function module adds a fusion channel in the end-side rendering pipeline for consuming pre-processed data according to the extension mechanism of the end-side 3D engine rendering pipeline. In the implementation scheme 2 shown in FIG9, the end-cloud collaborative module of the end-side 3D engine can provide a native end-cloud collaborative rendering pipeline, and the end-cloud collaborative rendering pipeline itself can have the consumption capability of pre-processed data; in this way, the pipeline adaptation function module shown in FIG7 is no longer needed.

[0228] In the second implementation scheme of this application, the method flow in the development phase and the method flow in the operation phase are similar to those of the aforementioned implementation scheme one and will not be repeated here.

[0229] In the embodiment of the present application, the first electronic device may be the aforementioned mobile device 100, and the second electronic device may be the aforementioned mobile device 300; the first request may be the aforementioned pre-processing request; the first application may be the aforementioned 3D application (e.g., a game APP); the first end-side scene may be the aforementioned end-side scene 1, the first high-level rendering type may be the aforementioned high-level rendering type 1, and the first cloud-side scene may be the aforementioned cloud-side scene 1; the second high-level rendering type may be the aforementioned high-level rendering type 2, and the second cloud-side scene may be the aforementioned cloud-side scene 2; the first basic rendering data may be the aforementioned basic rendering data 4, the first state data may be the aforementioned state synchronization data 2, and the first pre-processing data may be the aforementioned pre-processing data 4; the first image may be the aforementioned image 4. The first coordinate may be the aforementioned coordinate 1; the third basic rendering data may be the aforementioned basic rendering data 5, and the third image may be the aforementioned image 5

[0230] The following describes the structure of a mobile device 100 provided in an embodiment of the present application. FIG10 shows a schematic diagram of the structure of the mobile device 100.

[0231] The mobile device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0232] It should be understood that the structure illustrated in the embodiments of the present invention does not constitute a specific limitation on the mobile device 100. In other embodiments of the present application, the mobile device 100 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0233] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0234] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0235] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0236] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface.

[0237] The I2C interface is a bidirectional synchronous serial bus that includes a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple I2C busses. The processor 110 may be coupled to the touch sensor 180K, the charger, the flash, the camera 193, and the like via different I2C bus interfaces. For example, the processor 110 may be coupled to the touch sensor 180K via the I2C interface, enabling communication between the processor 110 and the touch sensor 180K via the I2C bus interface, thereby enabling touch functionality of the mobile device 100.

[0238] The I2S interface can be used for audio communication. In some embodiments, the processor 110 can include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to enable communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 via the I2S interface, enabling the function of answering calls through a Bluetooth headset.

[0239] The PCM interface can also be used for audio communication, sampling, quantizing, and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 can be coupled via a PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 via the PCM interface, enabling the function of answering calls via a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0240] The UART interface is a universal serial data bus used for asynchronous communication. This bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is typically used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 via the UART interface to implement Bluetooth functionality. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 via the UART interface, enabling the function of playing music through Bluetooth headphones.

[0241] The MIPI interface can be used to connect the processor 110 to peripheral devices such as the display 194 and the camera 193. MIPI interfaces include the camera serial interface (CSI) and the display serial interface (DSI). In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to implement the camera function of the mobile device 100. The processor 110 and the display 194 communicate via the DSI interface to implement the display function of the mobile device 100.

[0242] The GPIO interface can be configured via software. The GPIO interface can be configured as either a control signal or a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 to the camera 193, display 194, wireless communication module 160, audio module 170, sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0243] The USB interface 130 is an interface that complies with USB standards and may be a Mini USB interface, a Micro USB interface, a USB Type-C interface, or the like. The USB interface 130 can be used to connect a charger to charge the mobile device 100 and to transfer data between the mobile device 100 and peripheral devices. It can also be used to connect headphones to play audio. This interface can also be used to connect other electronic devices, such as augmented reality devices.

[0244] It is understood that the interface connection relationship between the modules illustrated in the embodiment of the present invention is merely an illustrative illustration and does not constitute a structural limitation on the mobile device 100. In other embodiments of the present application, the mobile device 100 may also adopt a different interface connection method from the above embodiment, or a combination of multiple interface connection methods.

[0245] The charging management module 140 is configured to receive charging input from a charger. The charger can be either a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 can receive charging input from the wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 can receive wireless charging input via the wireless charging coil of the mobile device 100. While charging the battery 142, the charging management module 140 can also provide power to the electronic device via the power management module 141.

[0246] The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and provides power to the processor 110, the internal memory 121, the display 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage, impedance). In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0247] The wireless communication function of the mobile device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0248] Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in mobile device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In other embodiments, the antennas can be used in conjunction with a tuning switch.

[0249] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied on the mobile device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0250] The modem processor may include a modulator and a demodulator. The modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to the speaker 170A, the receiver 170B, etc.) or displays an image or video through the display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0251] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc. applied on the mobile device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, demodulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, frequency modulate it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0252] In some embodiments, antenna 1 of mobile device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, so that mobile device 100 can communicate with a network and other devices via wireless communication technologies. The wireless communication technologies may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0253] Mobile device 100 implements display functionality through a GPU, display 194, and an application processor. A GPU is a microprocessor for image processing that connects display 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0254] Display screen 194 is used to display images, videos, and the like. Display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-oLed, or a quantum dot light-emitting diode (QLED). In some embodiments, mobile device 100 may include one or N display screens 194, where N is a positive integer greater than one.

[0255] The mobile device 100 can implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor.

[0256] The ISP processes data fed back by camera 193. For example, when taking a photo, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, which is then passed to the ISP for processing and converted into a visible image. The ISP can also perform algorithmic optimization on image noise and brightness. It can also optimize parameters such as exposure and color temperature of the captured scene. In some embodiments, the ISP can be located within camera 193.

[0257] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then passes the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the mobile device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0258] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the mobile device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0259] Video codecs are used to compress or decompress digital video. Mobile device 100 may support one or more video codecs. This allows mobile device 100 to play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, and MPEG4.

[0260] The NPU is a neural network (NN) computing processor. Drawing on the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it rapidly processes input information and can continuously self-learn. The NPU enables intelligent cognitive applications in the mobile device 100, such as image recognition, face recognition, speech recognition, and text comprehension.

[0261] The internal memory 121 may include one or more random access memories (RAM) and one or more non-volatile memories (NVM).

[0262] Random access memory may include static random-access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM, for example, the fifth generation DDR SDRAM is generally referred to as DDR5 SDRAM), etc.; non-volatile memory may include disk storage devices and flash memory.

[0263] Flash memory can be divided into NOR FLASH, NAND FLASH, 3D NAND FLASH, etc. according to the operating principle; single-level cell (SLC), multi-level cell (MLC), triple-level cell (TLC), quad-level cell (QLC), etc. according to the storage cell potential level; universal flash storage (UFS) and embedded multi media card (eMMC) can be divided into UFS and eMMC according to the storage specification.

[0264] The random access memory can be directly read and written by the processor 110, and can be used to store executable programs (such as machine instructions) of the operating system or other running programs, and can also be used to store user and application data.

[0265] The non-volatile memory may also store executable programs and user and application data, etc., and may be loaded into the random access memory in advance for direct reading and writing by the processor 110 .

[0266] The external memory interface 120 can be used to connect to an external non-volatile memory to expand the storage capacity of the mobile device 100. The external non-volatile memory communicates with the processor 110 via the external memory interface 120 to implement data storage. For example, files such as music and videos can be stored in the external non-volatile memory.

[0267] The mobile device 100 can implement audio functions such as music playback and recording through the audio module 170 , the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0268] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be provided in the processor 110, or some functional modules of the audio module 170 can be provided in the processor 110.

[0269] The speaker 170A, also called a "speaker", is used to convert audio electrical signals into sound signals. The mobile device 100 can listen to music or listen to hands-free calls through the speaker 170A.

[0270] The receiver 170B, also called a "handset", is used to convert audio electrical signals into sound signals. When the mobile device 100 receives a call or a voice message, the user can place the receiver 170B close to the ear to hear the voice.

[0271] Microphone 170C, also known as a "microphone" or "speaker," is used to convert sound signals into electrical signals. When making a call or sending a voice message, a user can speak by approaching microphone 170C with their mouth to input the sound signal into microphone 170C.

[0272] The headphone jack 170D is used to connect a wired headphone and can be the USB interface 130 or a 3.5mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0273] The pressure sensor 180A is used to sense the pressure signal and convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be disposed on the display screen 194 .

[0274] The gyro sensor 180B may be used to determine the motion posture of the mobile device 100. In some embodiments, the angular velocity of the mobile device 100 around three axes (ie, x, y, and z axes) may be determined by the gyro sensor 180B.

[0275] The air pressure sensor 180C is used to measure air pressure. In some embodiments, the mobile device 100 calculates altitude using the air pressure value measured by the air pressure sensor 180C to assist in positioning and navigation.

[0276] The magnetic sensor 180D includes a Hall sensor, and the mobile device 100 can use the magnetic sensor 180D to detect whether the flip cover is opened or closed.

[0277] The acceleration sensor 180E can detect the magnitude of the acceleration of the mobile device 100 in various directions (generally three axes).

[0278] The distance sensor 180F is used to measure distance. The mobile device 100 can measure distance using infrared or laser.

[0279] The proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector such as a photodiode.

[0280] The ambient light sensor 180L is used to sense the brightness of the ambient light. The mobile device 100 can adaptively adjust the brightness of the display screen 194 according to the sensed brightness of the ambient light.

[0281] The fingerprint sensor 180H is used to collect fingerprints. The mobile device 100 can use the collected fingerprint characteristics to implement fingerprint unlocking, access application locks, fingerprint photography, fingerprint call answering, etc.

[0282] The temperature sensor 180J is used to detect temperature. In some embodiments, the mobile device 100 uses the temperature detected by the temperature sensor 180J to execute a temperature processing strategy.

[0283] The touch sensor 180K is also referred to as a "touch-sensitive device." The touch sensor 180K can be disposed on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, also referred to as a "touch screen." The touch sensor 180K is used to detect touch operations applied thereto or in the vicinity thereof. The touch sensor can transmit the detected touch operations to the application processor to determine the type of touch event. Visual output related to the touch operations can be provided via the display screen 194. In other embodiments, the touch sensor 180K can also be disposed on the surface of the mobile device 100, at a location different from that of the display screen 194.

[0284] The bone conduction sensor 180M can obtain a vibration signal. In some embodiments, the bone conduction sensor 180M can obtain a vibration signal of a vibrating bone mass in a human vocal part.

[0285] The buttons 190 include a power button, a volume button, and the like. The buttons 190 may be mechanical buttons or touch buttons. The mobile device 100 may receive key inputs and generate key signal inputs related to user settings and function control of the mobile device 100.

[0286] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts and can also be used for touch vibration feedback.

[0287] The indicator 192 may be an indicator light, which may be used to indicate the charging status, power level changes, messages, missed calls, notifications, etc.

[0288] The SIM card interface 195 is used to connect a SIM card.

[0289] The following describes the structure of a server 200 provided in an embodiment of the present application. FIG11 exemplarily shows the structure of a server 200 provided in an embodiment of the present application.

[0290] As shown in Figure 11, the server 200 may include: one or more processors 1001, memory 1002, communication interface 1003, transmitter 1005, receiver 1006, coupler 1007, and antenna 1008. These components may be connected via a bus 1004 or other means, with Figure 11 using a bus connection as an example.

[0291] The communication interface 1003 can be used for the server 200 to communicate with other communication devices, such as the terminal device 100. Specifically, the communication interface 1003 can be a 3G communication interface, a 4G communication interface, a 5G communication interface, or a communication interface of a future new air interface. Not limited to a wireless communication interface, the server 200 can also be configured with a wired communication interface 1003, such as a local access network (LAN) interface. The transmitter 1005 can be used to transmit and process the signal output by the processor 1001. The receiver 1006 can be used to receive and process the mobile communication signal received by the antenna 1008.

[0292] In some embodiments of the present application, transmitter 1005 and receiver 1006 can be considered a wireless modem. In server 200, the number of transmitters 1005 and receivers 1006 can be one or more. Antenna 1008 can be used to convert electromagnetic energy in a transmission line into electromagnetic waves in free space, or vice versa. Coupler 1007 is used to split the mobile communication signal received by antenna 1008 into multiple paths and distribute them to multiple receivers 1006.

[0293] Memory 1002 is coupled to processor 1001 and is used to store various software programs and / or multiple sets of instructions. Specifically, memory 1002 may include high-speed random access memory and may also include non-volatile memory, such as one or more disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 1002 may store a network communication program that can be used to communicate with one or more additional devices, one or more terminal devices, or one or more network devices.

[0294] In some embodiments of the present application, the memory 1002 may be used to store an implementation program of the application distribution method provided by one or more embodiments of the present application on the server 200 side. For implementation of the application distribution method provided by one or more embodiments of the present application, please refer to the above embodiments.

[0295] The processor 1001 may be configured to read and execute computer-readable instructions. Specifically, the processor 1001 may be configured to call a program stored in the memory 1002, such as a program for implementing the application distribution method provided in one or more embodiments of the present application on the server 200 side, and execute the instructions contained in the program.

[0296] It should be noted that the server 200 shown in FIG11 is only one implementation of the embodiment of the present application. In actual applications, the server 200 may also include more or fewer components, which is not limited here.

[0297] For more details about the functions and working principles of the server 200, please refer to the relevant content in the above embodiments, which will not be repeated here.

[0298] The various implementation modes of this application can be combined arbitrarily to achieve different technical effects.

[0299] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described herein are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated therein. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0300] Those skilled in the art will appreciate that all or part of the process steps in the above-described method embodiments can be implemented by a computer program instructing the relevant hardware. The program can be stored in a computer-readable storage medium, and when executed, the program can include the process steps in the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

[0301] In short, the above description is only an embodiment of the technical solution of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made based on the disclosure of the present invention should be included in the scope of protection of the present invention.

Claims

1. A rendering method for end-cloud collaboration, characterized in that: include: The first electronic device performs basic rendering on the first terminal side scene to obtain first basic rendering data; The first electronic device stores at least one terminal side scenario, and the first terminal side scenario is any one of the at least one terminal side scenario; The first electronic device sends a scene identifier and a first high-level rendering type of the first terminal-side scene to the server; The first electronic device sends first status data to the server, where the first status data includes data related to preprocessing of the first high-order rendering type in the status data of the first terminal side scene; The first electronic device receives the first pre-processed data sent by the server; The first pre-processed data is obtained by performing pre-processing of the first high-level rendering type on the first cloud-side scene after the server updates the first cloud-side scene according to the first state data; the server stores the first cloud-side scene converted from the first terminal-side scene for the first high-level rendering type; The first electronic device acquires a first image according to the first basic rendering data and the first pre-processed data, wherein the first image has a rendering effect of the first high-order rendering type; The first electronic device displays the first image.

2. The method according to claim 1, characterized in that The method further comprises: The first electronic device updates the scene information of the first terminal side scene; The first electronic device performs basic rendering on the updated first terminal-side scene to obtain second basic rendering data; When the network speed between the first electronic device and the server is lower than a preset value or the network between the first electronic device and the server is disconnected, the first electronic device determines a second image according to the second basic rendering data; The first electronic device displays the second image.

3. The method according to claim 1, characterized in that The first electronic device sends the scene identifier and the first high-order rendering type of the first terminal-side scene to the server, including: The first electronic device sends a first request to the server, the first request including a scene identifier of the first terminal-side scene and a first high-order rendering type; the first request is used to trigger the server to perform preprocessing preparation, the preprocessing preparation including loading a first cloud-side scene and preparing a rendering pipeline for preprocessing of the first high-order rendering type; Before the first electronic device sends the first status data to the server, the method further includes: The first electronic device receives confirmation information sent by the server, where the confirmation information is used to indicate that the preprocessing preparation has been completed; The first electronic device sending first status data to the server includes: Based on the determination information, the first electronic device sends first status data to the server.

4. The method according to any one of claims 1 to 3, characterized in that: The state data of the scene on the first end side includes part or all of the following: scene identification information, light source information, character information, camera information and scene update information.

5. The method according to any one of claims 1 to 4, characterized in that: The server stores a first correspondence between the first terminal-side scene, the first high-level rendering type, and the first cloud-side scene.

6. The method according to any one of claims 1 to 5, characterized in that: The scene identifier of the first terminal side scene and the first high-level rendering type are obtained by the first electronic device from the application package of the first application; The scene identifier of the first cloud-side scene, and the first corresponding relationship between the first terminal-side scene, the first high-order rendering type, and the first cloud-side scene are deployed on the server during the application development of the first application.

7. The method according to any one of claims 1 to 6, characterized in that: The server stores a second cloud-side scene converted from the first terminal-side scene for a second high-level rendering type.

8. The method according to any one of claims 1 to 7, characterized in that: The first electronic device receives the first pre-processed data sent by the server, including: The first electronic device receives the first pre-processed data after compression encoding sent by the server; Before the first electronic device acquires the first image according to the first basic rendering data and the first pre-processing data, the method further includes: The first electronic device decodes the received first pre-processed data to obtain the decoded first pre-processed data.

9. The method according to any one of claims 1 to 8, characterized in that: The first high-level rendering type is global illumination; The first pre-processed data includes the irradiance of each pixel in the imaging picture obtained by pre-processing the global illumination, and the first basic rendering data includes the irradiance of each pixel in the imaging picture after basic rendering; The irradiance of the pixel at the first coordinate in the first image is equal to the product of the irradiance of the pixel at the first coordinate in the first preprocessing data and the first basic rendering data, and the pixel value of the pixel at the first coordinate in the first image is determined according to the irradiance of the pixel at the first coordinate.

10. A rendering method for end-cloud collaboration, characterized in that: The method comprises: The server receives a scene identifier and a first high-level rendering type of the first terminal-side scene sent by the first electronic device; The server receives first status data sent by the first electronic device, where the first status data includes data related to preprocessing of the first high-order rendering type in status data of the first terminal-side scene; The server updates the first cloud-side scene according to the first state data, performs preprocessing of the first high-order rendering type on the updated first cloud-side scene, and obtains first preprocessing data; The server sends the first pre-processed data to the first electronic device; The first pre-processed data is used to fuse the first basic rendering data to obtain a first image; the first image has the light and shadow effects of the first high-order rendering type; the first basic rendering data is obtained by the first electronic device performing basic rendering on the first end-side scene.

11. The method according to claim 10, characterized in that The server receives the scene identifier and the first high-order rendering type of the first terminal-side scene sent by the first electronic device, including: The server receives a first request sent by the first electronic device, where the first request includes a scene identifier of the first terminal-side scene and a first high-level rendering type; Before the server receives the first status data sent by the first electronic device, the method further includes: Based on the first request, the server performs preprocessing preparation, wherein the preprocessing preparation includes loading a first cloud-side scene and preparing a preprocessing rendering pipeline for the first high-order rendering type; The server sends confirmation information to the first electronic device, where the confirmation information is used to indicate that the preprocessing preparation has been completed; The first status data is sent by the first electronic device based on the determination information.

12. The method according to claim 10 or 11, characterized in that: The state data of the scene on the first end side includes part or all of the following: scene identification information, light source information, character information, camera information and scene update information.

13. The method according to any one of claims 10 to 12, characterized in that: The server stores a first correspondence between the first terminal-side scene, the first high-level rendering type, and the first cloud-side scene. Before the server updates the first cloud-side scene according to the first status data, the method further includes: The server determines, based on the first corresponding relationship, the scene identifier of the first terminal-side scene, and the first high-order rendering type, that the cloud-side scene to be rendered is the first cloud-side scene.

14. The method according to any one of claims 11 to 13, characterized in that: The scene identifier of the first terminal side scene and the first high-level rendering type are obtained by the first electronic device from the application package of the first application; The scene identifier of the first cloud-side scene, and the first corresponding relationship between the first terminal-side scene, the first high-order rendering type, and the first cloud-side scene are deployed on the server during the application development of the first application.

15. The method according to any one of claims 10 to 14, characterized in that The server stores a second cloud-side scene converted from the first terminal-side scene for a second high-level rendering type.

16. The method according to any one of claims 10 to 15, characterized in that The server sending the first pre-processed data to the first electronic device includes: The server sends the first pre-processed data after compression encoding to the first electronic device.

17. The method according to any one of claims 10 to 16, characterized in that The first high-level rendering type is global illumination; The first pre-processed data includes the irradiance of each pixel in the imaging picture obtained by pre-processing the global illumination, and the first basic rendering data includes the irradiance of each pixel in the imaging picture after basic rendering; The irradiance of the pixel at the first coordinate in the first image is equal to the product of the irradiance of the pixel at the first coordinate in the first preprocessing data and the first basic rendering data, and the pixel value of the pixel at the first coordinate in the first image is determined according to the irradiance of the pixel at the first coordinate.

18. The method according to any one of claims 10 to 16, characterized in that The method further comprises: If the first preprocessed data is perspective-independent preprocessed data, the server sends the first preprocessed data to the second electronic device; the first electronic device and the second electronic device are electronic devices of users participating in the same copy of the first end-side scene.

19. An electronic device, characterized in that: include: One or more processors, one or more memories; the one or more memories are coupled to the one or more processors; The one or more memories are used to store a computer program, and when the computer program runs on the processor, the electronic device executes the method according to any one of claims 1 to 9.

20. A server, characterized in that: include: One or more processors, one or more memories; the one or more memories are coupled to the one or more processors; The one or more memories are used to store a computer program, and when the computer program is executed on the processor, the server executes the method according to any one of claims 10 to 18.

21. A computer readable medium, characterized in that A computer program is stored, and the computer program can be executed by a processor to implement the method according to any one of claims 1 to 18.

Images