Edge-cloud collaborative rendering method and related apparatus

Abstract

The application discloses an end-cloud cooperative rendering method, which comprises the following steps: an electronic device performs basic rendering on a first end-side scene and obtains first basic rendering data; the electronic device sends a scene identifier of the first end-side scene and a first high-order rendering type to a server; the server stores a first cloud-side scene converted by the first end-side scene for the first high-order rendering type; the electronic device sends first state data to the server; the server performs preprocessing of the first high-order rendering type on the first cloud-side scene after updating the first cloud-side scene according to the first state data, and obtains first preprocessing data; the server sends the first preprocessing data to the electronic device; and the electronic device obtains a first image according to the first basic rendering data and the first preprocessing data, and displays the first image. In this way, the requirement and load of the GPU power of the end side are reduced, the electronic device with weak GPU power can also present the high-order rendering effect, and the user experience is effectively improved.

Description

Technical Field

[0001] This application relates to the field of electronic technology, and in particular to rendering methods and related devices for edge-cloud collaboration. Background Technology

[0002] The number of three-dimensional (3D) applications (such as 3D games and 3D modeling software) on mobile devices and other terminal devices is increasing. Providing rendering capabilities for these 3D applications is the core function of terminal devices supporting their operation. However, the graphics processing units (GPUs) (referred to as lightweight GPUs for ease of description) of terminal devices (such as mobile devices) with limited size and power consumption lag significantly behind those of personal computer (PC) GPUs. For example, in 2021, Apple's A15 had a single-precision computing power of 1.5 T FLOPS; while in 2010, the NVIDIA GTX 580 already had a single-precision computing power of 1.58 T FLOPS; and in 2021, the NVIDIA RTX 3080 Ti reached a computing power of 27.955 TFLOPS. This shows that the development of computing power in mobile device lightweight GPUs is about 10 years behind that of PC-level GPUs.

[0003] Cloud-based server GPUs (i.e., PC-level GPUs) have a significant computing power advantage over the lightweight GPUs of mobile devices. With the development of cloud computing technology, transferring computing power from the edge to the cloud and coordinating cloud processing for edge-side tasks is gradually becoming a technological evolution direction. Currently, how to achieve efficient rendering processing in collaboration with the cloud to achieve ray tracing rendering effects on the edge still requires further research. Summary of the Invention

[0004] This application provides a rendering method and related apparatus for edge-cloud collaboration, which reduces the computing power requirements and load on edge GPUs, enabling electronic devices with weaker GPU computing power to present high-level rendering effects, such as the rendering effects of ray tracing technology, effectively improving the user experience.

[0005] In a first aspect, this application provides an edge-cloud collaborative rendering method applied to a communication system. The communication system includes a first electronic device and a server. The first electronic device stores at least one edge scene of a first application, where the first edge scene is any one of the at least one edge scene. The server stores a first cloud scene converted from the first edge scene to a first high-order rendering type. The method includes: the first electronic device performing basic rendering on the first edge scene to obtain first basic rendering data; the first electronic device sending a scene identifier and a first high-order rendering type of the first edge scene to the server; the first electronic device sending first state data to the server, the first state data including data related to preprocessing of the first high-order rendering type in the state data of the first edge scene; the server updating the first cloud scene according to the first state data, performing preprocessing of the first high-order rendering type on the updated first cloud scene, and obtaining first preprocessed data; the server sending the first preprocessed data to the first electronic device; the first electronic device obtaining a first image based on the first basic rendering data and the first preprocessed data, the first image having the lighting and shadow effects of the first high-order rendering type; and the first electronic device displaying the first image.

[0006] Implementing the embodiments of this application, a single edge scene is transformed into a cloud-side scene for each high-order rendering type, and the server stores the cloud-side scenes. For a specific high-order rendering type, the electronic device only needs to have the ability to perform basic rendering on the edge scene, while the server performs high-order rendering preprocessing on the corresponding cloud-side scene. After the basic rendering data is fused with the high-order rendering preprocessing data, the final rendered image presents the lighting and shadow effects of the specific high-order rendering type. In this way, the computing power requirements and load on the edge GPU can be reduced for various high-order rendering types; electronic devices (such as mobile phones, tablets, etc.) do not need to have high-standard hardware performance to present high-order rendering effects. In addition, since the server only performs high-order rendering preprocessing and does not run the full application, the consumption of cloud-side computing resources is reduced.

[0007] In one implementation, the method further includes: a first electronic device updating scene information of a first terminal scene; the first electronic device performing basic rendering on the updated first terminal scene to obtain second basic rendering data; in the event that the network speed between the first electronic device and the server is lower than a preset value, or the network between the first electronic device and the server is disconnected, the first electronic device determining a second image based on the second basic rendering data; and the first electronic device displaying the second image. By implementing this embodiment, even under poor network conditions, the electronic device can still perform basic rendering, ensuring the normal operation of the game app and thus guaranteeing the user's basic operational experience.

[0008] In one implementation, the first electronic device sends a scene identifier of a first edge-side scene and a first higher-order rendering type to the server, including: the first electronic device sending a first request to the server, the first request including the scene identifier of the first edge-side scene and the first higher-order rendering type; before the first electronic device sends first state data to the server, the method further includes: based on the first request, the server performing preprocessing preparation, the preprocessing preparation including loading a first cloud-side scene and preparing a rendering pipeline for preprocessing of the first higher-order rendering type; the server sending confirmation information to the first electronic device, the confirmation information indicating that preprocessing preparation has been completed; the first electronic device sending first state data to the server includes: based on the confirmation information, the first electronic device sending first state data to the server. Implementing this embodiment of the application, state data is uploaded to the server only after confirming that the server has completed preprocessing preparation; this avoids invalid uploading of state data and wasted network traffic due to the server's inability to perform higher-order rendering.

[0009] In one implementation, the state data of the first-side scene includes some or all of the following: scene identification information, light source information, character information, camera information, and scene update information.

[0010] In one implementation, the server stores a first correspondence between a first client-side scene, a first higher-order rendering type, and a first cloud-side scene. Before the server updates the first cloud-side scene based on first state data, the method further includes: the server determining the cloud-side scene to be rendered as the first cloud-side scene based on the first correspondence, the scene identifier of the first client-side scene, and the first higher-order rendering type. Implementing this embodiment, the server stores the correspondence between the first client-side scene, the first higher-order rendering type, and the first cloud-side scene to facilitate rapid location of the cloud-side scene to be rendered based on the scene identifier and the first higher-order rendering type of the first client-side scene uploaded from the client.

[0011] In one implementation, the scene identifier and the first higher-order rendering type of the first terminal scene are obtained by the first electronic device from the application package of the first application; the scene identifier of the first cloud scene, as well as the first correspondence between the first terminal scene, the first higher-order rendering type and the first cloud scene, are deployed on the server during the application development of the first application.

[0012] In one implementation, the server stores a second cloud-side scene converted from a first client-side scene for a second higher-order rendering type. In this embodiment, a client-side scene is converted into a cloud-side scene for each higher-order rendering type.

[0013] In one implementation, the server sends first preprocessed data to the first electronic device, including: the server sending compressed and encoded first preprocessed data to the first electronic device; before the first electronic device obtains the first image based on the first basic rendering data and the first preprocessed data, the method further includes: the first electronic device decoding the received first preprocessed data to obtain decoded first preprocessed data. By implementing the embodiments of this application, the server feeding back compressed and encoded preprocessed data to the first electronic device can reduce the amount of data transmitted, thereby reducing transmission latency.

[0014] In one implementation, the first higher-order rendering type is global illumination; the first preprocessing data includes the irradiance of each pixel in the image obtained by preprocessing with global illumination, and the first preprocessing data includes the irradiance of each pixel in the image after basic rendering; the irradiance of the pixel at the first coordinate in the first image is equal to the product of the irradiance of the pixel at the first coordinate in the first image and the irradiance of the pixel at the first coordinate in the first basic rendering data, and the pixel value of the pixel at the first coordinate in the first image is determined based on the irradiance of the pixel at the first coordinate.

[0015] In one implementation, the system further includes a second electronic device, wherein the first and second electronic devices are electronic devices belonging to users participating in the same replica of the first edge-side scenario; the method further includes: if the first preprocessed data is view-independent preprocessed data, the server sends the first preprocessed data to the second electronic device. Implementing this embodiment, if multiple users are in the same replica of the edge-side scenario, the server can share view-independent preprocessed data in that replica with all users in that replica, reducing the consumption of cloud-side computing resources.

[0016] In one implementation, the method further includes: a second electronic device performing basic rendering on a first-side scene to obtain third basic rendering data; the second electronic device obtaining a third image based on the third basic rendering data and the first preprocessed data, the third image having lighting and shadow effects of a first high-order rendering type; and the second electronic device displaying the third image.

[0017] Secondly, this application provides a cloud-edge collaborative rendering method, comprising: a first electronic device performing basic rendering on a first edge scene to obtain first basic rendering data; the first electronic device storing at least one edge scene, wherein the first edge scene is any one of the at least one edge scene; the first electronic device sending a scene identifier and a first higher-order rendering type of the first edge scene to a server; the first electronic device sending first state data to the server, wherein the first state data includes data related to preprocessing of the first higher-order rendering type in the state data of the first edge scene; the first electronic device receiving first preprocessing data sent by the server; the first preprocessing data is obtained by the server performing preprocessing of the first higher-order rendering type on the first cloud scene after updating the first cloud scene according to the first state data; the server storing a first cloud scene converted from the first edge scene for the first higher-order rendering type; the first electronic device obtaining a first image based on the first basic rendering data and the first preprocessing data, wherein the first image has the rendering effect of the first higher-order rendering type; and the first electronic device displaying the first image.

[0018] Implementing the embodiments of this application, a single edge scene is transformed into a cloud-side scene for each high-order rendering type, and the server stores the cloud-side scenes. For a specific high-order rendering type, the electronic device only needs to have the ability to perform basic rendering on the edge scene, while the server performs high-order rendering preprocessing on the corresponding cloud-side scene. After the basic rendering data is fused with the high-order rendering preprocessing data, the final rendered image presents the lighting and shadow effects of the specific high-order rendering type. In this way, the computing power requirements and load on the edge GPU can be reduced for various high-order rendering types; electronic devices (such as mobile phones, tablets, etc.) do not need to have high-standard hardware performance to present high-order rendering effects. In addition, since the server only performs high-order rendering preprocessing and does not run the full application, the consumption of cloud-side computing resources is reduced.

[0019] In one implementation, the method further includes: the first electronic device updating scene information of the first terminal scene; the first electronic device performing basic rendering on the updated first terminal scene to obtain second basic rendering data; if the network speed between the first electronic device and the server is lower than a preset value, or if the network between the first electronic device and the server is disconnected, the first electronic device determines a second image based on the second basic rendering data; and the first electronic device displays the second image.

[0020] In one implementation, the first electronic device sends a scene identifier of a first edge scene and a first higher-order rendering type to the server, including: the first electronic device sending a first request to the server, the first request including the scene identifier of the first edge scene and the first higher-order rendering type; the first request is used to trigger the server to perform preprocessing preparation, the preprocessing preparation including loading a first cloud-side scene and preparing a rendering pipeline for preprocessing of the first higher-order rendering type; before the first electronic device sends first state data to the server, the method further includes: the first electronic device receiving confirmation information sent by the server, the confirmation information indicating that preprocessing preparation has been completed; the first electronic device sending first state data to the server includes: based on the confirmation information, the first electronic device sending first state data to the server.

[0021] In one implementation, the state data of the first-side scene includes some or all of the following: scene identification information, light source information, character information, camera information, and scene update information.

[0022] In one implementation, the server stores a first correspondence between a first client-side scene, a first higher-order rendering type, and a first cloud-side scene.

[0023] In one implementation, the scene identifier and the first higher-order rendering type of the first terminal scene are obtained by the first electronic device from the application package of the first application; the scene identifier of the first cloud scene, as well as the first correspondence between the first terminal scene, the first higher-order rendering type and the first cloud scene, are deployed on the server during the application development of the first application.

[0024] In one implementation, the server stores a second cloud-side scene converted from the first end-side scene to the second higher-order rendering type.

[0025] In one implementation, the first electronic device receives first preprocessed data sent by the server, including: the first electronic device receiving compressed and encoded first preprocessed data sent by the server; before the first electronic device obtains the first image based on the first basic rendering data and the first preprocessed data, the method further includes: the first electronic device decoding the received first preprocessed data to obtain the decoded first preprocessed data.

[0026] In one implementation, the first higher-order rendering type is global illumination; the first preprocessing data includes the irradiance of each pixel in the image obtained by preprocessing with global illumination, and the first basic rendering data includes the irradiance of each pixel in the image after basic rendering; the irradiance of the pixel at the first coordinate in the first image is equal to the product of the irradiance of the pixel at the first coordinate in the first preprocessing data and the first basic rendering data, and the pixel value of the pixel at the first coordinate in the first image is determined based on the irradiance of the pixel at the first coordinate.

[0027] Thirdly, this application provides an edge-cloud collaborative rendering method, the method comprising: a server receiving a scene identifier and a first high-order rendering type of a first edge scene sent by a first electronic device; the server receiving first state data sent by the first electronic device, the first state data including data related to preprocessing of the first high-order rendering type in the state data of the first edge scene; the server updating a first cloud scene according to the first state data, performing preprocessing of the first high-order rendering type on the updated first cloud scene, and obtaining first preprocessed data; the server sending the first preprocessed data to the first electronic device; the first preprocessed data being used to fuse first basic rendering data to obtain a first image; the first image having the lighting and shadow effects of the first high-order rendering type; the first basic rendering data being obtained by the first electronic device performing basic rendering on the first edge scene.

[0028] Implementing the embodiments of this application, a single edge scene is transformed into a cloud-side scene for each high-order rendering type, and the server stores the cloud-side scenes. For a specific high-order rendering type, the electronic device only needs to have the ability to perform basic rendering on the edge scene, while the server performs high-order rendering preprocessing on the corresponding cloud-side scene. After the basic rendering data is fused with the high-order rendering preprocessing data, the final rendered image presents the lighting and shadow effects of the specific high-order rendering type. In this way, the computing power requirements and load on the edge GPU can be reduced for various high-order rendering types; electronic devices (such as mobile phones, tablets, etc.) do not need to have high-standard hardware performance to present high-order rendering effects. In addition, since the server only performs high-order rendering preprocessing and does not run the full application, the consumption of cloud-side computing resources is reduced.

[0029] In one implementation, the server receives a scene identifier and a first higher-order rendering type of a first edge scene sent by a first electronic device, including: the server receiving a first request sent by the first electronic device, the first request including a scene identifier and a first higher-order rendering type of the first edge scene; before the server receives first state data sent by the first electronic device, the method further includes: based on the first request, the server performing preprocessing preparation, the preprocessing preparation including loading a first cloud-side scene and preparing a rendering pipeline for preprocessing of the first higher-order rendering type; the server sending confirmation information to the first electronic device, the confirmation information indicating that preprocessing preparation has been completed; the first state data is sent by the first electronic device based on the confirmation information.

[0030] In one implementation, the state data of the first-side scene includes some or all of the following: scene identification information, light source information, character information, camera information, and scene update information.

[0031] In one implementation, the server stores a first correspondence between a first terminal scene, a first higher-order rendering type, and a first cloud scene. Before the server updates the first cloud scene based on the first state data, the method further includes: the server determining the cloud scene to be rendered as the first cloud scene based on the first correspondence, the scene identifier of the first terminal scene, and the first higher-order rendering type.

[0032] In one implementation, the scene identifier and the first higher-order rendering type of the first terminal scene are obtained by the first electronic device from the application package of the first application; the scene identifier of the first cloud scene, as well as the first correspondence between the first terminal scene, the first higher-order rendering type and the first cloud scene, are deployed on the server during the application development of the first application.

[0033] In one implementation, the server stores a second cloud-side scene converted from the first end-side scene to the second higher-order rendering type.

[0034] In one implementation, the server sends first preprocessed data to the first electronic device, including: the server sending compressed and encoded first preprocessed data to the first electronic device.

[0035] In one implementation, the first higher-order rendering type is global illumination; the first preprocessing data includes the irradiance of each pixel in the image obtained by preprocessing with global illumination, and the first basic rendering data includes the irradiance of each pixel in the image after basic rendering; the irradiance of the pixel at the first coordinate in the first image is equal to the product of the irradiance of the pixel at the first coordinate in the first preprocessing data and the first basic rendering data, and the pixel value of the pixel at the first coordinate in the first image is determined based on the irradiance of the pixel at the first coordinate.

[0036] In one implementation, the system further includes a second electronic device, wherein the first electronic device and the second electronic device are electronic devices of the same user participating in the first end-side scene; the method further includes: if the first preprocessed data is view-independent preprocessed data, the server sends the first preprocessed data to the second electronic device.

[0037] Fourthly, embodiments of this application provide an electronic device, which includes a processor and a memory, the memory being coupled to the processor, the memory being used to store computer program code, the computer program code including computer instructions, and when the processor reads the computer instructions from the memory, causing the electronic device to execute the end-to-cloud collaborative image enhancement method described in the first aspect.

[0038] Fifthly, embodiments of this application provide a server, the server including: a processor and a memory, the memory being coupled to the processor, the memory being used to store computer program code, the computer program code including computer instructions, and when the processor reads the computer instructions from the memory, causing the server to execute the end-to-cloud collaborative rendering method described in the second aspect.

[0039] Sixthly, embodiments of this application provide a computer storage medium including computer instructions, which, when executed on an electronic device, cause a communication device to execute the end-to-cloud collaborative rendering method in any of the possible implementations of any of the above aspects.

[0040] Seventhly, embodiments of this application provide a computer program product that, when run on a computer, causes the computer to execute the end-to-cloud collaborative rendering method in any of the possible implementations of any of the above aspects. Attached Figure Description

[0041] Figure 1 A system architecture diagram of the communication system provided in the embodiments of this application;

[0042] Figure 2A A system architecture diagram of a rendering system provided in this application embodiment;

[0043] Figure 2B A system architecture diagram of another rendering system provided in this application embodiment;

[0044] Figure 3A This application provides a GI-free rendering effect in its embodiments.

[0045] Figure 3B The rendering effect with GI provided in the embodiments of this application;

[0046] Figure 3C The non-reflective rendering effect provided in the embodiments of this application;

[0047] Figure 3D The rendering effect with reflection provided in the embodiments of this application;

[0048] Figure 4 A data flow diagram provided for an embodiment of this application;

[0049] Figure 5 A system architecture diagram of another rendering system provided in this application embodiment;

[0050] Figure 6 A flowchart illustrating the development phase of the edge-cloud collaborative rendering method provided for embodiments of this application is shown.

[0051] Figure 7 A schematic diagram illustrating end-to-end cloud collaborative rendering for embodiments of this application;

[0052] Figure 8 A flowchart illustrating the runtime phase of the edge-cloud collaborative rendering method provided in this application embodiment;

[0053] Figure 9 A system architecture diagram of another rendering system provided in this application embodiment;

[0054] Figure 10 This is a schematic diagram of the structure of a mobile device provided in an embodiment of this application;

[0055] Figure 11 This is a schematic diagram of the server structure provided in an embodiment of this application. Detailed Implementation

[0056] The technical solutions in the embodiments of this application will be clearly and thoroughly described below with reference to the accompanying drawings. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B; the word "and / or" in the text is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, in the description of the embodiments of this application, "multiple" refers to two or more than two.

[0057] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature, and in the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0058] The term "user interface (UI)" used in the following embodiments of this application refers to the medium interface through which an application or operating system interacts and exchanges information with the user. It realizes the conversion between the internal form of information and the form that the user can accept. The user interface is source code written in a specific computer language such as Java or Extensible Markup Language (XML). The interface source code is parsed and rendered on the electronic device, ultimately presenting content that the user can recognize. A common form of user interface is the graphical user interface (GUI), which refers to a user interface related to computer operation displayed graphically. It can be visible interface elements such as text, icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, and widgets displayed on the screen of an electronic device.

[0059] 3D rendering technology is the process of projecting a model in a pre-built 3D scene into a two-dimensional digital image according to a pre-defined viewpoint, light source, and material information.

[0060] Traditional 3D rendering techniques, such as rasterization, divide the object model in a 3D scene into triangles, transform the three-dimensional coordinates of the triangle vertices into two-dimensional coordinates on the image through geometric transformations, and finally fill the triangles on the image with textures, thereby mapping the 3D object model onto a two-dimensional screen to achieve image rendering. This technique usually has difficulty in realistically reproducing the light reflection, object shadows, and refraction effects in a 3D scene, so the rendered image is difficult to present a realistic 3D visual experience.

[0061] Compared to traditional 3D rendering techniques, ray tracing technology provides a more realistic 3D visual experience. Ray tracing technology simulates the propagation of light in a 3D scene through reflection, refraction, shadows, and scattering, calculating the color and brightness values ​​of each pixel. The resulting 2D image's lighting effects closely resemble real-world physical laws, thus simulating a more realistic 3D virtual scene on electronic devices.

[0062] Ray tracing technology requires simulating a vast number of light paths to achieve visual effects that closely resemble the real world. This means that the GPU computing power requirements of this technology increase explosively with the number of simulated light rays. NVIDIA first implemented real-time hardware ray tracing technology on PC-level GPUs in its RTX 20 series graphics cards in 2018; however, due to power consumption and computing power limitations, most mobile device edge GPUs currently cannot implement hardware ray tracing technology, and large-scale commercial deployment requires further research. In the edge-cloud collaborative rendering method provided in this application embodiment, the mobile device collaborates with the cloud-side PC-level GPU to render 3D scenes, thereby presenting the rendering effects achievable by ray tracing technology on the mobile device, reducing the computing power requirements and load on the mobile device's edge GPU.

[0063] The communication system used in the end-to-end cloud collaborative rendering method provided in this application embodiment is described below.

[0064] Figure 1 The system architecture of the communication system 10 provided in an embodiment of this application is illustrated by way of example. Figure 1 As shown, the communication system 10 includes a terminal device 100 and cloud-side infrastructure, which includes a server 200. The terminal device 100 can communicate with the cloud-side infrastructure through a communication network, and the cloud-side infrastructure can provide rendering services for 3D applications (APPs) to the terminal device 100.

[0065] The aforementioned 3D applications provide 3D visual effects, where the 3D visuals are two-dimensional images rendered from models within a 3D scene. Examples include 3D game apps, 3D modeling apps, 3D navigation apps, 3D home decoration apps, etc. Subsequent embodiments will use a game app as an example for illustrative purposes.

[0066] Terminal device 100 can be a mobile phone, tablet computer, desktop computer, laptop computer, handheld computer, notebook computer, ultra-mobile personal computer (UMPC), netbook, cellular phone, personal digital assistant (PDA), augmented reality (AR) device, virtual reality (VR) device, artificial intelligence (AI) device, wearable device (e.g., smart bracelet), in-vehicle device, smart home device (e.g., smart TV, smart screen, large screen device, etc.) and / or smart city device. This application embodiment does not impose special limitations on the specific type of terminal device 100; terminal device 100 can also be referred to as an edge device or electronic device.

[0067] The terminal device 100 can be a mobile device or a non-mobile device. Mobile devices are generally smaller, which limits their GPU computing power. Therefore, the advantages of the edge-cloud collaborative rendering scheme provided in this application are more significant. The following embodiments will use the mobile device 100 as an example for illustrative purposes.

[0068] Server 200 can be a single server, a server cluster consisting of multiple servers, or a cloud computing center. The server 200 involved in this application embodiment can also be referred to as a cloud server, cloud-side, or cloud-based. It is not limited to server 200; cloud-side infrastructure can also include many other devices, which are not specifically limited here.

[0069] The aforementioned communication network may include local area networks (LANs) and / or wide area networks (WANs). This communication network can be implemented using any known network communication protocol, which may be a variety of wired or wireless communication protocols, such as Ethernet, Universal Serial Bus (USB), FireWire, Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), Bluetooth, Wireless Fidelity (Wi-Fi), NFC, Voice over Internet Protocol (VoIP), communication protocols supporting network slicing architecture, or any other suitable communication protocol.

[0070] It should be understood that Figure 1 The system architecture diagram of the communication system provided in this application embodiment is merely a schematic diagram and does not constitute a specific limitation on the communication system 10. The communication system 10 may include more or fewer devices than shown in the diagram. For example, it may also include wireless relay devices and wireless backhaul devices. Figure 1 (not shown in the text), which is not limited here.

[0071] In the edge-cloud collaborative rendering method (i.e., method one) provided in this application embodiment, all rendering work of the game APP is transferred to the cloud side, utilizing the GPU computing power of the cloud side to improve the rendering effect of the mobile device. For example, as... Figure 2A As shown, in this method, the mobile device 100 does not need to install the game, but only the game client of the game APP. The game client includes an operation instruction processing module and a video decoding module. The server 200 can run the game APP, which includes a rendering module and a logic module. All rendering work of the game APP is performed in the virtual host / container of the server 200.

[0072] The mobile device 100's operation command processing module is responsible for collecting user operation command information (such as left and right movement, view switching, clicking, etc.) and uploading the operation commands to the cloud-based game app. The cloud-based game app sends the received operation command information to the logic module. After updating the game state based on the operation command information, the logic module triggers the rendering module to render the game scene in the latest state. The cloud-based game app then transmits the rendered game screen to the mobile device 100's game client as a video stream. The mobile device 100's video decoding module decodes the video stream to obtain the game screen and triggers the mobile device 100 to display the game screen. Optionally, relying on the powerful computing power of the cloud-based PC-level GPU, the cloud-based rendering module can use ray tracing technology to render the game screen corresponding to the latest state of the game scene.

[0073] Implementing the above-mentioned edge-cloud collaborative rendering method can reduce the computing power requirements and load on the edge GPU. However, the above rendering method still has the following problems: (1) All game rendering tasks are executed on the cloud side, the cloud side has high computing costs, and the edge side is only used as a video player, which wastes the edge GPU resources. (2) The rendering results on the cloud side cannot be shared among multiple users, which further increases the computing costs on the cloud side. (3) High transmission bandwidth costs. Using video streaming to transmit the rendered game screen, the transmission bandwidth is about 10Mbps at 1080P@60fps, resulting in high outbound bandwidth costs for the cloud server and downlink bandwidth costs for users. (4) Poor experience in weak networks. Mobile users often encounter poor network conditions; when the network is poor, the user's operation response latency is very high, affecting the game operation experience.

[0074] In another edge-cloud collaborative rendering method (i.e., method two) provided in this application embodiment, the edge performs basic rendering of the game scene, while the cloud performs preprocessing for high-order rendering of the game scene. The edge fuses the basic rendering data and the preprocessed high-order rendering data to obtain the game screen of the game scene after high-order rendering. For example, as shown... Figure 2B As shown, in this method, the mobile device 100 can install a game app, which includes an operation instruction processing module, a rendering module, and a logic module; the server 200 includes a high-level rendering service.

[0075] The operation command processing module of mobile device 100 is responsible for collecting user operation command information and sending the operation commands to the logic module of mobile device 100. After the logic module updates the game state according to the operation command information, it triggers the rendering module of mobile device 100 to perform basic rendering of the game scene 1 (i.e., the client-side scene 1) in the latest state and sends state synchronization data to server 200. The advanced rendering service of server 200 updates the cloud-side scene 1 corresponding to client-side scene 1 according to the state synchronization data; it performs advanced rendering preprocessing based on the updated cloud-side scene 1 and sends the preprocessed data to mobile device 100. After the rendering module of mobile device 100 merges the basic rendering data and the advanced rendering preprocessed data, it can obtain the final rendered game screen, which has advanced rendering effects and can achieve the rendering effects achievable by ray tracing technology. If multiple players are in the same game scene instance, server 200 can share the advanced rendering preprocessed data, which is independent of the viewpoint, with all players in the instance.

[0076] Implementing the embodiments of this application can achieve the following beneficial effects: Only basic rendering and blending are performed on the device side, without the need for high-level rendering ray tracing, reducing the computational requirements and load on the device-side GPU; gamers' mobile devices (such as phones and tablets) do not need high-standard hardware performance to achieve the rendering effects achievable by ray tracing technology, allowing for smooth 3D gaming. Furthermore, since the cloud side only performs high-level rendering preprocessing and does not run the full game app, the consumption of cloud-side computing resources is reduced; simultaneously, the high-level rendering service on the cloud side can share the preprocessed data for high-level rendering of the same instance with the corresponding device-side clients of multiple game users in the same instance, further reducing the consumption of cloud-side computing resources. The data transmitted from the cloud side to the device side can be compressed high-level rendering preprocessed data, requiring lower transmission bandwidth compared to transmitting video streams (e.g., 720p / 1080p@30fps / 60fps, 1080p@30fps video streams); compared to the aforementioned method one, due to the lower required transmission bandwidth, transmission latency is lower under poor network conditions, reducing the impact of weak networks on the user's operating experience. Furthermore, even under poor network conditions, basic rendering can still be performed on the client side, ensuring the normal operation of the game app and thus guaranteeing the user's basic operating experience.

[0077] The following section provides a detailed introduction to Method 2, starting with an introduction to the technical concepts involved in the aforementioned end-to-cloud collaborative rendering method.

[0078] A 3D scene (i.e., a virtual scene) refers to a computer-generated virtual three-dimensional environment (such as a city, park, forest, street, or mountain). It provides a multimedia virtual world where users can control operable virtual objects through devices or interfaces, observing objects, animals, people, and scenery from the perspective of these virtual objects. The mobile device 100 can construct a 3D object model based on each virtual object, thereby building the entire scene model. For example, a scene in a game app may include virtual scenes of one or more game levels, or one or more specific areas within a game level (such as a house, a room, or a garden).

[0079] Basic rendering and advanced rendering are relative concepts. Basic rendering requires relatively low GPU computing power, including rendering tasks that can be completed independently using the GPU computing power of mobile device 100. The game scene generated by basic rendering ensures the normal operation of the game app and normal user interaction. When the communication network between mobile device 100 and server 200 is poor or disconnected, and the preprocessed data of advanced rendering fed back from the cloud is not received, the game app on mobile device 100 can directly display the game screen after basic rendering to ensure normal user operation and viewing of the game app.

[0080] In this embodiment, the rendered image generated after basic rendering can be referred to as the basic rendered image. The basic rendering data generated by basic rendering can be intermediate data in the process of generating the basic rendered image (e.g., the irradiance of each pixel obtained through basic rendering), or it can be the image data of the basic rendered image. The GPU computing power required by the rendering technology used for basic rendering is provided by the mobile device 100, and this embodiment does not specifically limit the rendering technology. In some embodiments, rasterization technology is used to perform basic rendering of the scene to be rendered.

[0081] High-level rendering requires relatively high GPU computing power, including rendering tasks that rely on cloud-based GPU computing power. High-level rendering tasks are typically decoupled from basic rendering tasks. Preprocessing data for specific high-level rendering types is used to obtain the lighting and shadow effects of that specific high-level rendering type in the image to be rendered. The preprocessing data for different high-level rendering types can be different, and correspondingly, the fused basic rendering data can also be different. For example, if the specific high-level rendering type mentioned above is GI rendering, the preprocessing data for GI includes the irradiance of each pixel after GI rendering, and the basic rendering data includes the irradiance of each pixel after basic rendering; if the specific high-level rendering type mentioned above is reflection rendering, the preprocessing data for reflection rendering includes the cube map captured by the reflection probe, and the basic rendering data includes the color of each pixel after basic rendering.

[0082] In some embodiments, high-order rendering is a rendering task related to lighting characteristics, such as rendering of lighting characteristics such as global illumination (GI), ambient occlusion (AO), soft shadows, reflection, refraction, and caustics. It may also include rendering of other characteristics that are computationally intensive on the cloud side and can be integrated with the basic rendering on the edge side.

[0083] Reflection refers to the phenomenon where light, when traveling to different substances, changes its direction of propagation at the interface and returns to its original substance. Global Illumination (GI) is a lighting characteristic formed by both direct and indirect lighting, with GI more closely resembling realistic lighting in reality. Direct lighting is the phenomenon where light emitted from a light source directly illuminates objects in a scene, while indirect lighting is the phenomenon where light emitted from a light source is reflected from the surfaces of objects in the scene. For example, Figure 3A and Figure 3B This demonstrates the rendering effects of the same game screen displayed by the game app with and without graphics (GI); for example... Figure 3A As shown, the game can still run and display normally without GI, but the visuals are dim and lack indirect lighting details, such as the light and shadow details created by reflections from walls onto objects; for example,... Figure 3B As shown, with GI (Graphics Interaction) enabled, the image quality is brighter and more realistic. For example, Figure 3C and Figure 3D This demonstrates the rendering effects of the same game screen with and without reflections; for example... Figure 3C As shown, the game screen can still run and display normally without reflection, but the visual effects of reflection are missing; for example, mirrors in the screen do not display their projected content. Figure 3D As shown, when there is reflection, the mirror in the picture displays the content of the mirror projection.

[0084] In some embodiments, the basic rendering tasks of a game app include a minimum set of rendering tasks that ensure the normal operation of the game screen; the higher-order rendering tasks of a game app include rendering tasks other than the basic rendering tasks mentioned above. In some embodiments, the higher-order rendering tasks other than the minimum set mentioned above may also include rendering tasks that can be implemented solely by the computing power of the edge GPU. Shifting these rendering tasks to the cloud can reduce the computing power requirements and load on the edge GPU.

[0085] In some embodiments, the game app can determine the current high-order rendering type suitable for end-to-end cloud collaborative rendering based on real-time network conditions. In this embodiment, the server 200 needs to feed back pre-processed data for high-order rendering to the mobile device 100. The data type and data volume of the pre-processed data for different high-order rendering types can be different. For example, compared to reflection, the data volume of GI pre-processed data is lower. Under normal network conditions (e.g., round-trip time 50-100ms, transmission bandwidth 10Mbps), fewer high-order rendering types are suitable for end-to-end cloud collaborative rendering, provided that the maximum transmission latency of the pre-processed data is guaranteed to meet the user's normal operation. Under very good network conditions (e.g., RTT 10-20ms, bandwidth 100Mbps), more high-order rendering types are suitable for end-to-end cloud collaborative rendering, provided that the maximum latency of the user's operation is guaranteed.

[0086] In this embodiment, developers can determine the basic rendering tasks and high-order rendering tasks of an application (e.g., a game app) based on actual application needs and the GPU capabilities of the mobile device; no specific limitations are made here.

[0087] Regarding the aforementioned edge-cloud collaborative rendering method, this application provides two specific implementation schemes. Among them,

[0088] In Implementation Option 1, the client-side 3D engine independently downloads the client-cloud collaboration plugin SDK, which is used to implement the aforementioned client-cloud collaboration rendering method. In Implementation Option 2, the native client-side 3D engine already supports the aforementioned client-cloud collaboration rendering method. The two implementation options are described in detail below.

[0089] For example, Figure 4 A data flow diagram of a mobile device 100 involved in implementing scheme one is shown. For example... Figure 4 As shown, the mobile device 100 includes an input / output (IO) layer, a logic layer, a 3D engine layer, an operating system (OS) layer, and a hardware layer. The IO layer obtains user input from the game app's operation quality processing module in the OS layer and sends it to the logic layer. The logic layer runs the game app's logic code (i.e., the aforementioned logic module) and updates the game state based on user input. The logic layer instructs the 3D engine layer to render the game screen in the latest game state. The 3D engine layer calls the GPU through the OS layer to render the game screen.

[0090] For example, Figure 5 A schematic diagram of the system architecture of an edge-cloud collaborative rendering system involved in implementing Scheme 1 is shown. For example... Figure 5As shown, the rendering system includes a mobile device 100 and a server 200. The game app on the mobile device 100 mainly includes the following modules: logic code, a client-side 3D engine, and a client-cloud collaborative plugin SDK; the server 200 mainly includes the following modules: advanced rendering service, a client-cloud collaborative framework, and a cloud-side 3D engine.

[0091] Logic code: This includes the logic code that implements the game logic of the game app.

[0092] The device-side 3D engine comprises key modules such as the engine framework, scene management, and rendering pipeline. The engine framework provides the device-side game app with capabilities for game logic parsing, model parsing, and animation parsing. Scene management manages the scene information of one or more game scenes (i.e., device-side scenes) built by the game app. The rendering pipeline is the process of converting 3D scene models into screen pixel space output, including the following functions: converting the 3D coordinates of objects into 2D coordinates in screen pixel space; and coloring each pixel on the screen.

[0093] The edge-side 3D engine operates in two modes: development and runtime. In development mode, it provides developers with an Integrated Development Environment (IDE), allowing them to develop the logic code for the game app and render 3D scenes. After development, the relevant files can be packaged into an application package (APK) for the game app, which can run on mobile devices. In runtime mode, it provides a runtime environment for the game app. This runtime environment typically has cross-platform capabilities, supporting the game app's operation on mobile devices running different operating systems such as HarmonyOS, Android, and iOS. In this embodiment, during runtime, the game app can call the aforementioned rendering pipeline to perform basic rendering of the edge-side scene to be rendered, generating basic rendering data. It can also call the edge-cloud collaborative plugin SDK to request server 200 for high-level rendering preprocessing. Furthermore, it can fuse the basic rendering data and the high-level rendering preprocessed data, obtaining a displayable high-level rendered image based on the fused data.

[0094] Edge-Cloud Collaboration Plugin SDK: This SDK encapsulates the key capabilities of edge-cloud collaborative rendering and, based on the plugin mechanism provided by the edge 3D engine, enables third-party edge 3D engines to possess edge-cloud collaborative rendering capabilities. The Edge-Cloud Collaboration Plugin SDK includes some or all of the following sub-functions: scene transformation, state synchronization, transmission communication, decoding, pipeline adaptation, and authentication. In this embodiment, edge-cloud collaborative rendering is achieved by providing an Edge-Cloud Collaboration Plugin SDK to third-party 3D engines, avoiding intrusive modifications to the third-party 3D engines.

[0095] Scene Conversion: This function converts the scene to be rendered (i.e., the edge scene) that the edge 3D engine needs to perform edge-cloud collaborative rendering into the scene required for high-level rendering preprocessing on the cloud side (i.e., the cloud scene). The scene information of the cloud scene may or may not be the same as that of the edge scene.

[0096] End-side state synchronization: This is used to synchronize state data (hereinafter referred to as state synchronization data) that affects the cloud-side preprocessing and preprocessing result distribution strategy to the cloud side. For example, state synchronization data includes some or all of the following: scene identification information of the scene to be rendered (e.g., game identifier (ID), level ID, scene ID, instance ID, etc.), light source information (e.g., type, position, pose, brightness, quantity, etc. of light sources), character information (e.g., position, pose, animation, skills, quantity, etc. of virtual characters), camera information (e.g., camera position, pose, field of view (FOV), etc.), and scene update information (e.g., information related to object movement, object animation, destruction, etc.). The camera's field of view (i.e., the game player's perspective) determines the final image content after 3D scene rendering and also affects the contribution of light sources in the scene to the imaging.

[0097] End-side transmission communication: used to upload end-side state synchronization data to the cloud side, and to receive high-level rendering preprocessing data sent from the cloud side.

[0098] Decoding on the client side: This is used to decode the compressed and encoded preprocessed data sent from the cloud side into preprocessed data that can be consumed by the rendering pipeline on the client side.

[0099] End-side pipeline adaptation: Based on the extension mechanism of the end-side 3D engine rendering pipeline, it adds the ability to consume and render decoded preprocessed data on the basis of the original end-side rendering pipeline, that is, the ability to fuse basic rendering data and preprocessed data to obtain high-level rendering effects.

[0100] End-side authentication and authorization: Responsible for authentication and authorization between the end-to-cloud collaborative plugin SDK on the end side and the advanced rendering service on the cloud side.

[0101] Cloud-based 3D engine: Includes major functional modules such as scene management, rendering pipeline and engine framework, providing a capability foundation for edge-cloud collaborative framework and high-level rendering services.

[0102] End-to-cloud collaboration framework: includes some or all of the following sub-functions: state synchronization, transmission communication, decoding, session management and authentication.

[0103] Cloud-side state synchronization: Used to receive state synchronization data of the end-side scene (e.g., end-side scene 1) sent by the end-side, and update the scene information of the corresponding cloud-side scene (e.g., cloud-side scene 1 corresponding to end-side scene 1) in the scene management module of the cloud-side 3D engine.

[0104] Cloud-side encoding: Used to compress and encode preprocessed data for high-level rendering services to reduce the amount of data transmitted.

[0105] Cloud-side transmission communication: used to receive status synchronization data transmitted from the end side, and to send compressed and encoded preprocessed data to the end side.

[0106] Session Management: Used to assign clients accessing the client side to the corresponding high-level rendering service. For example, assignment can be based on information such as the game ID, level / scene ID, and instance ID uploaded by the client side. In other words, server 200 can establish sessions with multiple client sides, and establish corresponding high-level rendering services for each session, thus providing high-level rendering services to multiple clients simultaneously.

[0107] Cloud-side authentication and authorization: Used for authentication and authorization between the advanced rendering service on the cloud side and the end-to-cloud collaborative plugin SDK on the client side.

[0108] Advanced rendering services: Based on the cloud-side 3D engine and the edge-cloud collaborative framework, we provide advanced rendering services, namely advanced rendering preprocessing, such as GI preprocessing, AO preprocessing, reflection preprocessing, soft shadow preprocessing, and other advanced rendering services.

[0109] In some embodiments, to save cloud computing power and facilitate development and debugging, during the development phase of a game app, Figure 5 The rendering system shown can be used to simulate and test the modules of both the mobile device 100 and the server 200 on the same device. That is, the cloud-based 3D engine and the client-side 3D engine in development mode can also be set up on the same device, which can be a mobile device or a cloud-based server; no specific limitation is made here. Preferably, the device can be a mobile device.

[0110] Based on the foregoing embodiments, this application provides a terminal-cloud collaborative rendering method. The method is applied to a terminal-cloud collaborative rendering system, which includes a game APP on a mobile device 100. The game APP includes a terminal-side 3D engine and a terminal-cloud collaborative plugin SDK. Figure 6 A flowchart of a cloud-edge collaborative rendering method is shown during the development phase. The method includes some or all of steps S101 to S125.

[0111] S101, The IDE of the edge 3D engine determines the edge scene 1 and high-level rendering type 1 for edge-cloud collaborative rendering.

[0112] In this embodiment, during the APP development phase, the development state of the edge 3D engine provides an IDE for developers. Developers select the edge scene (e.g., edge scene 1) to be rendered using edge-cloud collaborative rendering and the high-order rendering type for that edge scene within the IDE. For game APPs, the edge scene can be one or more levels in a game, or a portion of a space within a level (e.g., the interior of a building). The high-order rendering type refers to the rendering type corresponding to the high-order rendering task described in the preceding embodiments. Exemplarily, high-order rendering types include GI, AO, soft shadows, reflection, refraction, and caustics. This embodiment may also include other high-order rendering types, which are not specifically limited here. Subsequent embodiments mainly use GI and reflection as examples for illustrative purposes.

[0113] S102, The edge 3D engine sends the scene information of edge scene 1 and high-level rendering type 1 to the edge-cloud collaborative plugin SDK.

[0114] In some embodiments, scene information for an edge scene includes descriptions of 3D object models (e.g., buildings, characters, or other objects) in the scene (e.g., the object model's name, geometry, texture, material, roughness, and transparency), and may also include descriptions of the background environment, light sources, scene identifiers, character information, camera information, etc. The geometry of the 3D object model can be represented in various ways; for example, it can be represented using a triangular mesh, where the surface of the 3D object is approximated by multiple meshes. The mesh information of a mesh can include useful attributes and functions such as vertex coordinates, normals, texture coordinates, and triangle rendering sequences.

[0115] In this embodiment, some high-order rendering types can obtain lighting information of various points in the scene by deploying probes in the scene. In one implementation, the edge device can deploy probes for implementing specific high-order rendering types (e.g., GI) in the 3D scene according to preset deployment rules (e.g., uniformly distributed at a preset density within a preset spatial range of the 3D scene). Each probe can capture, store, and update the lighting information received at its location. When it is necessary to shade a shading point (e.g., shading point 1) in the 3D scene, the probes of neighboring shading point 1 can be queried, and shading point 1 can be shaded according to the lighting information of the locations of the neighboring probes. Then, the pixels corresponding to shading point 1 in the image can be rendered.

[0116] In some embodiments, the scene information of the edge scene 1 may further include information related to probes required for high-order rendering deployed in the scene, such as probe deployment rules, deployment spatial range, number, historical fusion coefficients, and deviation values. This application embodiment does not specifically limit the information related to the probes. In this application embodiment, probes deployed to implement GI can be simply referred to as GI probes, and probes deployed to implement reflection can be simply referred to as reflection probes.

[0117] S103, the End-to-Cloud Collaboration Plugin SDK converts the scene information of the end-side scene 1 into the scene information of the cloud-side scene 1 according to the preset conversion rules based on the high-order rendering type 1.

[0118] In this embodiment of the application, the information content of the scene information of the terminal scene 1 and the cloud scene 1 may be the same or different.

[0119] In some embodiments, after the cloud collaboration plugin SDK obtains the scene information of the terminal scene 1, it can simplify the information content of the scene information of the terminal scene 1 according to the preset conversion rules, retaining only the scene information required for preprocessing of high-order rendering type 1. The simplified scene information indicates the cloud scene 1 corresponding to the terminal scene 1.

[0120] For example, in one implementation, if the developer chooses high-order rendering types such as GI, AO, and soft shadows, the converted cloud-side scene 1 only needs to retain the surface attribute information (e.g., mesh information and color information) of each 3D object model in the endpoint scene 1; even compared to the precision of the mesh information of each model in the endpoint scene 1, the precision of the mesh information in the cloud-side scene 1 can be lower. Reducing the precision of the mesh information helps to reduce the complexity of subsequent high-order rendering preprocessing. If high-order rendering types such as reflection and refraction are chosen, the converted cloud-side scene 1 and endpoint scene 1 need to maintain a higher similarity, for example, the information content of the scene information in the cloud-side scene 1 and endpoint scene 1 should be consistent.

[0121] In some embodiments, the data formats of the scene information in edge-side scenario 1 and cloud-side scenario 1 may or may not be the same. The data format of the scene information indicates which types of information the scene information includes, the representation methods of each type of information, and the arrangement of each type of information in the scene information.

[0122] In some embodiments, the cloud-side 3D engine and the edge-side 3D engine can use the same 3D engine (i.e., the same 3D engine from the same manufacturer and the same version), or they can use different 3D engines. The data formats of scene information managed by different 3D engines are usually different. When the cloud-side 3D engine and the edge-side 3D engine are different 3D engines, it is necessary to convert the scene information of the edge-side scene 1 into the scene information of the cloud-side scene 1 based on the data format of the scene information corresponding to the high-level rendering type 1 managed by the cloud-side 3D engine. This conversion process may also involve the simplification of the above-mentioned information content.

[0123] In some embodiments, when the cloud-side 3D engine and the edge-side 3D engine are the same 3D engine, the data format of the scene information of edge-side scene 1 and cloud-side scene 1 is the same, and the scene information of edge-side scene 1 and cloud-side scene 1 can remain consistent without conversion.

[0124] In this embodiment, developers can manually convert edge scenario 1 to cloud scenario 1 using the edge-cloud collaboration plugin SDK according to preset conversion rules; alternatively, they can use the developed automatic conversion tool to automatically convert edge scenario 1 to cloud scenario 1 according to preset conversion rules.

[0125] Step S103 is optional. In some embodiments, S103 does not need to be executed, the edge scenario and the cloud scenario remain completely consistent, and the cloud scenario 1 involved in the following embodiments is equivalent to the edge scenario 1.

[0126] It is understood that in the embodiments of this application, a terminal scene is converted into a cloud scene for each higher-order rendering type. For example, for higher-order rendering type 1 (e.g., GI), terminal scene 1 can be converted into cloud scene 1; for higher-order rendering type 2 (e.g., reflection), terminal scene 1 can be converted into cloud scene 2.

[0127] S104. The End-to-Cloud Collaboration Plugin SDK sends the scene information of Cloud Scene 1 and High-Level Rendering Type 1 to the IDE of the Cloud 3D Engine.

[0128] S105, the IDE of the cloud-side 3D engine adjusts the scene information of cloud-side scene 1.

[0129] In some embodiments, during the conversion of edge-side scene 1 to cloud-side scene 1, conversion errors may occur, resulting in the scene conversion not strictly adhering to the aforementioned preset conversion rules. For example, human error may occur during manual scene conversion by the developer, or a bug may occur during automatic scene conversion by the automatic conversion tool, leading to conversion errors. After importing the scene information of cloud-side scene 1 into the IDE of the cloud-side 3D engine, developers can check and adjust the scene information of cloud-side scene 1 through the IDE. For example, the adjustments include: repairing the material, geometry, size, or position of the problematic 3D object model, and correcting the spatial range and number of GI probes, etc.

[0130] Step S105 is optional. In some embodiments, it is not necessary to perform S105.

[0131] S106, the IDE of the edge 3D engine performs basic rendering on edge scene 1 based on the state data 1 of edge scene 1, and obtains basic rendering data 1.

[0132] The state data of a 3D scene includes adjustable data from the runtime scene information, indicating the real-time state of the 3D scene. During development, the IDE of the edge 3D engine can simulate and generate various state data of edge scene 1 (e.g., state data 1) to debug the rendering effect of edge scene 1 during development.

[0133] In some embodiments, the IDE of the edge 3D engine updates the scene information of the edge scene 1 based on state data 1; the IDE of the edge 3D engine performs basic rendering on the updated edge scene 1 using rasterization technology to obtain basic rendering data 1. In this embodiment, when the network is poor, the game screen after basic rendering can be obtained based on the basic rendering data 1. This game screen can ensure the normal operation of the game APP and the normal interaction of user operations. For example, see Figure 7In one implementation, the edge rendering pipeline includes a base rendering channel and a conversion channel. The base rendering channel uses rasterization technology to render the edge scene 1, and the output base rendering data 1 includes the irradiance of each pixel in the image of the edge scene 1. The conversion channel determines the pixel value (e.g., RGB color information) of each pixel in the image based on the above base rendering data 1, and outputs the base rendering image as the rendering image to be displayed.

[0134] S107, The IDE of the edge 3D engine sends the status synchronization data 1 of the edge scene 1 to the edge cloud collaboration plugin SDK.

[0135] S108, The End-to-Cloud Collaboration Plugin SDK sends the status synchronization data 1 of the End-to-Cloud Collaboration Framework of the server 200 to the End-to-Cloud Collaboration Framework of the End-to-End Scene 1.

[0136] The S109 and server 200 end-to-cloud collaborative framework sends the state synchronization data 1 of the end-side scene 1 to the IDE of the cloud-side 3D engine.

[0137] In some embodiments, state synchronization data 1 includes state data required to implement the higher-order rendering type 1 of the cloud-side scene 1. State synchronization data 1 may include some or all of the data in state data 1. The edge-cloud collaboration plugin SDK synchronizes state synchronization data 1 to the cloud-side 3D engine IDE, and the cloud-side 3D engine IDE can update the cloud-side scene 1 to the latest state based on state synchronization data 1.

[0138] The S110 cloud-side 3D engine's IDE updates the cloud-side scene 1 based on the status synchronization data 1, performs high-level rendering type 1 preprocessing on the updated cloud-side scene 1, and obtains high-level rendering type 1 preprocessed data 1.

[0139] Higher-order rendering type 1 can be rendering of any of the lighting characteristics such as GI, AO, soft shadows, reflection, refraction, and caustics. This application does not specifically limit the rendering algorithms for lighting characteristics such as GI, AO, soft shadows, reflection, refraction, and caustics. For example, rendering algorithms implementing GI include, but are not limited to: ray tracing, path tracing, dynamic diffuse global illuminating (DDGI) spherical harmonic lighting, voxel-based global illuminating, and point-based global illuminating, etc.

[0140] In this embodiment of the application, the IDE of the cloud-side 3D engine can synchronize data 1 according to the state of the terminal scene 1, update the scene information of the cloud-side scene 1 corresponding to the terminal scene 1, and use the rendering algorithm 1 to perform high-order rendering type 1 (e.g., GI) preprocessing on the updated cloud-side scene 1 to obtain preprocessed data; the preprocessed data can indicate the lighting and shadow effects of high-order rendering type 1 in the cloud-side scene 1.

[0141] For example, taking GI as the high-order rendering type 1 and DDGI as the rendering algorithm 1, the preprocessing of high-order rendering type 1 is explained in an exemplary manner. The DDGI algorithm uses GI probes to store the lighting information of the scene and uses ray tracing to dynamically update it, thereby achieving a real-time dynamic diffuse global illumination effect. The DDGI algorithm packages a group of probes into a DDGI Volume (i.e., a cubic region in 3D space). Simply drag the Volume into the scene to be rendered, and the Volume will automatically place probes within it; the shading points within the Volume will automatically capture lighting information through the surrounding probes.

[0142] In some embodiments, the scenario information for cloud-side scenario 1 includes information related to the GI probe (e.g., deployment rules, deployment location, and data); see also Figure 7 Based on the scene information of cloud-side scene 1, the IDE of the cloud-side 3D engine utilizes the DDGI rendering channel of the cloud-side rendering pipeline to deploy and run GI probes in cloud-side scene 1. The GI probes acquire and store the lighting information at their location, generating Probe layer data. The Probe layer data includes the lighting information of each GI probe, which includes irradiance. The shading points corresponding to each pixel in the image are determined in cloud-side scene 1 based on the camera's viewpoint. For any shading point within a volume in cloud-side scene 1 (e.g., shading point 1), the DDGI rendering pipeline acquires the lighting information of eight GI probes surrounding shading point 1. Based on the lighting information of these eight GI probes, the irradiance is interpolated to obtain the irradiance of shading point 1. The preprocessed GI data includes the irradiance of the shading points corresponding to each pixel in the image.

[0143] The probe deployed by DDGI stores spherical information. The DDGI algorithm encodes the spherical data into a two-dimensional texture map through octahedral mapping. The smallest unit of the texture map is a texel, and one texel corresponds to one or more pixels. The lighting information of the GI probe can include the irradiance received from the hemisphere in the direction of the texel (w), the distance r(w) between the probe and the nearest object seen from the texel direction, and the square of the distance r. 2 (w). Here, irradiance is encoded as a 3D vector texture, r(w) and r... 2(w) are encoded together as a two-dimensional vector texture (x component stores r(w), y component stores r). 2 (w)).

[0144] In one implementation, probe 1 is any one of the eight GI probes mentioned above. The process of interpolating the irradiance based on the illumination information of the eight GI probes to obtain the irradiance of the shading point 1 includes: obtaining three weighting coefficients for probe 1, namely the trilinear interpolation coefficient, the orientation coefficient, and the Chebyshev coefficient; using the normalized value of the product of these three coefficients as the weight of probe 1; and weighting the irradiance of the eight GI probes based on the weight of each probe to obtain the irradiance of the shading point 1. Specifically, the trilinear interpolation coefficient indicates the distance between probe 1 and shading point 1; if this coefficient is large, the weight of probe 1 is reduced. The orientation coefficient indicates the angle between the direction from shading point 1 to probe 1 and the surface normal of shading point 1; if this coefficient is too large, the weight of probe 1 is reduced. The Chebyshev coefficient indicates the probability of an obstruction between probe 1 and shading point 1; if this coefficient is large, the weight of probe 1 is reduced. The Chebyshev coefficient is calculated based on the distance r(w) and the squared distance r. 2 (w) is certain.

[0145] In some embodiments, in addition to the illumination information of probe 1, the Probe layer data also includes the classification information and relocation information of probe 1; the classification information is used to indicate whether the state of probe 1 is a valid probe, and then to determine whether to use the illumination information of probe 1 based on the state of probe 1; the relocation information is used to indicate whether and how to adjust the position of probe 1.

[0146] For example, taking high-order rendering type 1 as reflection and rendering algorithm 1 as a reflection capture algorithm based on reflection probes as an example, the preprocessing of high-order rendering type 1 is explained in an exemplary manner. The cloud-side 3D engine's IDE uses the cloud-side rendering pipeline to deploy and run reflection probes in cloud-side scene 1, and obtains the lighting information of the reflection probes. The lighting information of the reflection probes includes the cube map of the location of the reflection probe. This cube map is synthesized from the reflection texture maps obtained by the reflection probe along the six directions of front, back, left, right, up, and down of the camera. The cube map can be used to implement environment mapping, which can simulate the environment around the probe. The preprocessed data of reflection includes the cube maps of all reflection probes in cloud-side scene 1, or one or more cube maps belonging to the location of the camera uploaded from the edge, or the cube map after merging the one or more cube maps, or the cube map of the reflection probe closest to the camera.

[0147] S111, The cloud-side 3D engine IDE sends preprocessed data 1 of high-level rendering type 1 to the end-cloud collaborative plugin SDK.

[0148] In some embodiments, in step S110, the IDE of the cloud-side 3D engine performs preprocessing of the high-order rendering type 1, and the acquired preprocessed data includes preprocessed data for all view ranges of the camera. The IDE of the cloud-side 3D engine clips the preprocessed data according to the view range of the camera in the state synchronization data 1, and obtains preprocessed data 1 within the view range of the camera, that is, preprocessed data related to the end-side user of the mobile device 100, and sends preprocessed data 1 to the end-cloud collaboration plugin SDK.

[0149] S112, The End-to-Cloud Collaboration Plugin SDK sends preprocessed data 1 of high-level rendering type 1 to the IDE of the edge 3D engine.

[0150] In some embodiments, the cloud-side 3D engine IDE uses a preset encoding algorithm to compress and encode preprocessed data 1; and sends the encoded preprocessed data to the edge-cloud collaboration plugin. The edge-cloud collaboration plugin SDK uses the decoding algorithm corresponding to the preset encoding algorithm to decode the received preprocessed data, obtain the decoded preprocessed data 1, and send it to the edge-side 3D engine IDE. This application embodiment does not specifically limit the aforementioned preset encoding algorithm.

[0151] S113, after the IDE of the edge 3D engine merges the basic rendering data 1 and the preprocessed data 1 of the high-order rendering type 1, it obtains an image 1 with the rendering effect of the high-order rendering type 1.

[0152] It is understandable that, since the scene information of the edge scene 1 includes the scene information of the cloud scene 1, and the edge scene 1 and the cloud scene 1 are synchronized for the same state data (such as camera information), the pixels in the image image indicated by the basic rendering data 1 generated by the basic rendering and the pixels in the image image indicated by the preprocessing data 1 of the high-level rendering can correspond one-to-one.

[0153] In some embodiments, the higher-order rendering type 1 is GI, the rendering algorithm is DDGI, the basic rendering data 1 includes the irradiance of each pixel in the image rendered by rasterization, and the preprocessing data 1 includes the irradiance of each pixel in the image rendered by DDGI; see also Figure 7The edge-cloud collaboration plugin SDK module adds a fusion channel to the edge-side rendering pipeline to consume preprocessed data, based on the expansion mechanism of the edge-side 3D engine rendering pipeline. The basic rendering channel outputs basic rendering data 1 to the fusion channel. The edge-side 3D engine IDE inputs the preprocessed data 1 received from the cloud-side DDGI rendering channel into the edge-side DDGI rendering channel, which outputs preprocessed data 1 to the fusion channel. The fusion channel outputs the fused rendering data, which includes the fused irradiance and the irradiance of the pixel at coordinate 1, equal to the product of the irradiance of the pixel at coordinate 1 in the basic rendering data 1 and the irradiance of the pixel at coordinate 1 in the preprocessed data 1. The fused rendering data is then input into the conversion channel, which determines the pixel value of each pixel based on the fused rendering data, thereby outputting the rendered image (i.e., image 1).

[0154] See Figure 7 When the network is poor, and the DDGI rendering channel on the device side does not receive the preprocessed data 1 from the cloud, the basic rendering data 1 is input into the blending channel, and then the blending channel outputs the basic rendering data 1 to the conversion channel. The conversion channel determines the pixel values ​​of each pixel based on the basic rendering data 1, thus outputting the basic rendered image as the rendered screen to be displayed. The basic rendered image does not have the lighting and shadow effects of higher-order rendering type 1. The game app can display the above-mentioned basic rendered image to ensure the normal operation of the game app.

[0155] In some embodiments, the higher-order rendering type 1 is reflection, the rendering algorithm is a reflection probe-based reflection capture algorithm, and the basic rendering data (and basic rendering image) includes pixel information (e.g., color) of each pixel in the image rendered by rasterization. The preprocessed data 1 of reflection includes the cubemap of the reflection probe in the cloud-side scene 1; the pixel information (e.g., color) of each pixel in the image rendered after reflection is determined according to the cubemap of the reflection probe; the pixel information of pixels at the same coordinate in the image rendered by the basic rendering and the image rendered by reflection are fused as the pixel information of pixels at the same coordinate in image 1.

[0156] In some embodiments, determining the pixel information (e.g., color) of each pixel in the image after reflection rendering based on the cubemap of the reflection probe includes: the preprocessed data 1 of the reflection includes the cubemap of all reflection probes in the cloud-side scene 1; determining one or more cubemaps to which the camera position belongs in the cubemaps of all the above reflection probes based on the camera position; determining the cubemap 1 after the above one or more cubemaps are fused; determining the intersection point P of the reflection vector R after the light emitted from the camera is reflected by the ground and the bounding sphere that encloses the cubemap 1; taking the vector CP from the center point C of the cubemap 1 to the intersection point P as the new reflection vector R'; and sampling the cubemap 1 using the above reflection vector R', that is, determining the pixel information (e.g., color) of the corresponding pixel in the image based on the texture of the intersection point of the reflection vector R' and the cubemap 1.

[0157] The preprocessed data 1 for reflection may also include one or more cubemaps representing the location of the camera uploaded from the end-side, or a cubemap fused from the one or more cubemaps, or a cubemap of the nearest reflection probe of the camera. The specific calculation of pixel information for each pixel in the image after reflection rendering can be described in the above embodiments and will not be repeated here.

[0158] S114. Based on the high-level rendering effect presented by image 1, the developer adjusts the preprocessing-related parameters 1 in the scene information of cloud scene 1 through the IDE of the cloud-side 3D engine.

[0159] In this embodiment of the application, when the high-order rendering effect presented by image 1 is unsatisfactory, the preprocessing-related parameters of the cloud-side scene (e.g., parameter 1) can be adjusted on the cloud side. For example, if the developer observes that the rendering effect of virtual object 1 in image 1 does not meet expectations, the material of the virtual object can be adjusted; if the developer observes that the lighting and shadow effects of GI in image 1 are not realistic enough, the position information of the GI probe, the parameter information of GI calculation, etc. can be adjusted.

[0160] S115, the cloud-side 3D engine's IDE performs high-level rendering type 1 preprocessing based on the cloud-side scene 1 after adjusting parameter 1, and obtains the updated preprocessed data 2.

[0161] S116, The cloud-side 3D engine's IDE sends the adjusted parameter 1 and updated preprocessed data 2 to the end-cloud collaborative plugin.

[0162] S117. The edge-cloud collaboration plugin sends the adjusted parameter 1 and the updated preprocessed data 2 to the IDE of the edge 3D engine.

[0163] The adjusted parameter 1 and the updated preprocessed data 2 can be sent simultaneously or separately.

[0164] S118, the IDE of the edge 3D engine updates parameter 1 in the scene information of edge scene 1, so that parameter 1 in the scene information of edge scene 1 and cloud scene 1 is synchronized; based on the edge scene 1 after updating parameter 1, basic rendering is performed to obtain basic rendering data 2.

[0165] S119, the IDE of the edge 3D engine merges the basic rendering data 2 and the preprocessed data 2 to obtain an image 2 with the rendering effect of high-level rendering type 1.

[0166] Based on the rendering effect of image 2, the S120 edge 3D engine IDE adjusts parameter 2 of edge scene 1 and / or the processing logic for consuming preprocessed data.

[0167] In this embodiment of the application, if the high-order rendering effect of image 2 is not satisfactory, the relevant parameters of the scene on the edge can be adjusted (e.g., parameter 2). If the rendering latency of image 2 is large, the processing logic of consuming preprocessed data can be adjusted, such as adjusting the encoding / decoding algorithm of preprocessed data, so as to improve the encoding and decoding rate of preprocessed data and reduce the rendering latency of edge-cloud collaborative rendering.

[0168] S121, The IDE of the edge 3D engine performs basic rendering based on the edge scene 1 after adjusting parameter 2, and obtains basic rendering data 3.

[0169] S122, the rendering pipeline of the edge 3D engine's IDE merges the basic rendering data 3 and the preprocessed data 2 of the higher-order rendering type 1, and then obtains an image 3 with the rendering effect of the higher-order rendering type 1.

[0170] In some embodiments, if the scene information of cloud-side scene 1 does not include parameter 2, step S122 is executed after step S121, that is, the preprocessed data 2 is fused with the basic rendering data 3 to obtain the rendered image to be displayed. If the scene information of cloud-side scene 1 includes parameter 2, after adjusting parameter 2 of terminal scene 1, parameter 2 of cloud-side scene 1 is also adjusted synchronously, and the preprocessed data of high-order rendering is updated to preprocessed data 3 according to the adjusted cloud-side scene 1. The basic rendering data 3 is fused with the preprocessed data 3 to obtain the rendered image to be displayed. In one implementation, steps S121 are further followed by A1 to A7.

[0171] A1. The IDE of the edge 3D engine sends the adjusted parameter 2 to the edge-cloud collaboration plugin SDK, so that parameter 2 in the scene information of edge scene 1 and cloud scene 1 remains synchronized.

[0172] A2. The End-to-Cloud Collaboration Plugin SDK sends the adjusted parameter 2 to the End-to-Cloud Collaboration Framework of server 200.

[0173] A3. The cloud-end collaboration framework of server 200 sends the adjusted parameter 2 to the IDE of the cloud-side 3D engine.

[0174] A4. The IDE of the cloud-side 3D engine updates parameter 2 in the scene information of the terminal scene 1, so that parameter 1 in the scene information of the terminal scene 1 and the cloud-side scene 1 are kept synchronized; the updated cloud-side scene 1 is preprocessed with high-order rendering type 1 to obtain the preprocessed data 3 of high-order rendering type 1.

[0175] A5. The cloud-side 3D engine's IDE sends preprocessed data of high-level rendering type 1 to the end-cloud collaborative plugin.

[0176] A6. The edge-cloud collaboration plugin sends preprocessed data 3 of high-level rendering type 1 to the IDE of the edge 3D engine.

[0177] A7. The rendering pipeline of the edge 3D engine's IDE merges the basic rendering data 3 and the preprocessed data 3 of the higher-order rendering type 1 to obtain an image 4 with the rendering effect of higher-order rendering type 1.

[0178] Some or all of the steps in S114 to S122 above are optional. In some embodiments, the developer only triggers parameter adjustment on the cloud side, that is, executes steps S114 to S119, without executing steps S120 to S122. In some embodiments, the developer only triggers parameter adjustment on the device side, that is, executes steps S120 to S122, without executing steps S114 to S119.

[0179] This application embodiment does not specifically limit the execution order of steps S114 to S119 and S120 to S122. In some embodiments, developers may first trigger parameter adjustment on the terminal side (i.e., steps S114 to S119) and then trigger parameter adjustment on the cloud side (S120 to S122).

[0180] In some embodiments, after step S122, if the rendering effect of the adjusted high-level rendering image still does not meet the developer's rendering requirements, then steps S114 to S119 and steps S120 to S122 are repeated, that is, the cloud side and the terminal side are iteratively adjusted until the rendering effect meets the developer's rendering requirements.

[0181] In some embodiments, in step S101, a high-order rendering type 1 (e.g., GI) and a terminal scene 1 are selected. For different state data of the terminal scene 1, steps S106 to S122 are executed; when the terminal scene 1 is selected, steps S101 to S122 are executed for different high-order rendering types; and steps S101 to S122 are executed for different terminal scenes. This allows for testing and adjustment of the dual-side scenes involved in the edge-cloud collaborative rendering scheme, ensuring that the edge-cloud collaborative rendering scheme provided in this application can achieve better high-order rendering effects for any high-order rendering type, any terminal scene, and any state of the terminal scene.

[0182] S123, the End-to-Cloud Collaboration Plugin SDK packages the adjusted end-side scenes and related files into a game app.

[0183] S124, the cloud-side 3D engine's IDE, packages the adjusted cloud-side scene and related files, and uploads them to the cloud-side advanced rendering service.

[0184] Referring to steps S101 to S122 above, after the developer has debugged the rendering effects of all game scenes (i.e., client-side scenes) of the game app, they package the project files of the adjusted client-side scenes into a client-side app. For example, they package the scene information, scene ID, and high-order rendering type 1 of the adjusted client-side scene 1 into the game app's application package. The developer then packages and uploads the adjusted cloud-side scenes to the cloud-side high-order rendering service. For example, they package the scene information of cloud-side scene 1, as well as the scene ID of cloud-side scene 1, the correspondence between the scene ID of client-side scene 1 and high-order rendering type 1, etc. The scene management module in the cloud-side high-order rendering service is responsible for managing the uploaded cloud-side scenes and records the correspondence between the scene IDs of various client-side scenes, various high-order rendering types, and the scene IDs of cloud-side scenes.

[0185] S125. The developer associates the authentication and authorization information of the cloud-side advanced rendering service with the game APP on the mobile device 100. The authentication and authorization information is used to verify the legitimacy of the mobile device 100.

[0186] In this embodiment of the application, no specific limitations are made on the authentication and authorization methods and authentication and authorization information between the cloud-side high-level rendering service and the client-side APP.

[0187] In some embodiments, the developer associates the authentication and authorization information of the cloud-side advanced rendering service with the game app on the mobile device 100. This includes: the developer registering the game app's advanced rendering service on the portal website and obtaining a key pair, namely, public key 1 and private key 1; storing public key 1 on the game app on the client side and storing private key 1 on the cloud-side end-to-cloud collaboration framework. The authentication and authorization information on the client side is public key 1, which is used to encrypt the session and verify digital signatures; the authentication and authorization information on the cloud side is the private key 1 corresponding to public key 1, which is required to decrypt data encrypted by public key 1.

[0188] Mobile game apps also include logic modules. Figure 8 A flowchart of a cloud-edge collaborative rendering method is shown during the runtime phase. The method includes some or all of steps S201 to S215.

[0189] S201, the logic module of the mobile device 100 sends a rendering request to the runtime of the edge 3D engine. The rendering request includes the scene ID of the edge scene 1 to be rendered and the high-order rendering type 1. The rendering request is used to request rendering based on the latest state of the edge scene 1 in order to obtain an image with the rendering effect of high-order rendering type 1.

[0190] In some embodiments, the game scene of the game app is refreshed according to a preset refresh rate. When the logic module detects that the game scene has been updated, it sends a rendering request to the runtime of the client-side 3D engine to render the updated game scene. The updated game scene is then displayed in the new refresh cycle. For example, when the user switches the game's perspective, the game app updates the game scene according to the switched perspective. The mobile device 100 can obtain the scene information of the client-side scene 1 corresponding to the current game scene, the scene ID of the client-side scene 1, and the higher-order rendering type 1 from the game app's application package.

[0191] S202, the edge 3D engine Rumtime performs basic rendering based on the current scene information of edge scene 1 and obtains basic rendering data 4.

[0192] For details, please refer to the relevant description of step S106 above, which will not be repeated here.

[0193] S203, the runtime of the edge 3D engine sends a preprocessing request for high-order rendering type 1 to the edge-cloud collaborative plugin SDK. The preprocessing request includes the scene ID of edge scene 1 and high-order rendering type 1. The preprocessing request is used to instruct the cloud scene corresponding to edge scene 1 and high-order rendering type 1 to perform preprocessing of high-order rendering type 1.

[0194] S204. The End-to-Cloud Collaboration Plugin SDK sends a preprocessing request to the End-to-Cloud Collaboration Framework.

[0195] S205. The edge-cloud collaboration framework performs authentication and authorization based on the received preprocessing request; after successful authentication and authorization, it sends the preprocessing request to the high-level rendering service.

[0196] In some embodiments, the runtime of the edge 3D engine sends a preprocessing request encrypted with public key 1 to the edge-cloud collaboration plugin SDK; the edge-cloud collaboration framework decrypts the received preprocessing request using private key 1; if decryption is successful, authentication is passed, and the decrypted preprocessing request is sent to the high-level rendering service. This application embodiment does not specifically limit the authentication method between the high-level rendering service of server 200 and the game APP of mobile device 100.

[0197] S206. The advanced rendering service performs preprocessing preparation, including loading cloud-side scene 1 and preparing the rendering pipeline for advanced rendering type 1. Cloud-side scene 1 is determined based on the scene ID of end-side scene 1 and advanced rendering type 1 in the preprocessing request.

[0198] In some embodiments, the high-order rendering service in step S124 packages the correspondence between the scene ID of the terminal scene, the high-order rendering type, and the scene ID of the cloud scene. For example, the correspondence between the scene ID of the terminal scene 1, the high-order rendering type 1, and the scene ID of the cloud scene 1, as well as the scene information of the cloud scene. Based on the above correspondence, it can be determined that the cloud scene 1 is the cloud scene transformed from the terminal scene 1 for the high-order rendering type 1, that is, the scene to be rendered.

[0199] In some embodiments, the preprocessing request may also include the game ID of the game to which the client-side scene 1 belongs. The advanced rendering service can record the scene IDs of the client-side scenes of each game according to the game ID, so that the advanced rendering service can quickly locate the client-side scene 1 and the cloud-side scene 1 corresponding to the client-side scene 1.

[0200] S207. The advanced rendering service sends a confirmation message to the end-to-end cloud collaboration plugin SDK. The confirmation message indicates that preprocessing preparation has been completed.

[0201] S208, The End-to-Cloud Collaboration Plugin SDK sends state synchronization data 2 related to the preprocessing of high-level rendering type 1 to the high-level rendering service.

[0202] Specifically, the description of state synchronization data 2 can be found in the aforementioned description of state synchronization data 1, and will not be repeated here.

[0203] S209. The advanced rendering service updates the cloud-side scene 1 according to the received status synchronization data 2, and performs advanced rendering type 1 preprocessing based on the scene information of the updated cloud-side scene 1 to obtain preprocessed data 4.

[0204] Specifically, please refer to the relevant description of step S110 above, which will not be repeated here. In the embodiments of this application, the high-order rendering service only performs high-order rendering preprocessing and does not run the full game APP; compared with the aforementioned method one, it reduces the consumption of cloud computing resources.

[0205] S210, The advanced rendering service sends preprocessed data related to the end-user to the end-cloud collaborative framework.

[0206] In some embodiments, in step S209, the high-level rendering service performs preprocessing of the high-level rendering type 1, which can obtain preprocessed data within all viewing angles of the camera. The high-level rendering service clips the preprocessed data based on the current camera viewing angle in the status synchronization data 2 uploaded by the mobile device 100, obtaining preprocessed data 4 within the current viewing angle of the camera, i.e., preprocessed data related to the end-user of the mobile device 100, and sends the preprocessed data 4 to the end-cloud collaboration framework in step S210. In some embodiments, the preprocessed data 4 generated in step S209 is the preprocessed data within the current viewing angle of the camera, and no clipping is required.

[0207] S211. The edge-cloud collaboration framework compresses and encodes the preprocessed data 4 to obtain encoded data 1.

[0208] S212, The end-to-end cloud collaboration framework sends encoded data 1 to the end-to-end cloud collaboration plugin SDK.

[0209] S213. The End-to-Cloud Collaboration Plugin SDK decodes the encoded data 1 to obtain the preprocessed data 4.

[0210] In some embodiments, the cloud-based 3D engine IDE uses a preset encoding algorithm to compress and encode the preprocessed data 4 to obtain encoded data 1; then, it encapsulates the encoded data 1 using a preset communication protocol and sends it to the end-cloud collaboration plugin SDK. The end-cloud collaboration plugin decapsulates the received data to obtain encoded data 1; the end-cloud collaboration plugin then uses the preset encoding algorithm to decode the received encoded data 1 to obtain the decoded preprocessed data 4. This application embodiment does not specifically limit the preset encoding algorithm and preset communication protocol mentioned above; for example, the preset encoding algorithm may be the HEVC algorithm, and the preset communication protocol may be the UDP protocol.

[0211] In this embodiment, the data transmitted from the cloud side to the terminal side is pre-processed data after compression encoding, which requires lower transmission bandwidth than the video stream data transmitted by the aforementioned method one.

[0212] S214. The edge-cloud collaboration plugin SDK sends preprocessed data to the runtime of the edge 3D engine.

[0213] S215, the runtime of the edge 3D engine merges the basic rendering data 4 and the preprocessed data 4 to obtain an image 4 with the rendering effect of high-order rendering type 1.

[0214] For details, please refer to the relevant description of step S113 above, which will not be repeated here.

[0215] In some embodiments, in step S201, the rendering request and preprocessing request may also include multiple higher-order rendering types. The server 200 can perform preprocessing on the cloud-side scene 1 according to each of the multiple higher-order rendering types in the preprocessing request, and feed back the preprocessed data of each higher-order rendering type to the mobile device 100. The mobile device 100 can fuse the basic rendering data and the preprocessed data of the multiple higher-order rendering types to generate an image with the lighting effects of the multiple higher-order rendering types.

[0216] In this embodiment, the client-side game app only needs to have the ability to perform basic rendering of 3D scenes. The game app achieves high-level rendering of 3D game scenes by calling the client-cloud collaborative plugin SDK. After the basic rendering data is fused with the preprocessed data of high-level rendering, the final rendered image presents the lighting and shadow effects of high-level rendering, such as the lighting and shadow effects that can be achieved by ray tracing. In this way, 3D applications (such as game apps) running on mobile devices with weak GPU capabilities have high-level rendering capabilities (such as ray tracing capabilities), effectively improving the user experience.

[0217] For example, the edge-side scene 1 to be rendered is a multiplayer game scene in a game app, where multiple players simultaneously participate in the same instance of the game scene, such as user 1 on mobile device 100 and user 2 on mobile device 300. If the lighting and shadow effects of the current high-order rendering type 1 (e.g., GI) are independent of the user's perspective, the preprocessed data of high-order rendering type 1 can be shared with other players in the game so that they can achieve the high-order rendering effects on the edge-side. It can be understood that for the latest state of edge-side scene 1, server 200 does not need to perform preprocessing for each user for the perspective-independent high-order rendering type; it only needs to perform preprocessing once and share it with other users. This further reduces the consumption of cloud-side computing resources in the edge-cloud collaborative rendering solution.

[0218] In some embodiments, the preprocessing request may further include the instance ID of the game instance currently running in scenario 1 on the device side. Based on the game ID, scenario ID, and instance ID, other players in the same instance as the user on the device 100 can be queried.

[0219] In some embodiments, the preprocessing of high-order rendering type 1 in the latest state of the edge scene 1 can be either view-independent or view-dependent. View-independent processing refers to processing that is independent of the current viewpoint of the edge user (i.e., the viewpoint of the aforementioned camera); for example, in the latest state of the game scene, even if user 1 and user 2 have different viewpoints in the game scene, the lighting and shadow effects related to high-order rendering type 1 seen by user 1 and user 2 are the same, therefore the preprocessing of high-order rendering type 1 is also the same. View-dependent processing refers to processing that is related to the current viewpoint of the edge user of the mobile device 100 (i.e., the viewpoint of the aforementioned camera); for example, in the latest state of the game scene, although user 1 and user 2 are in the same copy of the same game scene, because their viewpoints are different, the lighting and shadow effects related to high-order rendering type 1 seen by user 1 and user 2 are different, therefore the high-order rendering of high-order rendering type 1 is also different.

[0220] In some embodiments, the method further includes: the runtime of the edge 3D engine of the mobile device 200 performs basic rendering of the edge scene 1 to obtain basic rendering data 5; the high-order rendering service of the server 200 determines whether the preprocessing of the high-order rendering type 1 of the cloud scene 1 is view-independent processing; if so, the preprocessed data of high-order rendering type 1 can be shared with other users participating in the edge scene 1 at the same time, such as user 2, that is, the edge-cloud collaboration framework sends the encoded preprocessed data 4 to the edge-cloud collaboration plugin SDK of the mobile device 200. The edge-cloud collaboration plugin SDK of the mobile device 200 sends the decoded preprocessed data 4 to the runtime of the edge 3D engine, and the runtime of the edge 3D engine merges the basic rendering data 5 and the preprocessed data 4 to obtain an image 5 with the rendering effect of high-order rendering type 1.

[0221] In the second implementation scheme provided in this application embodiment, the native edge 3D engine supports the above-mentioned edge-cloud collaborative rendering method.

[0222] For example, Figure 9 A schematic diagram of the system architecture of an edge-cloud collaborative rendering system involved in implementing Scheme 2 is shown. Compared to Figure 5 The system architecture shown is for deploying the cloud collaboration plugin SDK. Figure 9 The main difference between the edge-cloud collaborative rendering systems shown is:

[0223] (1) Figure 5The first implementation scheme shown provides edge-cloud collaborative rendering capabilities to the edge-side 3D engine through an edge-cloud collaborative plugin SDK. This edge-cloud collaborative plugin SDK can be obtained through app stores and browser downloads, making its existence readily apparent to developers. Figure 9 The second implementation scheme shown is that the edge 3D engine natively supports the above-mentioned edge-cloud collaborative rendering method; the functions in the edge-cloud collaborative plugin SDK can be built into the edge 3D engine in the form of modules / pre-built plugins, and developers do not need to download and install the functions through app stores or other means.

[0224] (2) Figure 5 In the first implementation scheme shown, the edge-side 3D engine is usually different from the cloud-side 3D engine, therefore, scene conversion capabilities need to be provided in the edge-cloud collaborative plugin SDK. Figure 9 In the second implementation scheme shown, when the cloud-side 3D engine and the edge-side 3D engine are identical, the scene conversion function module is no longer needed in the edge-cloud collaboration module. It should be noted that even without scene conversion in implementation scheme two, the cloud-side scene and the edge-side scene may not be completely identical; for example, the cloud-side scene may reduce the accuracy of the mesh information.

[0225] (3) Figure 5 In the first implementation scheme shown, it is necessary to add the ability to consume and render preprocessed data on top of the original rendering pipeline. Therefore, pipeline adaptation capabilities need to be provided in the end-to-cloud collaborative plugin SDK. For example, see... Figure 7 The pipeline adaptation module adds a fusion channel to the edge rendering pipeline to consume preprocessed data, based on the expansion mechanism of the edge 3D engine rendering pipeline. Figure 9 In the second implementation scheme shown, the edge-cloud collaborative module of the edge 3D engine can provide a native edge-cloud collaborative rendering pipeline, which itself can consume preprocessed data; thus, it is no longer necessary to Figure 7 The pipeline adaptation functional modules are shown.

[0226] In the second implementation scheme of this application, the method flow in the development phase and the method flow in the runtime phase are similar to those in the first implementation scheme mentioned above, and will not be repeated here.

[0227] In this embodiment, the first electronic device can be the aforementioned mobile device 100, and the second electronic device can be the aforementioned mobile device 300; the first request can be the aforementioned preprocessing request; the first application can be the aforementioned 3D application (e.g., a game app); the first terminal scene can be the aforementioned terminal scene 1, the first high-order rendering type can be the aforementioned high-order rendering type 1, and the first cloud scene can be the aforementioned cloud scene 1; the second high-order rendering type can be the aforementioned high-order rendering type 2, and the second cloud scene can be the aforementioned cloud scene 2; the first basic rendering data can be the aforementioned basic rendering data 4, the first state data can be the aforementioned state synchronization data 2, the first preprocessing data can be the aforementioned preprocessing data 4; the first image can be the aforementioned image 4; the first coordinate can be the aforementioned coordinate 1; the third basic rendering data can be the aforementioned basic rendering data 5, and the third image can be the aforementioned image 5.

[0228] The structure of a mobile device 100 provided in the embodiments of this application is described below. Figure 10 A schematic diagram of the structure of the mobile device 100 is shown.

[0229] The mobile device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0230] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the mobile device 100. In other embodiments of this application, the mobile device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0231] Processor 110 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). These different processing units may be independent devices or integrated into one or more processors.

[0232] The controller can generate operation control signals based on the instruction opcode and timing signals to complete the control of instruction fetching and execution.

[0233] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0234] In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0235] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple I2C buses. The processor 110 can couple to the touch sensor 180K, charger, flash, camera 193, etc., through different I2C bus interfaces. For example, the processor 110 can couple to the touch sensor 180K through the I2C interface, enabling the processor 110 and the touch sensor 180K to communicate through the I2C bus interface, thereby realizing the touch function of the mobile device 100.

[0236] The I2S interface can be used for audio communication. In some embodiments, the processor 110 may include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to enable communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 via the I2S interface to enable the function of answering phone calls through a Bluetooth headset.

[0237] The PCM interface can also be used for audio communication, sampling, quantizing, and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 can be coupled via the PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 via the PCM interface, enabling the function of answering phone calls through a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0238] The UART interface is a universal serial data bus used for asynchronous communication. This bus can be a bidirectional communication bus. It converts the data to be transmitted between serial and parallel communication. In some embodiments, the UART interface is typically used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 via the UART interface to implement Bluetooth functionality. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 via the UART interface to enable music playback through Bluetooth headphones.

[0239] The MIPI interface can be used to connect the processor 110 to peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI) and a display serial interface (DSI). In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to enable the shooting function of the mobile device 100. The processor 110 and the display screen 194 communicate via the DSI interface to enable the display function of the mobile device 100.

[0240] The GPIO interface can be configured via software. It can be configured as a control signal or a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 to a camera 193, a display screen 194, a wireless communication module 160, an audio module 170, a sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0241] USB port 130 is a USB standard compliant interface, specifically a Mini USB port, Micro USB port, or USB Type-C port. USB port 130 can be used to connect a charger to charge mobile device 100, and can also be used for data transfer between mobile device 100 and peripheral devices. It can also be used to connect headphones for audio playback. This interface can also be used to connect other electronic devices, such as AR devices.

[0242] It is understood that the interface connection relationships between the modules illustrated in the embodiments of the present invention are merely illustrative and do not constitute a structural limitation on the mobile device 100. In other embodiments of this application, the mobile device 100 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0243] The charging management module 140 receives charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 receives charging input from the wired charger via a USB interface 130. In some wireless charging embodiments, the charging management module 140 receives wireless charging input via the wireless charging coil of the mobile device 100. While charging the battery 142, the charging management module 140 can also supply power to the electronic device via the power management module 141.

[0244] The power management module 141 connects the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, providing power to the processor 110, internal memory 121, display screen 194, camera 193, and wireless communication module 160, etc. The power management module 141 can also monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance). In some other embodiments, the power management module 141 may also be located within the processor 110. In other embodiments, the power management module 141 and the charging management module 140 may be located in the same device.

[0245] The wireless communication function of the mobile device 100 can be implemented through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor, and baseband processor.

[0246] Antennas 1 and 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the mobile device 100 can be used to cover one or more communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with a tuning switch.

[0247] The mobile communication module 150 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the mobile device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In some embodiments, at least some functional modules of the mobile communication module 150 may be housed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be housed in the same device.

[0248] The modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs sound signals through an audio device (not limited to speaker 170A, receiver 170B, etc.) or displays images or videos through the display screen 194. In some embodiments, the modem processor may be a separate device. In other embodiments, the modem processor may be independent of the processor 110 and may be housed in the same device as the mobile communication module 150 or other functional modules.

[0249] The wireless communication module 160 can provide solutions for wireless communication applications on the mobile device 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, demodulates and filters the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, frequency modulate and amplify them, and then convert them into electromagnetic waves for radiation via antenna 2.

[0250] In some embodiments, antenna 1 of mobile device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, enabling mobile device 100 to communicate with networks and other devices via wireless communication technology. The wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. The GNSS may include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the BeiDou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or satellite-based augmentation systems (SBAS).

[0251] The mobile device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0252] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a miniature LED, a microLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, mobile device 100 may include one or N displays 194, where N is a positive integer greater than 1.

[0253] The mobile device 100 can perform shooting functions through an ISP, camera 193, video codec, GPU, display 194, and application processor.

[0254] The ISP (Image Signal Processor) is used to process data fed back from the camera 193. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, converting it into an image visible to the naked eye. The ISP can also perform algorithmic optimization on image noise and brightness. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0255] Camera 193 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, the mobile device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0256] Digital signal processors (DSPs) are used to process digital signals. Besides digital image signals, they can also process other digital signals. For example, when the mobile device 100 selects a frequency, the DSP performs Fourier transforms on the frequency energy.

[0257] Video codecs are used to compress or decompress digital video. Mobile device 100 may support one or more video codecs. Thus, mobile device 100 can play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG 2, MPEG 3, MPEG 4, etc.

[0258] NPU stands for Neural Network (NN) Computing Processor. By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs enable intelligent cognitive applications in mobile devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0259] Internal memory 121 may include one or more random access memory (RAM) and one or more non-volatile memory (NVM).

[0260] Random access memory can include static random-access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), and double data rate synchronous dynamic random access memory (DDR SDRAM, such as fifth-generation DDR SDRAM, which is generally called DDR5 SDRAM). Non-volatile memory can include disk storage devices and flash memory.

[0261] Flash memory can be classified according to its operating principle, including NOR FLASH, NAND FLASH, 3D NAND FLASH, etc.; according to the level of the storage cell, including single-level cell (SLC), multi-level cell (MLC), triple-level cell (TLC), quad-level cell (QLC), etc.; and according to the storage specification, including universal flash storage (UFS) and embedded multimedia card (eMMC), etc.

[0262] The random access memory can be directly read and written by the processor 110. It can be used to store executable programs (such as machine instructions) of the operating system or other running programs, as well as user and application data.

[0263] Non-volatile memory can also store executable programs and user and application data, and can be pre-loaded into random access memory for direct reading and writing by the processor 110.

[0264] The external memory interface 120 can be used to connect to external non-volatile memory, thereby expanding the storage capacity of the mobile device 100. The external non-volatile memory communicates with the processor 110 through the external memory interface 120 to perform data storage functions. For example, music, video, and other files can be stored in the external non-volatile memory.

[0265] The mobile device 100 can implement audio functions, such as music playback and recording, through an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, and an application processor.

[0266] The audio module 170 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 170 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 170 may be located in the processor 110, or some functional modules of the audio module 170 may be located in the processor 110.

[0267] The speaker 170A, also known as a "loudspeaker," is used to convert audio electrical signals into sound signals. The mobile device 100 can listen to music or make hands-free calls through the speaker 170A.

[0268] The receiver 170B, also known as the "earpiece," is used to convert audio electrical signals into sound signals. When the mobile device 100 receives a phone call or voice message, it can listen to the voice by bringing the receiver 170B close to the user's ear.

[0269] Microphone 170C, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. When making a phone call or sending a voice message, the user can speak by bringing their mouth close to microphone 170C, inputting the sound signal into microphone 170C.

[0270] The 170D headphone jack is used to connect wired headphones. The 170D headphone jack can be a USB 130 interface or a 3.5mm Open Mobile Terminal Platform (OMTP) standard interface, a CTIA (Cellular Telecommunications Industry Association of the USA) standard interface.

[0271] The pressure sensor 180A is used to sense pressure signals and can convert the pressure signals into electrical signals. In some embodiments, the pressure sensor 180A may be disposed on the display screen 194.

[0272] The gyroscope sensor 180B can be used to determine the motion attitude of the mobile device 100. In some embodiments, the angular velocity of the mobile device 100 about three axes (i.e., the x, y, and z axes) can be determined by the gyroscope sensor 180B.

[0273] The barometric pressure sensor 180C is used to measure air pressure. In some embodiments, the mobile device 100 calculates altitude using the air pressure value measured by the barometric pressure sensor 180C to assist in positioning and navigation.

[0274] The magnetic sensor 180D includes a Hall sensor. The mobile device 100 can use the magnetic sensor 180D to detect the opening and closing of the flip cover.

[0275] The accelerometer 180E can detect the magnitude of acceleration of the mobile device 100 in various directions (typically three axes).

[0276] The distance sensor 180F is used to measure distance. The mobile device 100 can measure distance via infrared or laser.

[0277] The proximity light sensor 180G may include, for example, a light-emitting diode (LED) and a light detector, such as a photodiode.

[0278] An ambient light sensor 180L is used to sense the ambient light intensity. The mobile device 100 can adaptively adjust the brightness of its display screen 194 according to the sensed ambient light intensity.

[0279] The fingerprint sensor 180H is used to collect fingerprints. The mobile device 100 can use the collected fingerprint characteristics to achieve fingerprint unlocking, app access lock, fingerprint photography, fingerprint answering of incoming calls, etc.

[0280] Temperature sensor 180J is used to detect temperature. In some embodiments, mobile device 100 uses the temperature detected by temperature sensor 180J to execute a temperature processing strategy.

[0281] Touch sensor 180K, also known as a "touch device," can be located on display screen 194. The touch sensor 180K and display screen 194 together form a touchscreen, also known as a "touchscreen." Touch sensor 180K detects touch operations applied to or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 194. In other embodiments, touch sensor 180K may also be located on the surface of mobile device 100, in a different position than display screen 194.

[0282] The bone conduction sensor 180M can acquire vibration signals. In some embodiments, the bone conduction sensor 180M can acquire vibration signals from the vibrating bone segments of the human vocal cords.

[0283] Buttons 190 include a power button, volume buttons, etc. Buttons 190 can be mechanical buttons or touch-sensitive buttons. The mobile device 100 can receive button input and generate key signal inputs related to user settings and function control of the mobile device 100.

[0284] Motor 191 can generate vibration alerts. Motor 191 can be used for incoming call vibration alerts or for touch vibration feedback.

[0285] Indicator 192 can be an indicator light, used to indicate charging status, power changes, or to indicate messages, missed calls, notifications, etc.

[0286] The SIM card interface 195 is used to connect the SIM card.

[0287] The following describes the structure of a server 200 provided in an embodiment of this application. Figure 11 An exemplary embodiment of the present application illustrates the structure of a server 200.

[0288] like Figure 11As shown, server 200 may include: one or more processors 1001, memory 1002, communication interface 1003, transmitter 1005, receiver 1006, coupler 1007, and antenna 1008. These components can be connected via bus 1004 or other means. Figure 11 Taking a bus connection as an example:

[0289] The communication interface 1003 can be used by the server 200 to communicate with other communication devices, such as the terminal device 100. Specifically, the communication interface 1003 can be a 3G communication interface, a 4G communication interface, a 5G communication interface, or a future new radio interface, etc. Not limited to wireless communication interfaces, the server 200 can also be configured with a wired communication interface 1003, such as a local access network (LAN) interface. The transmitter 1005 can be used to transmit and process signals output by the processor 1001. The receiver 1006 can be used to receive and process mobile communication signals received by the antenna 1008.

[0290] In some embodiments of this application, transmitter 1005 and receiver 1006 can be considered as a wireless modem. In server 200, the number of transmitters 1005 and receivers 1006 can be one or more. Antenna 1008 can be used to convert electromagnetic energy in a transmission line into electromagnetic waves in free space, or to convert electromagnetic waves in free space into electromagnetic energy in a transmission line. Coupler 1007 is used to split the mobile communication signal received by antenna 1008 into multiple paths and distribute them to multiple receivers 1006.

[0291] Memory 1002 is coupled to processor 1001 and is used to store various software programs and / or sets of instructions. Specifically, memory 1002 may include high-speed random access memory and may also include non-volatile memory, such as one or more disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 1002 may store network communication programs that can be used to communicate with one or more auxiliary devices, one or more terminal devices, or one or more network devices.

[0292] In some embodiments of this application, the memory 1002 may be used to store the implementation program of the application distribution method provided in one or more embodiments of this application on the server 200 side. For the implementation of the application distribution method provided in one or more embodiments of this application, please refer to the above embodiments.

[0293] The processor 1001 can be used to read and execute computer-readable instructions. Specifically, the processor 1001 can be used to invoke a program stored in the memory 1002, such as the implementation program of the application distribution method provided in one or more embodiments of this application on the server 200 side, and execute the instructions contained in the program.

[0294] It should be noted that, Figure 11 The server 200 shown is merely one implementation of the embodiments of this application. In actual applications, the server 200 may include more or fewer components, which is not limited here.

[0295] For more details on the functions and working principles of server 200, please refer to the relevant content in the above embodiments, which will not be repeated here.

[0296] The various embodiments of this application can be combined arbitrarily to achieve different technical effects.

[0297] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0298] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

[0299] In summary, the above description is merely an embodiment of the technical solution of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made according to the disclosure of the present invention should be included within the scope of protection of the present invention.

Claims

1. A rendering method for edge-cloud collaboration, characterized in that, include: The first electronic device performs basic rendering on the first edge scene and obtains the first basic rendering data; The first electronic device stores at least one end-side scene, and the first end-side scene is any one of the at least one end-side scenes; The first electronic device sends the scene identifier and the first higher-order rendering type of the first terminal scene to the server; wherein, the scene identifier and the first higher-order rendering type of the first terminal scene are obtained by the first electronic device from the application package of the first application; The first electronic device sends first status data to the server, the first status data including data related to the preprocessing of the first high-order rendering type in the status data of the first terminal scene; The first electronic device receives first preprocessed data sent by the server; the first preprocessed data is obtained by the server performing preprocessing of the first cloud-side scene with the first high-order rendering type after updating the first cloud-side scene according to the first state data, and the first preprocessed data is intermediate data used to indicate the lighting and shadow effects of the first high-order rendering type in the first cloud-side scene; the server stores the first cloud-side scene transformed from the first terminal scene for the first high-order rendering type, and the server stores the first correspondence between the first terminal scene, the first high-order rendering type and the first cloud-side scene; wherein, the scene identifier of the first cloud-side scene, and the first correspondence between the first terminal scene, the first high-order rendering type and the first cloud-side scene are deployed on the server during the application development of the first application; The first electronic device obtains a first image based on the first basic rendering data and the first preprocessed data, and the first image has the rendering effect of the first high-order rendering type; The first electronic device displays the first image.

2. The method according to claim 1, characterized in that, The method further includes: The first electronic device updates the scene information of the first terminal scene; The first electronic device performs basic rendering on the updated first edge scene to obtain second basic rendering data; If the network speed between the first electronic device and the server is lower than a preset value, or if the network between the first electronic device and the server is disconnected, the first electronic device determines the second image based on the second basic rendering data. The first electronic device displays the second image.

3. The method according to claim 1, characterized in that, The first electronic device sends the scene identifier and the first high-order rendering type of the first edge scene to the server, including: The first electronic device sends a first request to the server. The first request includes a scene identifier of the first edge scene and a first high-order rendering type. The first request is used to trigger the server to perform preprocessing preparation. The preprocessing preparation includes loading the first cloud-side scene and preparing a rendering pipeline for preprocessing of the first high-order rendering type. Before the first electronic device sends the first status data to the server, the method further includes: The first electronic device receives confirmation information sent by the server, the confirmation information being used to indicate that preprocessing preparation has been completed; The first electronic device sends first status data to the server, including: Based on the determined information, the first electronic device sends first status data to the server.

4. The method according to any one of claims 1 to 3, characterized in that, The state data of the first edge scene includes some or all of the following: scene identification information, light source information, character information, camera information, and scene update information.

5. The method according to any one of claims 1 to 3, characterized in that, The server stores a second cloud-side scene converted from the first edge-side scene to a second higher-order rendering type.

6. The method according to any one of claims 1 to 3, characterized in that, The first electronic device receives first preprocessed data sent by the server, including: The first electronic device receives the first preprocessed data after compression encoding sent by the server; Before the first electronic device acquires the first image based on the first basic rendering data and the first preprocessed data, it further includes: The first electronic device decodes the received first preprocessed data to obtain the decoded first preprocessed data.

7. The method according to any one of claims 1 to 3, characterized in that, The first higher-order rendering type is global illumination; The first preprocessing data includes the irradiance of each pixel in the image obtained by preprocessing global illumination, and the first basic rendering data includes the irradiance of each pixel in the image after basic rendering. The irradiance of the pixel at the first coordinate in the first image is equal to the product of the irradiance of the pixel at the first coordinate in the first preprocessed data and the first base rendering data. The pixel value of the pixel at the first coordinate in the first image is determined based on the irradiance of the pixel at the first coordinate.

8. A rendering method for edge-cloud collaboration, characterized in that, The method includes: The server receives a scene identifier and a first high-order rendering type of a first terminal scene sent by a first electronic device; the server stores a first correspondence between the first terminal scene, the first high-order rendering type, and the first cloud scene; wherein, the scene identifier and the first high-order rendering type of the first terminal scene are obtained by the first electronic device from the application package of the first application; the scene identifier of the first cloud scene, as well as the first correspondence between the first terminal scene, the first high-order rendering type, and the first cloud scene, are deployed on the server during the application development of the first application; The server receives first status data sent by the first electronic device, the first status data including data related to the preprocessing of the first high-order rendering type in the status data of the first terminal scene. The server updates the first cloud-side scene according to the first state data, performs preprocessing of the first high-order rendering type on the updated first cloud-side scene, and obtains first preprocessing data. The first preprocessing data is intermediate data used to indicate the lighting and shadow effects of the first high-order rendering type in the first cloud-side scene. The server sends the first preprocessed data to the first electronic device; The first preprocessed data is used to fuse the first basic rendering data to obtain the first image; the first image has the lighting and shadow effects of the first high-order rendering type; the first basic rendering data is obtained by the first electronic device performing basic rendering on the first edge scene.

9. The method according to claim 8, characterized in that, The server receives the scene identifier and first high-order rendering type of the first edge scene sent by the first electronic device, including: The server receives a first request sent by the first electronic device, the first request including a scene identifier of the first terminal scene and a first high-order rendering type; Before the server receives the first status data sent by the first electronic device, the method further includes: Based on the first request, the server performs preprocessing preparation, which includes loading the first cloud-side scene and preparing the rendering pipeline for preprocessing of the first high-order rendering type. The server sends a confirmation message to the first electronic device, the confirmation message indicating that preprocessing preparation has been completed. The first status data is sent by the first electronic device based on the determined information.

10. The method according to claim 8 or 9, characterized in that, The state data of the first edge scene includes some or all of the following: scene identification information, light source information, character information, camera information, and scene update information.

11. The method according to claim 8 or 9, characterized in that, The server stores a first correspondence between the first terminal scene, the first high-order rendering type, and the first cloud scene. Before the server updates the first cloud scene based on the first state data, the method further includes: Based on the first correspondence, the scene identifier of the first terminal scene, and the first high-order rendering type, the server determines the cloud-side scene to be rendered as the first cloud-side scene.

12. The method according to claim 8 or 9, characterized in that, The server stores a second cloud-side scene converted from the first edge-side scene to a second higher-order rendering type.

13. The method according to claim 8 or 9, characterized in that, The server sends the first preprocessed data to the first electronic device, including: The server sends the compressed and encoded first preprocessed data to the first electronic device.

14. The method according to claim 8 or 9, characterized in that, The first higher-order rendering type is global illumination; The first preprocessing data includes the irradiance of each pixel in the image obtained by preprocessing global illumination, and the first basic rendering data includes the irradiance of each pixel in the image after basic rendering. The irradiance of the pixel at the first coordinate in the first image is equal to the product of the irradiance of the pixel at the first coordinate in the first preprocessed data and the first base rendering data. The pixel value of the pixel at the first coordinate in the first image is determined based on the irradiance of the pixel at the first coordinate.

15. The method according to claim 8 or 9, characterized in that, The method further includes: If the first preprocessed data is view-independent preprocessed data, the server sends the first preprocessed data to the second electronic device; the first electronic device and the second electronic device are electronic devices of the same user participating in the first end-side scene.

16. An electronic device, characterized in that, include: One or more processors, one or more memories; the one or more memories are coupled to the one or more processors; The one or more memories are used to store a computer program that, when run on the processor, causes the electronic device to perform the method as described in any one of claims 1-7.

17. A server, characterized in that, include: One or more processors, one or more memories; the one or more memories are coupled to the one or more processors; The one or more memories are used to store a computer program that, when run on the processor, causes the server to perform the method as described in any one of claims 8-15.

18. A computer-readable medium, characterized in that, A computer program is stored, which can be executed by a processor to implement the method of any one of claims 1-15.

Images