A cloud rendering method and related device

Abstract

The specification provides a cloud rendering method and related equipment, applied to a cloud mobile phone server corresponding to a terminal device, wherein the cloud mobile phone server is used to provide rendering service to the terminal device. The method comprises: receiving interaction data sent by the terminal device, and generating corresponding rendering instructions based on the interaction data; sending the rendering instructions to the operation server of a third party connected with the cloud mobile phone server, so as to analyze the rendering instructions by the operation server, generate corresponding video stream based on the analysis result, and return the video stream to the terminal device, so that the terminal device displays based on the video stream; wherein the rendering capability of the cloud mobile phone server is lower than that of the operation server.

Description

TECHNICAL FIELD

[0001] One or more embodiments of the present specification relate to the technical field of cloud computing, and in particular to a cloud rendering method and related equipment. BACKGROUND

[0002] The metaverse is a virtual world built by humans using digital technology, which is mapped from or transcends the real world and can interact with the real world. The metaverse relies heavily on image rendering capabilities, so the image rendering capabilities of terminal devices running various metaverse games or applications are extremely high, and usually only terminal devices with strong hardware performance can meet the image rendering capabilities required by various metaverse games or applications, thereby resulting in high cost and failing to meet the actual use requirements of users. SUMMARY

[0003] Therefore, one or more embodiments of the present specification provide a cloud rendering method and related equipment.

[0004] In a first aspect, the present specification provides a cloud rendering method applied to a cloud mobile phone server corresponding to a terminal device, wherein the cloud mobile phone server is configured to provide rendering services to the terminal device; the method comprises:

[0005] receiving interaction data sent by the terminal device, and generating corresponding rendering instructions based on the interaction data;

[0006] sending the rendering instructions to a third-party computing server connected to the cloud mobile phone server, so that the computing server analyzes the rendering instructions, performs image rendering based on the analysis result to generate corresponding video streams, and returns the video streams to the terminal device, so that the terminal device displays based on the video streams;

[0007] wherein the rendering capability of the cloud mobile phone server is lower than the rendering capability of the computing server.

[0008] In a second aspect, the present specification provides a cloud rendering method applied to a third-party computing server connected to a cloud mobile phone server, wherein the cloud mobile phone server is configured to provide rendering services to a corresponding terminal device; the method comprises:

[0009] receiving rendering instructions sent by the cloud mobile phone server; the rendering instructions are instructions generated by the cloud mobile phone server based on received interaction data sent by the terminal device;

[0010] analyzing the rendering instructions and performing image rendering based on the analysis result to generate corresponding video streams;

[0011] sending the video streams to the terminal device, so that the terminal device displays based on the video streams;

[0012] The rendering capability of the cloud phone server is lower than the rendering capability of the operation server.

[0013] In a third aspect, the present specification provides a cloud rendering device applied to a cloud phone server corresponding to a terminal device, wherein the cloud phone server is configured to provide rendering service to the terminal device; the device comprises:

[0014] a receiving unit configured to receive interaction data sent by the terminal device and generate corresponding rendering instructions based on the interaction data;

[0015] a sending unit configured to send the rendering instructions to an operation server of a third party connected to the cloud phone server, so that the operation server analyzes the rendering instructions, generates corresponding video streams based on the analysis results, and returns the video streams to the terminal device, so that the terminal device displays based on the video streams;

[0016] The rendering capability of the cloud phone server is lower than the rendering capability of the operation server.

[0017] In a fourth aspect, the present specification provides a cloud rendering device applied to an operation server of a third party connected to a cloud phone server, wherein the cloud phone server is configured to provide rendering service to a corresponding terminal device; the device comprises:

[0018] a receiving unit configured to receive rendering instructions sent by the cloud phone server; the rendering instructions are instructions generated by the cloud phone server based on received interaction data sent by the terminal device;

[0019] a rendering unit configured to analyze the rendering instructions and generate corresponding video streams based on the analysis results;

[0020] a sending unit configured to send the video streams to the terminal device, so that the terminal device displays based on the video streams;

[0021] The rendering capability of the cloud phone server is lower than the rendering capability of the operation server.

[0022] Correspondingly, the present specification also provides a computing device comprising a memory and a processor; the memory stores a computer program executable by the processor; when the processor executes the computer program, the cloud rendering method of the first aspect or the second aspect is executed.

[0023] Correspondingly, the specification also provides a computer readable storage medium, which stores a computer program. When the computer program is run by a processor, the cloud rendering method according to the first aspect or the second aspect is executed.

[0024] In summary, the cloud phone server in the present application can receive the interaction data uploaded by the terminal device corresponding thereto, and generate a series of rendering instructions based on the interaction data. Subsequently, the cloud phone server can forward the rendering instructions to the operation server of the third party connected thereto, so that the operation server performs image rendering based on the rendering instructions, generates a video stream to be displayed, and returns the video stream to the terminal device, so that the terminal device displays based on the received video stream. In this way, the present application takes into account the problem that the existing cloud phone server is basically a CPU server, which has poor rendering capability. The image rendering originally performed in the cloud phone server can be directly transferred to the existing operation server with strong rendering capability. Not only the existing computing resources are reasonably utilized, but also the hardware requirements for image rendering of the cloud phone are effectively reduced, thereby reducing the implementation cost of the cloud phone, which is conducive to the large-scale application of the cloud phone. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 FIG. 1 is a schematic diagram of a system architecture provided by an example embodiment;

[0026] Figure 2 FIG. 2 is a flowchart of a cloud rendering method provided by an example embodiment;

[0027] Figure 3 FIG. 3 is a flowchart of another cloud rendering method provided by an example embodiment;

[0028] Figure 4 FIG. 4 is a schematic diagram of a cloud rendering device provided by an example embodiment;

[0029] Figure 5 FIG. 5 is a schematic diagram of another cloud rendering device provided by an example embodiment;

[0030] Figure 6 FIG. 6 is a schematic diagram of a computing device provided by an example embodiment. DETAILED DESCRIPTION

[0031] The exemplary embodiments will be described in detail herein with reference to the attached drawings. The following description is made with reference to the accompanying drawings in which like reference numerals represent like elements, unless the context of use indicates otherwise. The following exemplary embodiments described herein are not meant to represent all embodiments consistent with one or more embodiments of the present description. Rather, they are merely examples of apparatuses and methods consistent with some aspects of one or more embodiments of the present description as detailed in the appended claims.

[0032] It should be noted that the steps of the methods in other embodiments are not necessarily performed in the order shown and described in the present description. In some other embodiments, the steps included in the methods can be more or less than described in the present description. Furthermore, a single step described in the present description can be broken down into multiple steps in other embodiments; and multiple steps described in the present description can be combined into a single step in other embodiments.

[0033] In addition, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation portal for user to choose authorization or refusal.

[0034] First, some terms in the present description are explained and described to facilitate understanding by those skilled in the art.

[0035] (1) Meta universe: a virtual world built by humans using digital technology, which is mapped from or beyond the real world and can interact with the real world. The meta universe is a digital living space with a new social system, and VR / AR is the only way to realize the concept of the meta universe. At present, in order to meet the future infrastructure computing power demand of the meta universe, it is necessary to build a new ARM ecological cloud computing to support the future development of the meta universe.

[0036] (2) Cloud phone: a mobile phone operating system (mainly Android system) running on a cloud server. Cloud phone is an important infrastructure for cloud application of meta universe mobile terminal in the future. Cloud phone can meet various application scenarios (such as VR / AR games), realize cloud computing, terminal front-end and lightweight, so as to improve the immersion and interactivity of users.

[0037] (3) Image rendering, mainly applied in the fields of games (including VR / AR games), 3D graphics demonstration (such as 3D demonstration of a real estate) and virtual simulation. It should be noted that both CPU and GPU can perform image rendering, but the running mode of GPU is different. GPU uses GPU rendering acceleration, uses a large-scale parallel architecture composed of thousands of smaller and more efficient cores, and simultaneously processes rendering tasks. Therefore, under the same rendering workload, GPU performs image rendering faster and more efficiently.

[0038] As mentioned above, the metaverse relies heavily on image rendering capability, so the image rendering capability of the terminal device running various metaverse games or applications is extremely high. Usually, only terminal devices with strong hardware performance can support the image rendering of various metaverse games or applications, thereby resulting in a higher cost and failing to meet the actual use requirements of users. Based on this, in some possible implementation manners, cloud phones can be used to provide rendering services on the cloud to terminal devices, so that terminal devices with ordinary hardware conditions can also display smooth and clear pictures.

[0039] Cloud phones are mainly based on ARM ecology. Under the ARM ecology, cloud phone servers are mostly CPU servers. As mentioned above, compared with the excellent rendering capability of GPU, CPU basically cannot perform reliable rendering. Therefore, in order to realize efficient and reliable cloud rendering, the current cloud phone scheme (such as a container cloud phone based on container technology and a cloud phone based on virtualization technology such as KVM) often relies on an extra cloud phone dedicated GPU server for image rendering, thereby greatly increasing the implementation cost of the cloud phone, hindering the large-scale deployment and application of the cloud phone. At the same time, since the GPU server used by these cloud phone schemes is usually cloud phone dedicated and cannot be used for other purposes, the GPU utilization is extremely low, thereby causing waste of computing resources.

[0040] Based on this, the present specification provides a technical scheme for forwarding rendering instructions to be executed on a cloud phone server to an existing operation server with strong rendering capability, so as to perform image rendering by the operation server, thereby reducing the implementation cost of the cloud phone while ensuring the efficiency of cloud rendering.

[0041] In implementation, the cloud phone server can receive interaction data uploaded by a terminal device corresponding thereto, and generate a series of rendering instructions based on the interaction data. Subsequently, the cloud phone server can forward the rendering instructions to a third-party operation server connected thereto, so as to perform image rendering by the operation server based on the rendering instructions, generate a video stream to be displayed, and return the video stream to the terminal device, so that the terminal device displays based on the received video stream.

[0042] In the above technical solution, the present application considers the problem that the existing cloud phone server is basically a CPU server and has poor rendering capability. The image rendering originally performed in the cloud phone server can be directly transferred to an existing operation server with strong rendering capability. Not only the existing computing resources are reasonably utilized, but also the hardware requirements for image rendering of the cloud phone are effectively reduced, thereby reducing the implementation cost of the cloud phone, which is conducive to large-scale deployment and application of the cloud phone.

[0043] Please refer to Figure 1 , Figure 1 is a system architecture schematic diagram provided by an exemplary embodiment. The technical solution of the embodiments of the present application can be implemented in the system architecture shown in Figure 1 or similar system architecture. As shown in Figure 1 , the system can include a terminal device, a session server, a streaming server, a cloud phone server and an operation server.

[0044] As shown in Figure 1 , in an illustrated embodiment, the terminal device can be used to run a series of application programs and output a corresponding application interface to the user, and the user can perform corresponding interactive operations on the application interface. For example, the application program can include an AR or VR game, and the application interface can include a game screen, and the user can perform corresponding interactive operations (such as clicking and sliding) on the game screen to control the game character to shoot or jump, etc., which is not limited in the present application.

[0045] In an illustrated embodiment, the terminal device can be a smart wearable device (such as AR glasses and VR headgear) with the above functions, a smart phone, a tablet computer, a notebook computer, a desktop computer, etc., which is not limited in the present application.

[0046] In an illustrated embodiment, the terminal device can be a terminal device based on WEB, an android operating system or an IOS operating system, etc., which is not limited in the present application.

[0047] As shown in Figure 1 , in an illustrated embodiment, the cloud phone server can be used to build one or more cloud phones to provide a series of cloud computing for one or more terminal devices corresponding thereto. For example, the terminal device (including a cloud phone client) can send a cloud phone rental request to the cloud phone server, and the cloud phone server can assign a cloud phone to the terminal device to provide cloud computing. For example, the cloud phone can provide cloud rendering services for the terminal device to display smooth and clear pictures based on real-time operation even if the hardware condition of the terminal device is ordinary.

[0048] In one illustrated embodiment, the cloud phone server may include a server that provides computing power based on a CPU, i.e., a CPU server.

[0049] In one illustrated embodiment, the cloud phone server may include a server based on the Android operating system; that is, the operating system of the cloud phone server may be the Android operating system, or one or more cloud phones built on the cloud phone server may be running the Android operating system. In one illustrated embodiment, the operating system of the cloud phone server may also be any other possible operating system besides Android, such as the iOS operating system, etc., and this specification does not specifically limit it in this way.

[0050] In one of the illustrated embodiments, the cloud phone server may include an ARM-based server or any other possible architecture, which is not specifically limited in this specification.

[0051] like Figure 1 As shown, in one illustrated embodiment, the computing server can be a third-party server that interfaces with the cloud phone server, and can be used to assist the cloud phone server in achieving image rendering in the cloud. In one illustrated embodiment, as... Figure 1 As shown, a third-party computing server can remotely connect to the cloud phone server via wireless means such as a network. In one illustrated embodiment, the computing server can obtain rendering instructions generated by the cloud phone server based on the user's interactive operations on the terminal device, and perform corresponding image rendering based on these instructions to generate a corresponding video stream. The server then returns the video stream to the terminal device, enabling the terminal device to display the video stream. For example, in a game scenario, the terminal device can receive the video stream and display a smooth and clear game screen based on it.

[0052] In one illustrated embodiment, the computing server may include a server that provides computing power based on a GPU, i.e., a GPU server. Generally, the rendering capability of a cloud phone server is lower than that of a computing server.

[0053] In one illustrated embodiment, the computing server may include a server based on the Linux operating system. In another illustrated embodiment, the operating system of the computing server may also be any other possible operating system besides Linux, and this specification does not specifically limit this.

[0054] In an illustrated embodiment, the operation server can include an X86 architecture-based server, or any other possible architecture, which is not specifically limited in the specification. Thus, the architecture of the cloud phone server and the operation server in the specification can be different, and the cloud rendering method provided in the specification can be applied in a heterogeneous deployment.

[0055] It should be noted that, Figure 1 The operation server shown is not only used to provide rendering services to the cloud phone server. In an illustrated embodiment, the operation server can also be used to perform any other possible computation, such as video coding, deep learning training, search, big data recommendation, etc., which is not specifically limited in the specification.

[0056] As Figure 1 shown, in an illustrated embodiment, the session server is used to provide user identity authentication and authorization for the cloud phone server when the user intends to log in to the cloud phone, so as to control the user's login (or access), etc., which is not expanded in detail here, and is not specifically limited in the specification.

[0057] As Figure 1 shown, in an illustrated embodiment, the streaming server can be used to provide a data transmission channel for the terminal device and the cloud phone server, so that the terminal device can upload the user's interaction data to the cloud phone server through the streaming server; and can also be used to provide a data transmission channel for the terminal device and the operation server, so that the operation server can send the video stream obtained after image rendering to the terminal device through the streaming server.

[0058] In summary, as Figure 1 shown, in an illustrated embodiment, when the terminal device runs the corresponding application, it can send the user's interaction data (including one or more interaction operations performed by the user on the corresponding application interface) to the cloud phone server through the streaming server. Then, the cloud phone server generates corresponding rendering instructions based on the received interaction data, and sends the rendering instructions to the operation server connected thereto. Then, the operation server parses the received rendering instructions, generates corresponding video streams based on the parsing results, and sends the video streams to the terminal device (this process can also be referred to as video push streaming). Finally, the terminal device displays based on the received video stream. In this way, not only the existing computing resources are reasonably utilized, the GPU utilization is improved, but also the hardware requirements for image rendering of the cloud phone are effectively reduced, thereby reducing the implementation cost of the cloud phone, which is conducive to the large-scale deployment and application of the cloud phone.

[0059] It should be understood that in the game scene, the image rendering in the present application is generally real-time rendering, which needs to continuously receive the user's real-time interactive operation uploaded by the terminal device through the cloud phone server and generate a series of rendering instructions, and then continuously calculate and render the game picture based on the series of rendering instructions through the operation server, so that the terminal device can display correct and smooth, clear game picture based on the user's real-time interactive operation.

[0060] It should be noted that, Figure 1 For example, in some possible implementations, more or fewer devices can be included in the system structure, for example, a database can also be included, which can be used to store user information, such as including user account, password, etc., which is not limited in the present application

[0061] Please refer to Figure 2 , Figure 2 is a flowchart of a cloud rendering method provided by an exemplary embodiment. The method can be applied to Figure 1 The cloud phone server based on CPU in the system architecture provides computing power. As Figure 2 shown, the method can specifically include the following steps S101-S102.

[0062] Step S101, receiving the interactive data sent by the terminal device, and generating corresponding rendering instructions based on the interactive data.

[0063] In an illustrated embodiment, the terminal device runs a corresponding application program and displays a corresponding application interface. The terminal device uploads the interactive data generated based on the user operation during the running of the application program to the cloud phone server corresponding thereto. Correspondingly, the cloud phone server receives the interactive data uploaded by the terminal device, and generates corresponding rendering instructions based on the interactive data.

[0064] In an illustrated embodiment, the interactive data can include one or more interactive operations performed by the user on the application interface through the terminal device. For example, in the game scene, the interactive operation includes clicking, sliding, dragging and other interactive operations performed by the user on the game interface to realize user login, control game character movement, shooting, etc., which is not limited in the present application. Correspondingly, the rendering instruction is used to render the game picture that the terminal device needs to display based on the user's interactive operation. The game picture can include game background, various icons, text information, game characters and controls, etc., which is not limited in the present application.

[0065] In an illustrated embodiment, the rendering instruction generated by the cloud phone server can include a rendering instruction based on an EGL / OpenGL interface standard. In an illustrated embodiment, the rendering instruction can also include any other possible type of instruction, such as a rendering instruction based on an OpenGL ES interface standard, etc., which is not specifically limited in the specification.

[0066] In an illustrated embodiment, the terminal device can upload the interaction data to the cloud phone server corresponding thereto by a streaming server between the terminal device and the cloud phone server.

[0067] In an illustrated embodiment, the terminal device can first send a connection request to the corresponding streaming server to establish a data transmission channel between the terminal device and the streaming server.

[0068] In an illustrated embodiment, the cloud phone server can also send a connection request to the streaming server to establish a data transmission channel between the cloud phone server and the streaming server.

[0069] Then, the terminal device can send the above-mentioned interaction data to the streaming server based on the established data transmission channel between the terminal device and the streaming server, so that the streaming server further sends the interaction data to the cloud phone server based on the established data transmission channel between the cloud phone server and the streaming server. Correspondingly, the cloud phone server receives the interaction data sent by the terminal device through the streaming server based on the established data transmission channel between the cloud phone server and the streaming server.

[0070] Step S102, send a rendering instruction to a third-party computing server interfaced with the cloud server, so that the computing server analyzes the rendering instruction, generates a corresponding video stream based on the analysis result, and returns the video stream to the terminal device, so that the terminal device displays based on the video stream.

[0071] In an illustrated embodiment, after generating the corresponding rendering instruction based on the received interaction data, the cloud phone server can send the rendering instruction to a third-party computing server interfaced therewith.

[0072] In an illustrated embodiment, the third-party computing server can be a cloud service platform or a cloud computing service center, which can include one or more computing servers providing computing power based on GPU. Figure 1

[0073] ​In an illustrated embodiment, if the operation server includes multiple operation servers, the cloud phone server can send the rendering instruction to a target server in the multiple operation servers included in the operation server, so that the target server performs image rendering based on the rendering instruction. In an illustrated embodiment, the target server can be an operation server that interfaces with the cloud phone server among the multiple operation servers included in the operation server, or the target server can be a server with relatively idle computing resources (or relatively low GPU utilization) among the multiple operation servers included in the operation server, and the like, which are not specifically limited in the present specification.

[0074] In an illustrated embodiment, if the operation server includes multiple operation servers, the cloud phone server can also send the rendering instruction to multiple operation servers that interface with it in the operation server, so that the multiple operation servers jointly perform image rendering to further improve the image rendering efficiency, reduce the picture delay of the terminal device, and the like, which are not specifically limited in the present specification.

[0075] In an illustrated embodiment, after receiving the rendering instruction sent by the cloud phone server, the operation server can parse the rendering instruction to obtain a corresponding parsing result.

[0076] In an illustrated embodiment, the parsing result obtained by the operation server parsing the rendering instruction can include instructions supported by the operation server for image rendering, i.e., the operation server can convert the rendering instruction generated by the cloud phone server (CPU server) received into a rendering instruction suitable for the operation server (GPU server).

[0077] Then, the operation server can perform image rendering based on the parsing result to generate a corresponding video stream, and return the video stream to the terminal device, so that the terminal device displays based on the video stream.

[0078] Specifically, in an illustrated embodiment, the operation server can first perform image rendering based on the parsing result to generate a plurality of corresponding image frames. Then, the operation server can perform video encoding on the plurality of image frames to obtain a corresponding video stream. Correspondingly, in an illustrated embodiment, the operation server can specifically include an image renderer and a video encoder to realize image rendering and encoding of multiple image frames obtained by image rendering into a video stream.

[0079] It should be noted that the main methods of image rendering can include on-screen rendering and off-screen rendering. On-screen rendering refers to the GPU performing rendering operations within the current screen buffer used for display. Off-screen rendering refers to the GPU allocating a new buffer outside the current screen buffer for rendering operations. For example, the computing server may use off-screen rendering for image rendering. Specifically, off-screen rendering may primarily refer to PbufferSurface-based rendering in OpenGL.

[0080] In one illustrated embodiment, when the rendering video stream is returned to the corresponding terminal device by the computing server, the video stream can be returned to the corresponding terminal device through a streaming server between the terminal device and the computing server.

[0081] As mentioned above, the terminal device can first send a connection request to the corresponding streaming server to establish a data transmission channel between the terminal device and the streaming server.

[0082] In one illustrated embodiment, the computing server may also send a connection request to the streaming server to establish a data transmission channel between the computing server and the streaming server.

[0083] Then, the computing server can send the aforementioned video stream to the streaming server based on the established data transmission channel between the computing server and the streaming server. This allows the streaming server to further transmit the video stream to the terminal device based on the established data transmission channel between the terminal device and the streaming server. Correspondingly, the terminal device receives the video stream sent by the computing server through the streaming server based on the established data transmission channel between the terminal device and the streaming server.

[0084] This achieves the goal of providing rendering services to terminal devices through existing CPU-based cloud phone servers and by leveraging existing GPU servers in computing resources, thereby making reasonable use of existing computing resources and reducing the implementation cost of cloud phones.

[0085] Further, please refer to Figure 3 , Figure 3 This is a flowchart illustrating another cloud rendering method provided in an exemplary embodiment. The following will combine... Figure 3 This manual provides a detailed explanation of the cloud rendering method provided, focusing on the interaction between the cloud phone server, computing server, and terminal device. For example... Figure 3 As shown, the method may include the following steps S11-S26.

[0086] Step S11: Register the streaming server in the session server.

[0087] As Figure 3 shown in an embodiment, the whole process of providing rendering service by the cloud phone server to the terminal device can include three stages, namely, a connection establishment stage, a cloud rendering stage and a disconnection stage. Among them, the connection establishment stage is mainly used to establish a data transmission channel between the terminal device and the cloud phone server, the computing server through the streaming server, so as to support the transmission of subsequent interactive data and video stream.

[0088] First, as Figure 3 shown, the streaming server can initiate a registration request to the session server after starting, so as to register the streaming server in the session server, including registering the address of the streaming server. Correspondingly, the session server can return a corresponding registration success feedback, at this time the session server can store the address information of the streaming server.

[0089] Step S12, the computing server requests to establish a data transmission channel with the streaming server.

[0090] As Figure 3 shown, the computing server can send a connection request to the streaming server after starting, so as to establish a data transmission channel between the computing server and the streaming server. Correspondingly, the streaming server can return a corresponding creation success feedback.

[0091] In an embodiment, the computing server can include an encoder, and the data transmission channel is mainly used for the encoder to push the video stream to the corresponding terminal device.

[0092] Step S13, the cloud phone server requests to establish a data transmission channel with the streaming server.

[0093] As Figure 3 shown, the cloud phone server can send a connection request to the streaming server after starting, so as to establish a data transmission channel between the cloud phone server and the streaming server. Correspondingly, the streaming server can return a corresponding creation success feedback.

[0094] In an embodiment, the cloud phone server can include a plurality of cloud phones corresponding to a plurality of terminal devices one by one, and the plurality of cloud phones can send connection requests to the streaming server after starting, so as to respectively establish data transmission channels between the plurality of cloud phones and the streaming server. Subsequently, the plurality of cloud phones can respectively receive the interactive data uploaded by the terminal devices corresponding thereto through the respective established data transmission channels.

[0095] In an embodiment, the cloud phone server can also load a soft rendering (swiftshader) after starting, so as to ensure the smooth work of the cloud phone system.

[0096] Step S14, the user logs in the session server through the terminal device to obtain the streaming server connection information.

[0097] As shown in Figure 3 , the user logs in the session server through the terminal device based on the user account and password and other information to obtain the streaming server connection information returned by the session server. In an embodiment, the streaming server connection information can include the streaming server address.

[0098] Step S15, the terminal device requests to establish a data transmission channel with the streaming server.

[0099] As shown in Figure 3 , the terminal device sends a connection request to the corresponding streaming server based on the streaming server address obtained above to establish a data transmission channel between the terminal device and the streaming server. Correspondingly, the streaming server can return a corresponding creation success feedback.

[0100] So far, the connection establishment phase before formally starting the cloud rendering is completed. As described above, the present specification divides the communication channel between the terminal device and the cloud phone into two channels, including a data transmission channel for transmitting interactive data to the cloud phone and a data transmission channel for video stream transmission with the operation server. The two channels are uniformly provided by the streaming server to the terminal device for interactive access, so that the terminal device can normally access and use the cloud phone in any network environment, thereby improving the use reliability of the cloud phone.

[0101] In an embodiment, further, after the terminal device establishes the data transmission channel with the streaming server, the terminal device can notify the corresponding cloud phone in the cloud phone server to be unlocked through the streaming server, and start to start the cloud rendering service, and notify the renderer (for example, including the off-screen renderer) in the operation server to start the rendering work, and the encoder to start the video encoding and video streaming.

[0102] Step S16, the terminal device sends interactive data to the cloud phone server through the streaming server.

[0103] As shown in Figure 3As shown, the terminal device can send the above-mentioned interaction data to the streaming server based on the established data transmission channel between the terminal device and the streaming server, so that the streaming server further sends the interaction data to the cloud mobile server based on the established data transmission channel between the cloud mobile server and the streaming server. Correspondingly, the cloud mobile server receives the interaction data sent by the terminal device through the streaming server based on the established data transmission channel between the cloud mobile server and the streaming server. In an embodiment, further, the cloud mobile server can return the corresponding interaction feedback to the terminal device through the streaming server after receiving the interaction data, and the like, which are not limited in the present specification.

[0104] In an embodiment, the terminal device and the streaming server can perform data transmission through WebRTC real-time communication technology, and the streaming server and the cloud mobile server can perform data transmission through Transmission Control Protocol (TCP). In an embodiment, the streaming server and the cloud mobile server can also perform data transmission through other communication protocols other than TCP, which are not limited in the present specification.

[0105] In an embodiment, when the cloud mobile server includes multiple cloud mobiles, the terminal device can send the interaction data to the corresponding cloud mobile among the multiple cloud mobiles included in the cloud mobile server through the streaming server.

[0106] Step S17, the cloud mobile server generates corresponding rendering instructions based on the interaction data.

[0107] As shown, the cloud mobile server can generate corresponding rendering instructions based on the interaction data uploaded by the terminal device. In an embodiment, the rendering instructions can be EGL / OpenGL instructions. Figure 3

[0108] In an embodiment, before the terminal device sends the interaction data, the cloud mobile server can first load the cloud rendering EGL dynamic library, and then the cloud mobile server generates corresponding rendering instructions based on the received interaction data and the EGL dynamic library, and the like, which are not limited in the present specification.

[0109] Step S18, the cloud mobile server sends the rendering instructions to the computing server.

[0110] As shown, the cloud mobile server can send the rendering instructions to the computing server, so that the computing server executes the rendering instructions to generate the corresponding rendering result, and the like, which are not limited in the present specification. Figure 3 ​As shown, after generating rendering instructions based on the interactive data uploaded by the terminal device, the cloud phone server can send these rendering instructions to the computing server it interfaces with. Correspondingly, the computing server receives the rendering instructions sent by the cloud phone server.

[0111] In one illustrated embodiment, data transmission between the cloud phone server and the computing server can be performed via the TCP protocol. In another illustrated embodiment, data transmission between the cloud phone server and the computing server can also be performed via other communication protocols besides TCP; this specification does not specifically limit this to any particular protocol.

[0112] Step S19: The computing server parses the rendering instructions and performs image rendering based on the parsing results to generate the corresponding video stream.

[0113] like Figure 3 As shown, after receiving the rendering command from the cloud phone server, the computing server can parse the command to obtain the corresponding parsing result, and then perform image rendering based on the parsing result to generate multiple image frames. The computing server can then perform video encoding on these multiple image frames to obtain the corresponding video stream.

[0114] In one illustrated embodiment, the parsing result may include instructions for image rendering supported by the computing server. In other words, the computing server can convert the received Android image rendering instructions generated by the cloud phone server (CPU server under the Android system) into rendering instructions suitable for the GPU server, such as off-screen rendering instructions supported by the GPU server under the Linux system.

[0115] Thus, this application can separate (or decouple) the CPU and GPU computing of Android cloud phones, thereby utilizing existing GPU servers for GPU computing (including image rendering). This allows for more efficient use of existing CPU and GPU servers, improving resource utilization to a certain extent, especially in large-scale cloud deployments where resource utilization can be significantly enhanced. Furthermore, since this application can translate Android image rendering instructions into off-screen rendering instructions under Linux, and Linux GPU drivers are already more complete than Android GPU drivers in the server market, this application is less constrained by GPU hardware limitations when implementing image rendering, making it highly practical.

[0116] In step S20, the computing server sends a video stream to the terminal device through the streaming server.

[0117] like Figure 3As shown, the operation server can send the video stream to the streaming server based on the established data transmission channel between the operation server and the streaming server, so that the streaming server further sends the video stream to the terminal device based on the established data transmission channel between the terminal device and the streaming server. Correspondingly, the terminal device can receive the video stream sent by the operation server through the streaming server based on the established data transmission channel between the terminal device and the streaming server.

[0118] In an illustrated embodiment, the operation server and the streaming server can perform data transmission through the TCP protocol. In an illustrated embodiment, the operation server and the streaming server can also perform data transmission through other communication protocols in addition to the TCP protocol, which is not limited in the specification.

[0119] As described above, after the operation server in the present application performs rendering and encoding to obtain the video stream, the operation server can directly push the video stream to the terminal device through the streaming server without going through the cloud phone server, thereby greatly improving the video push stream efficiency, reducing the picture delay on the terminal device, and ensuring the user experience.

[0120] Step S21, the terminal device displays based on the received video stream.

[0121] As Figure 3 shown, after the terminal device receives the video stream sent by the operation server through the streaming server, the terminal device can display based on the video stream. In an illustrated embodiment, the terminal device can decode the video stream, and display based on the decoding result through the display screen or projector and other display devices set on the terminal device.

[0122] So far, the present application can push the video stream obtained after rendering by the cloud phone server according to the real-time operation of the user on the terminal device, so that the terminal device can also realize clear and smooth picture display under real-time operation under the premise of lightening, for example, including high-definition smooth game picture display.

[0123] In an illustrated embodiment, the disconnection stage can include Figure 1 steps S22-S26 as shown. As Figure 4As shown, when the user leaves, such as the user performs an exit game, or returns to the game login interface, etc. Operation, the terminal device can disconnect the data transmission channel established between it and the streaming server in response to the operation performed by the user. Further, the streaming server can notify the operation server to stop image rendering and video streaming. Correspondingly, the user account accessing the current cloud phone is exited, and the terminal device held by the user account will no longer be equipped with a cloud phone to provide cloud computing (including cloud rendering service) for it, finally, the terminal device is disconnected with the session server, etc. Here, no longer detailed, the present specification does not make specific limitations.

[0124] In summary, the cloud phone server in the present application can receive the interaction data uploaded by the terminal device corresponding thereto, and generate a series of rendering instructions based on the interaction data. Subsequently, the cloud phone server can forward the rendering instructions to the operation server of the third party connected thereto, so that the operation server generates a video stream to be displayed based on the rendering instructions, and returns the video stream to the terminal device, so that the terminal device displays based on the received video stream. In this way, the present application takes into account the fact that the existing cloud phone server is basically a CPU server, which has poor rendering capability. The image rendering originally performed in the cloud phone server can be transferred to the existing operation server (such as GPU server) with strong rendering capability. Not only the existing computing resources are reasonably utilized, the GPU utilization is improved, but also the hardware requirements for image rendering of the cloud phone are effectively reduced, thereby reducing the implementation cost of the cloud phone, which is conducive to large-scale deployment and application of the cloud phone.

[0125] Corresponding to the above method flow implementation, the embodiments of the present specification also provide a cloud rendering device applied to a cloud phone server corresponding to a terminal device, such as the cloud phone server based on CPU providing computing power as described above. Figure 4 The cloud phone server is used to provide rendering service to the terminal device. Please refer to Figure 4 , Figure 1 is a structural schematic diagram of a cloud rendering device provided by an exemplary embodiment. As Figure 5 shown, the device 30 comprises:

[0126] The receiving unit 301 is configured to receive the interaction data sent by the terminal device, and generate corresponding rendering instructions based on the interaction data;

[0127] The sending unit 302 is configured to send the rendering instructions to the operation server of the third party connected to the cloud phone server, so that the operation server analyzes the rendering instructions, generates corresponding video stream based on the analysis result, and returns the video stream to the terminal device, so that the terminal device displays based on the video stream.

[0128] The rendering capability of the cloud phone server is lower than the rendering capability of the operation server.

[0129] In an illustrated embodiment, the cloud phone server comprises a cloud phone server, and the operation server comprises at least one operation server; wherein the cloud phone server comprises a server providing computing power based on CPU, and the operation server comprises a server providing computing power based on GPU.

[0130] In an illustrated embodiment, the rendering instruction comprises a rendering instruction based on an EGL / OpenGL interface standard.

[0131] In an illustrated embodiment, the interaction data comprises at least one interaction operation input by a user on an application interface provided by the terminal device and received by the terminal device.

[0132] In an illustrated embodiment, the analysis result comprises an instruction supported by the operation server for image rendering; and the manner adopted by the operation server for image rendering comprises off-screen rendering.

[0133] In an illustrated embodiment, the device 30 further comprises:

[0134] The transmission channel establishing unit 303 is configured to send a connection request to a corresponding streaming server to establish a data transmission channel between the cloud phone server and the streaming server.

[0135] The receiving unit 301 is specifically configured to:

[0136] Based on the established data transmission channel between the cloud phone server and the streaming server, the receiving unit 301 receives the interaction data sent by the terminal device through the streaming server.

[0137] In an illustrated embodiment, the cloud phone server comprises a server based on an android operating system, and the operation server comprises a server based on a linux operating system.

[0138] In an illustrated embodiment, the cloud phone server comprises a server based on an ARM architecture, and the operation server comprises a server based on an X86 architecture.

[0139] Further, corresponding to the above method flow implementation, the embodiments of the present specification also provide another cloud rendering device, which is applied to a third-party operation server interfaced with a cloud phone server, for example, comprising the above Figure 5 The server providing computing power based on GPU. Please refer to Figure 5 ,Figures 1-3 FIG. 4 is a structural schematic diagram of another cloud rendering device provided by an example embodiment. As shown in the figure, the device 40 includes: Figure 6

[0140] a receiving unit 401 configured to receive a rendering instruction sent by a cloud phone server; the rendering instruction is generated by the cloud phone server based on received interaction data sent by a terminal device;

[0141] a rendering unit 402 configured to parse the rendering instruction and perform image rendering based on a parsing result to generate a corresponding video stream;

[0142] a sending unit 403 configured to send the video stream to the terminal device to enable the terminal device to perform display based on the video stream; wherein a rendering capability of the cloud phone server is lower than a rendering capability of the operation server.

[0143] In an example embodiment, the cloud phone server includes a cloud phone server, and the operation server includes at least one operation server; wherein the cloud phone server includes a server providing computing power based on a CPU, and the operation server includes a server providing computing power based on a GPU.

[0144] In an example embodiment, the rendering instruction includes a rendering instruction based on an EGL / OpenGL interface standard.

[0145] In an example embodiment, the interaction data includes at least one interaction operation input by a user on an application interface provided by the terminal device and received by the terminal device.

[0146] In an example embodiment, the rendering unit 402 is specifically configured to:

[0147] parse the rendering instruction to obtain a corresponding parsing result; the parsing result includes an instruction supported by the operation server for performing image rendering;

[0148] perform image rendering based on the parsing result to generate a plurality of image frames, and perform video encoding on the plurality of image frames to obtain the corresponding video stream; a manner adopted by the operation server for performing image rendering includes off-screen rendering.

[0149] In an example embodiment, the device 40 further includes:

[0150] a transmission channel establishing unit 404 configured to send a connection request to a corresponding streaming server to establish a data transmission channel between the operation server and the streaming server;

[0151] ​The sending unit 403 is specifically configured to:

[0152] Based on the established data transmission channel between the operation server and the streaming server, the video stream is sent to the streaming server, so that the streaming server further sends the video stream to the terminal device.

[0153] In an illustrated embodiment, the cloud mobile phone server includes a server based on an android operating system, and the operation server includes a server based on a linux operating system.

[0154] In an illustrated embodiment, the cloud mobile phone server includes a server based on ARM architecture, and the operation server includes a server based on X86 architecture.

[0155] The functions and effects of the various units in the above-described apparatus 30 and apparatus 40 are specifically described above Figure 6 The above-described apparatus 30 and apparatus 40 can be implemented by software, or by hardware or a combination of software and hardware. For example, as a logical device, the apparatus is formed by reading corresponding computer program instructions into the memory by the CPU (Central Process Unit, central processor) of the device and running. From the hardware level, in addition to the CPU and the memory, the device where the apparatus is located usually also includes other hardware such as a chip for wireless signal transceiving and / or other hardware such as a board for realizing network communication function.

[0156] The apparatus embodiments described above are only schematic, and the units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical modules, i.e., they can be located in one place or distributed on multiple network modules. Some or all of the units or modules can be selected according to actual needs to achieve the purposes of the schemes of the present specification. Those skilled in the art can understand and implement without creative labor.

[0157] The apparatus, units and modules described in the above embodiments can be specifically implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, and the specific form of the computer can be a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0158] Corresponding to the method embodiments described above, the embodiments of the present specification also provide a computing device. Please refer to Figure 6 , Figure 6 is a structural schematic diagram of a computing device provided by an exemplary embodiment. Figure 6 The computing device 1000 shown can be the cloud phone server or the computing server described above. As shown in the figure, Figures 1-3 The computing device 1000 includes a processor 1001 and a memory 1002, and can further include an input device 1004 (such as a keyboard, etc.) and an output device 1005 (such as a display, etc.). The processor 1001, the memory 1002, the input device 1004 and the output device 1005 can be connected by a bus or other means. As shown in the figure, ​ The memory 1002 includes a computer readable storage medium 1003 that stores computer programs capable of being run by the processor 1001. The processor 1001 can be a general-purpose central processor, a microprocessor, or an integrated circuit for controlling the execution of the above method embodiments. When running the stored computer programs, the processor 1001 can execute the steps of the cloud rendering method in the embodiments of the present specification, including: receiving interaction data sent by a terminal device, and generating corresponding rendering instructions based on the interaction data; sending the rendering instructions to a third-party computing server interfaced with a cloud phone server, so that the computing server parses the rendering instructions, generates a corresponding video stream based on the parsing result, and returns the video stream to the terminal device, so that the terminal device displays based on the video stream; wherein the rendering capability of the cloud phone server is lower than that of the computing server, etc. For detailed description of the steps of the above cloud rendering method, please refer to the previous content, which will not be described here.

[0159] Corresponding to the method embodiments described above, the embodiments of the present specification also provide a computer readable storage medium, which stores computer programs, and when the computer programs are run by a processor, the steps of the cloud rendering method in the embodiments of the present specification are executed. For specific description of the corresponding embodiments, this place will not be described here. ​

[0160] The above is only the preferred embodiment of the present specification, and does not limit the present specification. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present specification shall be included in the protection scope of the present specification.

[0161] In a typical configuration, the terminal device includes one or more processors (CPU), input / output interfaces, network interfaces, and memories.

[0162] ​Memory can include, without being limited to, non-persistent memory, random access memory (RAM), and / or non-volatile memory such as read only memory (ROM) or flash memory. Memory is an example of computer readable media.

[0163] Computer readable media includes permanent and non-permanent, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.

[0164] Examples of storage media include, without being limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that is accessible to a computing device. According to the definition provided herein, computer readable media does not include transitory media, such as modulated data signals and carrier waves.

[0165] It is also noted that the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0166] Those skilled in the art will appreciate that embodiments of the present specification can be devised for use with computer program products comprising computer readable storage media having instructions stored thereon, or used in connection with software programs, that when executed in hardware implement the embodiments described herein. Embodiments of the present specification can be implemented using computer readable media having instructions stored thereon, wherein the instructions are executable by one or more processors to implement the techniques described herein. Computer readable media can include, without being limited to, non-transitory computer storage media and / or communication media. Non-transitory computer storage media can include, without being limited to, volatile and / or non-volatile computer storage media, removable and / or non-removable computer storage media. Non-transitory computer storage media can include, without being limited to, RAM, ROM, electrically erasable programmable read only memory (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information for access by a computing device. Communication media can include, without being limited to, wired media such as a wired network or direct-wired connection, and / or wireless media such as acoustic, RF, infrared, and other wireless media. Combinations of the above can also be included within the scope of computer readable media.

Claims

1. A cloud rendering method applied to a cloud phone server corresponding to a terminal device, wherein, The cloud mobile phone server is configured to provide rendering service to the terminal device; the method comprises: receiving the interaction data sent by the terminal device through the data transmission channel established between the cloud mobile phone server and the streaming server, and generating corresponding rendering instructions based on the interaction data; sending the rendering instructions to the operation server of a third party connected to the cloud mobile phone server, so that the operation server analyzes the rendering instructions, generates corresponding video stream based on the analysis result, and returns the video stream to the terminal device through the data transmission channel established between the streaming server and the terminal device, so that the terminal device receives the video stream through the data transmission channel established between the streaming server and the terminal device, and displays based on the video stream; wherein the rendering capability of the cloud mobile phone server is lower than that of the operation server. 2.The method of claim 1, wherein the operation service end comprises at least one operation server. The cloud mobile phone server comprises a server providing computing power based on CPU, and the operation server comprises a server providing computing power based on GPU.

3. The method of claim 2, wherein the rendering instructions comprise rendering instructions based on the EGL / OpenGL interface standard.

4. The method of claim 2, wherein the interaction data comprises at least one interaction operation input by a user on an application interface provided by the terminal device.

5. The method of claim 2, wherein the analysis result comprises instructions supported by the operation server for image rendering; and the way adopted by the operation server for image rendering comprises off-screen rendering.

6. The method of claim 2, further comprising: sending a connection request to the corresponding streaming server to establish a data transmission channel between the cloud mobile phone server and the streaming server; receiving the interaction data sent by the terminal device through the data transmission channel established between the cloud mobile phone server and the streaming server comprises: receiving the interaction data sent by the terminal device through the streaming server based on the established data transmission channel between the cloud mobile phone server and the streaming server.

7. The method of claim 2, wherein the cloud mobile phone server comprises a server based on the android operating system, and the operation server comprises a server based on the linux operating system.

8. The method of claim 2-6, wherein the cloud mobile phone server comprises a server based on the ARM architecture, and the operation server comprises a server based on the X86 architecture. 9.A cloud rendering method applied to a third-party computing server interfacing with a cloud phone server, wherein, The cloud mobile phone server is configured to provide rendering service to the corresponding terminal device; the method comprises: receiving the rendering instructions sent by the cloud mobile phone server; the rendering instructions are generated by the cloud mobile phone server based on the received interaction data sent by the terminal device, and the interaction data is received by the cloud mobile phone server through the data transmission channel established between the cloud mobile phone server and the streaming server; analyzing the rendering instructions and generating corresponding video stream based on the analysis result; sending the video stream to the terminal device through the data transmission channel established between the terminal device and the streaming server, so that the terminal device receives the video stream through the data transmission channel established between the terminal device and the streaming server, and displays based on the video stream; wherein the rendering capability of the cloud mobile phone server is lower than the rendering capability of the operation server. 10.The method of claim 9, wherein the computing service end comprises at least one computing server; and wherein, The cloud mobile phone server includes a server providing computing power based on CPU, and the operation server includes a server providing computing power based on GPU.

11. The method of claim 10, wherein the rendering instruction comprises a rendering instruction based on an EGL / OpenGL interface standard.

12. The method of claim 10, wherein the interaction data comprises at least one interaction operation input by a user on an application interface provided by the terminal device.

13. The method of claim 10, wherein the parsing the rendering instruction and generating a corresponding video stream based on the parsing result comprises: parsing the rendering instruction to obtain a corresponding parsing result; the parsing result comprises an instruction supported by the operation server for image rendering; generating a plurality of image frames based on the parsing result, and video encoding the plurality of image frames to obtain the corresponding video stream; the operation server adopts an off-screen rendering mode for image rendering.

14. The method of claim 10, further comprising: sending a connection request to a corresponding streaming server to establish a data transmission channel between the operation server and the streaming server; the sending the video stream to the terminal device through the data transmission channel established between the terminal device and the streaming server comprises: sending the video stream to the streaming server based on the established data transmission channel between the operation server and the streaming server, so that the streaming server further sends the video stream to the terminal device.

15. The method of claim 10, wherein the cloud mobile phone server comprises a server based on an android operating system, and the operation server comprises a server based on a linux operating system.

16. The method of any one of claims 10-15, wherein the cloud mobile phone server comprises a server based on an ARM architecture, and the operation server comprises a server based on an X86 architecture. 17.A cloud rendering apparatus applied to a cloud phone server corresponding to a terminal device, wherein, The cloud mobile phone server is configured to provide rendering services to the terminal device; and the apparatus comprises: a receiving unit configured to receive interaction data sent by the terminal device through a data transmission channel established between the terminal device and a streaming server, and generate corresponding rendering instructions based on the interaction data; The sending unit is configured to send the rendering instruction to a third-party computing server connected to the cloud mobile server, so that the computing server analyzes the rendering instruction, generates a corresponding video stream based on the analysis result, and returns the video stream to the terminal device through a data transmission channel established between the streaming server and the terminal device. The rendering capability of the cloud mobile server is lower than that of the computing server.

18. A cloud rendering device applied to a third-party computing server interfacing with a cloud phone server, wherein, The cloud mobile server is configured to provide rendering service to a corresponding terminal device. The receiving unit is configured to receive a rendering instruction sent by the cloud mobile server. The rendering instruction is generated by the cloud mobile server based on received interaction data sent by the terminal device. The interaction data is received by the cloud mobile server through a data transmission channel established between the cloud mobile server and a streaming server. The rendering unit is configured to analyze the rendering instruction and perform image rendering to generate a corresponding video stream based on the analysis result. The sending unit is configured to send the video stream to the terminal device through the data transmission channel established between the streaming server and the terminal device, so that the terminal device receives the video stream through the data transmission channel established between the streaming server and the terminal device and displays the video stream. The rendering capability of the cloud mobile server is lower than that of the computing server.

19. A computing device comprising: A memory and a processor; The memory stores a computer program that can be executed by the processor; The processor executes the computer program to perform the method of any one of claims 1-8 or 9-16.

20. A computer readable storage medium storing a computer program, which, when executed by a processor, implements the method of any one of claims 1-8 or 9-16.

Images