End-cloud computing power collaborative scheduling method, related system and equipment

Abstract

The embodiments of the present application provide a method for collaborative scheduling of end-cloud computing power, related systems and devices. The method can be applied to an end-cloud computing power collaborative scheduling system, which may include a terminal device and a cloud server. Among them, the terminal device can execute a scheduling strategy to decide whether to schedule tasks in the program to be executed on the end side or on the cloud side. In this way, the cloud-side computing power becomes part of the end-side computing power, realizing collaborative scheduling of end-side and cloud-side tasks, solving the problem of insufficient computing power on the end side while avoiding putting all logic and computing power on the cloud side, resulting in idle computing power on the end side and high cloud-side costs.

Description

Technical Field

[0001] The present application relates to the field of electronic technology, and in particular to methods for coordinating end-cloud computing power scheduling, and related systems and equipment. Background Art

[0002] With the widespread practice of artificial intelligence (AI) technology in all walks of life, the deep learning models used by applications on mobile phones, tablets and other terminal devices are becoming more and more complex, resulting in an increase in the requirements of deep learning models for chip computing power. Limited by the process, power consumption, heat generation and cost of chips on the terminal device side, the computing power growth on the terminal device side has slowed down, making it difficult to meet the high computing power requirements of tasks such as AI algorithms and graphics processing unit (GPU) graphics rendering, which has also affected application innovation. Summary of the invention

[0003] The embodiments of the present application provide a method for collaboratively scheduling end-cloud computing power, a communication system, and related equipment, which can realize collaborative scheduling of computing power on the terminal device side and the cloud server side, and flexibly balance the performance and cost of program running.

[0004] In a first aspect, an embodiment of the present application provides a method for scheduling computing power in a terminal-cloud collaborative manner, which can be applied to a terminal device in a first communication system. The first communication system may include: the terminal device and a server, and a first program is installed on the terminal device. The software package of the first program includes a program code, and the program code includes a code for a cloud-side task and a code for an terminal-side task; the code for the cloud-side task includes a first code snippet, and the first code snippet is used to declare that the task type is a cloud-side task; the code for the terminal-side task includes a second code snippet, and the second code snippet is used to declare that the task type is an terminal-side task.

[0005] The method may include: the terminal device determines whether to schedule the to-be-executed task in the first program to be executed on the server according to the scheduling strategy; if it is scheduled to be executed on the server, the terminal device sends the code of the to-be-executed task to the server to request the server to use the cloud-side computing power resources to execute the to-be-executed task; otherwise, the terminal device uses the terminal-side computing power resources to execute the to-be-executed task. The terminal-side computing power resources include the processor resources on the terminal device, and the cloud-side computing power resources include the processor resources on the server. The terminal device receives the execution result of the to-be-executed task returned by the server.

[0006] Among them, the scheduling strategy can be formulated based on but not limited to one or more of the following factors: task type, power consumption of the terminal device, temperature of the terminal device, network status between the terminal device and the server, and cloud-side computing load.

[0007] In the second aspect, an embodiment of the present application provides a method for scheduling computing power in a terminal-cloud collaborative manner, which can be applied to a server in a first communication system. Among them, the first communication system may include: the terminal device and the server, and a first program is installed on the terminal device. The software package of the first program includes a program code, and the program code includes a code for a cloud-side task and a code for an terminal-side task; the code for the cloud-side task includes a first code snippet, and the first code snippet is used to declare that the task type is a cloud-side task; the code for the terminal-side task includes a second code snippet, and the second code snippet is used to declare that the task type is an terminal-side task.

[0008] The method may include: the server receives the code of the task to be executed in the first program sent by the terminal device, and the task to be executed is scheduled to be executed on the server by the terminal device. The server executes the task to be executed according to the cloud computing resources; the cloud computing resources include the processor resources on the server. The server sends the execution result of the task to be executed to the terminal device.

[0009] Implementing the methods of the first and second aspects can realize the coordinated operation of programs by utilizing the computing power resources on the terminal side and the computing power resources on the cloud side, while taking into account both the program operation performance and the cost.

[0010] In the first aspect and the second aspect, the tasks in the first program can be implemented in accordance with the first programming framework. The first programming framework, which is the end-cloud collaborative programming framework mentioned in the subsequent embodiments, may include interfaces for declaring task types: "MobileTask" and "CloudTask". Among them, the interface "MobileTask" can be used to declare that the task type of the task is an end-side task, and the interface "CloudTask" can be used to declare that the task type of the task is a cloud-side task. In addition to end-side tasks and cloud-side tasks, the end-cloud collaborative programming framework may also include an interface for declaring the task type of end-cloud tasks: "MobileCloudTask". Program developers can choose to have tasks inherit MobileTask, CloudTask, or MobileCloudTask according to the actual computing power requirements of each task in the program. Specifically, the end-side tasks in the first program inherit (or implement) the MobileTask interface, the cloud-side tasks inherit (or implement) the CloudTask interface, and the end-cloud tasks inherit (or implement) the MobileCloudTask interface.

[0011] In the software package of the first program, the code of the cloud-side task and the code of the terminal-side task have their own characteristics, namely: the code of the terminal-side task may include a first code snippet, and the code of the cloud-side task may include a second code snippet. Therefore, the terminal device can identify the task type according to the machine code of the task. Among them, the first code snippet may be a piece of machine code compiled from the source code (such as "extends MobileTask") used to declare that the task type is a terminal-side task, and the second code snippet may be a piece of machine code compiled from the source code (such as "extends CloudTask") used to declare that the task type is a cloud-side task.

[0012] In this way, through the end-cloud collaborative programming framework provided by the embodiment of the present application, program developers can achieve computing power coordination between the end side and the cloud side with one development, and can flexibly schedule tasks in the program to the cloud side or the end side for execution as needed based on factors such as the computing power load on the end side, the power consumption on the end side, the temperature control on the end side, the network status, and the computing power load on the cloud side, thereby turning the computing power on the cloud side into part of the computing power on the end side.

[0013] In conjunction with the first aspect, in some embodiments, the program code of the first program may further include the code of the end-cloud task. The code of the end-cloud task may include a third code snippet, which may be a machine code converted from a source code (such as "extends MobileCloudTask") for declaring that the task type is an end-cloud task.

[0014] In combination with the first aspect or the second aspect, in some embodiments, the software package of the first program may further include: a first file, the first file being used to indicate the dependencies between the various tasks in the first program. The first file is the dependency orchestration file mentioned in the subsequent embodiments. The terminal device may extract the first file from the software package of the first program to provide the dependencies between the tasks to the cloud server through the communication proxy service so that the cloud server executes the tasks according to the dependencies. In some embodiments, the terminal device does not need the dependency when executing the task, because the task call on the terminal side is a code-level call, and no additional configuration file is required.

[0015] In combination with the first aspect or the second aspect, in some embodiments, the method of the first aspect further includes: the terminal device sends a first file to the server, so that the server determines the dependency relationship of the task to be executed.

[0016] Specifically, the terminal device 20 can serialize the dependency relationship recorded in the first file into a byte array and put it in the message header of the request message. In this way, after receiving the request message, the cloud server can parse the dependency relationship from the message header. The dependency relationship placed in the message header can be exemplified as follows: taskDAGs:byte[]Codec(in:taskDAGs.json). Among them, "Codec(in:taskDAGs.json)" means encoding the dependency orchestration file taskDAGs.json into a bytecode array.

[0017] In conjunction with the first aspect, in some embodiments, the first file is specifically used to indicate the dependency relationship between the various tasks in the first program by recording the execution order of each task and context data; the context data includes: the input parameters and output parameters of the task. That is, the dependency relationship between tasks can be used to describe: the execution order between tasks, the input parameters and output parameters of each task. Task a's dependency on task b can be reflected in: the execution order of task b is earlier than the execution order of task a, and the input parameters of task a include the output parameters (execution results) of task b.

[0018] In combination with the first aspect or the second aspect, in some embodiments, the scheduling strategy may include but is not limited to one or more of the following:

[0019] 1. Task type strategy

[0020] Implement task scheduling based on task type. That is, choose to schedule end-side tasks to the end-side for execution, and choose to schedule cloud-side tasks to the cloud-side for execution. For end-cloud tasks, you can choose to schedule them to the end-side for execution, or you can choose to schedule them to the cloud-side for execution.

[0021] Different task types reflect different computing power requirements. Program developers generally declare light-load tasks as end-side tasks and heavy-load tasks as cloud-side tasks. Program developers can also declare heavy-load tasks as end-cloud tasks, and light-load tasks as end-cloud tasks.

[0022] The program developer can declare the task type of each task in the first program through the end-cloud collaborative programming framework. In the software package of the first program, the machine code of the cloud-side task and the machine code of the end-side task have their own characteristics, so the terminal device can identify tasks of different task types. The data characteristics of the software package of the first program will be explained later, so I will not expand on them here.

[0023] 2. Whitelist strategy

[0024] Perform whitelist verification, such as verifying whether the model of the terminal device belongs to the preset model, whether the type of the application belongs to the preset type, whether the terminal device is currently in the preset area, etc. If the whitelist verification passes, choose to schedule the task to the cloud side for execution, otherwise choose to schedule the task to the end side for execution. In some embodiments, only the end whitelist policy can be applied to the scheduling of cloud-side tasks and end-cloud tasks. End-side tasks can be scheduled to the end side for execution based on the task type policy.

[0025] 3. Client-side power consumption strategy

[0026] Considering the power consumption of the terminal device, if the remaining power of the terminal device is lower than the first power value, such as 30%, the end-cloud task is scheduled to the cloud side for execution to save the end-side power consumption. In some embodiments, only the end-side power consumption policy can be applied to the scheduling of end-cloud tasks. The scheduling of cloud-side tasks and end-side tasks can be considered based on task type policies, etc.

[0027] 4. Device-side temperature control strategy

[0028] Considering the temperature of the terminal device, if the temperature of the terminal device is higher than the first temperature, such as 38°C, the task is scheduled to be executed on the cloud side to alleviate the problem of excessive temperature on the terminal side. In some embodiments, only the terminal side temperature control policy can be applied to the scheduling of terminal cloud tasks. The scheduling of cloud side tasks and terminal side tasks can be considered based on task type policies, etc.

[0029] 5. Network Status Strategy

[0030] Considering the communication quality between the terminal device and the cloud server, if the communication quality is high, the task is scheduled to the cloud side for execution, otherwise the task is scheduled to the end side for execution. Among them, the communication quality may include but is not limited to: bandwidth, delay, etc. The standard of high communication quality may include but is not limited to: the communication bandwidth is greater than the first bandwidth (such as 200Mbps), and the communication delay is lower than the first delay (such as 1 millisecond). In some embodiments, the network status policy can be applied only to the scheduling of cloud-side tasks and end-cloud tasks. End-side tasks can be scheduled to the end side for execution based on the task type policy.

[0031] 6. Cloud-side computing load strategy

[0032] Considering the computing power load on the cloud side, if the computing power load on the cloud side is lower than the first ratio, such as 60%, the task is scheduled to be executed on the cloud side, otherwise the task is scheduled to be executed on the end side. Cloud side computing power load = cloud side computing power resource usage / total cloud side computing power resources. In some embodiments, only the network status policy can be applied to the scheduling of cloud side tasks and end cloud tasks. End side tasks can be scheduled to be executed on the end side based on the task type policy.

[0033] 7. Dynamic Factor Strategies

[0034] For task scheduling, the available bandwidth between the terminal device and the cloud server is dynamically detected, and further consideration is given to whether the task transmission time meets the task latency requirement, such as the latency requirement indicated by the parameter deadline in the submit() method. If the latency requirement is met, the task is scheduled to the cloud side for execution, otherwise it is scheduled to the client side for execution.

[0035] The transmission time of a task can be estimated based on the data packet size of the task and the real-time available bandwidth. The data packet size of a task mainly depends on the size of the machine code of the task. In some embodiments, the dynamic factor strategy can be applied only to the scheduling of cloud-side tasks and end-cloud tasks. End-side tasks can be scheduled to the end-side for execution based on the task type strategy.

[0036] By formulating a scheduling strategy, we can consider factors such as the computing power requirements of each task, end-side power consumption, end-side temperature control, network status, and cloud-side computing power load to implement task scheduling, so that task scheduling can be more flexible to adapt to different scenarios.

[0037] In addition to the above scheduling strategies, program developers can also customize scheduling strategies according to the business characteristics of the program. For example, judging the location of the user number, if the user is overseas, the task is determined not to be executed on the cloud side.

[0038] In combination with the first aspect or the second aspect, in some embodiments, the software package of the first program may further include: a second file, the second file is used to record the scheduling strategy. The second file is the scheduling strategy file mentioned in the subsequent embodiments. When the terminal device runs the application, the scheduling strategy in the file can be read, and the task scheduling can be performed according to the scheduling strategy. The file can be packaged in the software package of the application. The terminal device can obtain the file after downloading the software package of the application.

[0039] In combination with the first aspect or the second aspect, in some embodiments, the terminal device sends the code of the task to be executed to the server, specifically including: the terminal device sends a first request message to the server, the message header of the first request message carries a first file serialized into a byte array, and the message body of the first request message carries the code of the task to be executed. The machine code of the task to be executed can be organized into a binary file, such as a .so file. The message body can be a byte array, which can be used to store a file serialized into a byte array (such as a .so file).

[0040] In combination with the first aspect or the second aspect, in some embodiments, the first request message also carries the input parameters of the task to be executed. The input parameters can be specifically stored in the message body of the request message. The input parameters of a task can be the output parameters (such as execution results) of other tasks on which it depends. Assuming that task a is an image rendering task, the input parameter of task a can be a RAW image collected by the camera, which can be the output parameter of the image capture task on which task a depends, which can be encoded into a byte array and stored in the message body of the request message to be sent to the cloud server 30 for rendering processing.

[0041] In combination with the first aspect or the second aspect, in some embodiments, the terminal device may use a fast link establishment method to adjust the transmission rate of the task to be executed, thereby reducing the network transmission delay.

[0042] Specifically, the terminal device sending the code of the task to be executed to the server may specifically include: within the first round-trip time RRT for sending the code of the task to be executed to the server, the terminal device transmits data at a test rate v_test; the terminal device receives an acknowledgment ACK returned by the server for the first round of data sent; based on the reception time of the ACK, the terminal device adjusts the transmission rate of the code for sending the task to be executed to the starting rate v_start, v_start = m / △t, where m represents the amount of data sent in the first round; △t = t2-t1, where t1 represents the starting transmission time of the first round of data sending, and t2 represents the reception time of the ACK for the first round of data sending; wherein the test rate v_test is greater than the starting rate v_start.

[0043] In combination with the first aspect or the second aspect, in some embodiments, when the terminal device sends the code of the task to be executed to the server, the terminal device can also execute the congestion control method of RTT granularity, which can resist the noise interference of the weak network of the wide area network and estimate the available bandwidth more accurately. Specifically, the method may include: the terminal device obtains one or more data of delay information, packet loss rate, and receiving rate, and analyzes the data to identify the noise intensity of the weak network signal; including: the noise intensity of RTT jitter and random packet loss; if the noise intensity is greater than the first threshold, the terminal device corrects the abnormal data causing the noise; the terminal device determines the available bandwidth based on the corrected delay information, packet loss rate, and one or more data of the receiving rate. In this way, the noise interference of the weak network of the wide area network can be resisted, and the congestion control module with traffic feature perception is introduced to adjust the rate in time according to the traffic characteristics of the application to ensure transmission efficiency.

[0044] In combination with the first aspect or the second aspect, in some embodiments, when the terminal device sends the code of the task to be executed to the server, the terminal device can also perform congestion control based on traffic characteristics, and can adjust the rate in time according to the traffic characteristics of the application program to ensure transmission efficiency. Specifically, the method may include: the terminal device identifies the traffic trough period during the operation of the first program, and upwardly detects the network bandwidth during the traffic trough period and obtains the upward detection bandwidth; the traffic trough period refers to the stage when the terminal device transmits the code of the first task to the server, and the data volume of the code of the first task is greater than the first data volume; during the traffic peak period, the terminal device adjusts the transmission rate according to the upward detection bandwidth and the historical maximum available bandwidth, and the adjusted transmission rate is the smaller of the upward detection bandwidth and the historical maximum available bandwidth; the traffic peak period refers to the stage when the terminal device transmits the code of the second task to the server, and the data volume of the code of the second task is less than the second data volume; the second data volume is less than or equal to the first data volume.

[0045] In a third aspect, the present application provides a terminal device, comprising a processor and a memory; wherein the memory is coupled to the processor, the memory is used to store computer program code, the computer program code comprises computer instructions, and when the processor executes the computer instructions, the terminal device executes the method described in the first aspect and any possible implementation method of the first aspect.

[0046] In a fourth aspect, the present application provides a server comprising a processor and a memory; wherein the memory is coupled to the processor, the memory is used to store computer program code, the computer program code comprises computer instructions, and when the processor executes the computer instructions, the server executes the method described in the second aspect and any possible implementation manner of the second aspect.

[0047] In the fifth aspect, an embodiment of the present application provides a chip system, which is applied to a terminal device, and the chip system includes one or more processors, which are used to call computer instructions so that the terminal device executes the method described in the first aspect and any possible implementation method of the first aspect.

[0048] In the sixth aspect, an embodiment of the present application provides a chip system, which is applied to a server, and the chip system includes one or more processors, which are used to call computer instructions to enable the server to execute the method described in the second aspect and any possible implementation method of the second aspect.

[0049] In a seventh aspect, the present application provides a computer-readable storage medium comprising instructions, which, when executed on a terminal device, enables the terminal device to execute the method described in the first aspect and any possible implementation manner of the first aspect.

[0050] In an eighth aspect, the present application provides a computer program product comprising instructions, which, when executed on a server, enables the server to execute the method described in the second aspect and any possible implementation of the second aspect.

[0051] In a ninth aspect, the present application provides a communication system which may include a terminal device and a server; wherein the terminal device may be a terminal device in the first aspect and any possible implementation of the first aspect, and the server may be a server in the second aspect and any possible implementation of the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the background technology, the drawings required for use in the embodiments of the present application or the background technology will be described below.

[0053] Figure 1 The following is a workflow of a photo taking application;

[0054] Figure 2 A workflow of cloud gaming is shown;

[0055] Figure 3 An exemplary embodiment of the present invention shows a coordinated scheduling of end-cloud computing power.

[0056] Figure 4A-4C Several task scheduling scenarios are exemplified;

[0057] Figure 5 The end-cloud computing power collaborative scheduling system provided by the embodiment of the present application is exemplified;

[0058] Figure 6 It shows the different states that a task in a program may be in during its life cycle.

[0059] Figure 7 The example shows that the dependency relationship is graphically recorded in a configuration file in a YAML\XML format;

[0060] Figure 8 The functions of the access module in the cloud server are exemplified;

[0061] Fig. 9 The functions of the task scheduling execution module in the cloud server are exemplified;

[0062] Fig.10 The end-cloud collaborative programming framework provided by the embodiment of the present application is shown;

[0063] Fig.11 The overall process of the end-cloud computing power collaborative scheduling method provided by the embodiment of the present application is exemplified;

[0064] Fig.12 The function of the communication proxy service in the terminal device is shown;

[0065] Fig.13 The functions implemented by the communication proxy service and the scheduling strategy service in the terminal device are shown;

[0066] Fig.14 It shows a high-performance and reliable protocol transmission framework implemented on the terminal device side;

[0067] Fig.15 The process of the fast link establishment method provided in the embodiment of the present application is exemplified;

[0068] Fig.16 Existing congestion control methods are shown;

[0069] Fig.17 The architecture for implementing congestion control provided by the embodiment of the present application is exemplified;

[0070] Fig.18 The process of the congestion control method provided by the embodiment of the present application is exemplified;

[0071] Fig.19 The terminal device provided by the embodiment of the present application is shown;

[0072] Fig. 20 The server provided by the embodiment of the present application is shown. DETAILED DESCRIPTION

[0073] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application.

[0074] In order to meet the high computing power requirements of tasks such as AI algorithms and graphics processing unit (GPU) graphics rendering, the industry mainly has two solutions to solve this problem.

[0075] One solution is to optimize the AI ​​model used on the terminal device side, trim the AI ​​model to make it as small as possible, and reduce its computing power requirements. However, the problem caused by the simplification of the model is that the AI ​​processing effect is poor and cannot achieve the expected results.

[0076] Take the camera function of a mobile phone as an example. Figure 1As shown in the figure, after the original image (RAW image) is generated by the image sensor and the image signal processor (ISP), the mobile phone will execute some camera AI algorithms to perform noise reduction, beauty, super resolution (SR) and other processing for scenes such as portraits and macros to generate photos with higher image quality. Users can see the images processed by AI through applications such as the gallery. Compared with the original images, the AI-processed images can usually improve the image quality by about 20%. However, since camera performance is an important competitive point for major mobile phone manufacturers, the AI ​​algorithms for camera functions are becoming more and more complex, and the computing power of mobile phones is insufficient, resulting in poor execution of image processing algorithms. For example, there is a problem of image blur in portrait shooting scenes such as telephoto shooting, group photos, and motion capture, which cannot meet user expectations.

[0077] Another solution is to put all the tasks of the application on the terminal device side on the cloud server, and use the high computing power of the cloud server to make up for the lack of computing power on the terminal device side. For example, all the computing tasks of cloud games are put on the cloud server, and the terminal device in the user's hand is only used as an input and output device to receive user input, provide display, output sound, etc. However, this solution requires the application provider to pay high cloud server usage fees, and also causes the computing power on the terminal device side to be idle.

[0078] For example, Figure 2 As shown in the figure, all the computing tasks of cloud games are executed on the cloud server, and terminal devices such as mobile phone 1 and mobile phone 2 are only used to receive user input and display game video streams. By using the computing power of the cloud server, the running effect of cloud games can far exceed that of mobile phones, but cloud computing resources need to be allocated on the cloud server for each game opened on the mobile phone. In addition, the computing power cost of the cloud server side is high, and the cloud game provider needs to pay high cloud server usage fees, which also hinders the large-scale commercial use of cloud games.

[0079] The embodiments of the present application provide a device-cloud computing power collaborative scheduling system, a device-cloud computing power collaborative scheduling method, and related equipment, which can realize the collaborative operation of programs by utilizing device-side computing power resources and cloud-side computing power resources, while taking into account both program operation performance and cost.

[0080] Figure 3 An example of a coordinated scheduling of end-cloud computing power implemented in an embodiment of the present application is shown.

[0081] like Figure 3As shown, the application on the terminal device can be divided into multiple tasks. Among them, light-load tasks can be scheduled to the terminal device for execution, using the terminal-side computing resources, such as the central processing unit (CPU), graphics processing unit (GPU), neural processing unit (NPU) and other computing resources on the terminal device. Among them, heavy-load tasks can be scheduled to the cloud server for execution, using the cloud-side computing resources, such as the central processing unit (CPU), graphics processing unit (GPU), neural processing unit (NPU) and other computing resources on the cloud server.

[0082] Through the coordinated scheduling of end-cloud computing power, the computing power requirements of heavy-loaded tasks can be guaranteed, improving the program running effect, and the end-side computing power resources can be used to run light-loaded tasks, avoiding the entire program running being deployed on the cloud side, which can reduce the program running cost.

[0083] For example, in the field of graphics rendering, GPU rendering tasks are divided into heavy-loaded rendering tasks (such as ray tracing) and light-loaded rendering tasks, and heavy-loaded rendering tasks are scheduled to be executed on the cloud server, and light-loaded rendering tasks are scheduled to be executed on the terminal device. After the cloud server completes the execution of the heavy-loaded rendering task, it returns the image processing results such as video streams / pictures to the terminal device. In this way, the GPU rendering effect can be improved, such as improving the picture effect of ray tracing, without causing idle computing power on the terminal device side, reducing the cloud-side computing power usage cost of graphics rendering.

[0084] For another example, in the field of general computing, AI algorithms are divided into heavy-load algorithms and light-load algorithms, and heavy-load algorithms are scheduled to be executed on cloud servers, while light-load algorithms (such as portrait, food and other photo scene recognition algorithms) are scheduled to be executed on terminal devices. In this way, the operation effect of AI algorithms can be improved without causing idle computing power on the terminal device side, thus reducing the cost of using cloud computing power for AI algorithms.

[0085] In the embodiments of the present application, a heavy-load task refers to a task with high computing power requirements, and its computing power requirements generally need to be guaranteed by cloud-side computing power; a light-load task refers to a task with low computing power requirements, and its computing power requirements can be met by terminal-side computing power.

[0086] In the embodiment of the present application, a task is a functional module of a program. A program can be divided into multiple functional modules through tasks. A task can be a function, or it can be composed of multiple functions that complete the same functional unit. A function consists of a function header and a function body. The function header includes the function name, return value type and parameter list, and the function body includes the statement block executed by the function.

[0087] Figure 4A-4C Several task scheduling scenarios are shown as examples. Figure 4A All tasks (task 1 to task 4) in the application shown are light-load tasks. When the computing power resources on the terminal side are sufficient, all tasks in the application can be scheduled to be executed on the terminal device. Figure 4B All tasks (task 1 to task 4) in the application shown are heavy-load tasks, so all tasks in the application can be scheduled to be executed on the cloud server; Figure 4C Some tasks (Task 1, Task 4) in the application shown are light-load tasks, and other tasks (Task 2, Task 3) are heavy-load tasks. The light-load tasks can be scheduled to be executed on the terminal device, and the heavy-load tasks can be scheduled to be executed on the cloud server.

[0088] Figure 5 The end-cloud computing power collaborative scheduling system provided in the embodiment of the present application is exemplified.

[0089] like Figure 5 As shown, the system includes a terminal device 20 and a cloud server 30. Among them, a first program can be installed on the terminal device 20. The first program is developed by the program developer based on the end-cloud collaborative programming framework, which will be introduced later and will not be expanded here. The tasks in the first program may include the following types: cloud-side tasks and end-side tasks. End-side tasks can be tasks scheduled to be executed on the terminal device 20, and cloud-side tasks can be tasks scheduled to be executed on the cloud server 30. These multiple tasks can also include end-cloud tasks, which are tasks that can be scheduled to be executed on both the terminal device 20 and the cloud server 30.

[0090] The terminal device 20 can be used to send cloud-side tasks or end-cloud tasks to the cloud server 30, and request the cloud server 30 to execute the task. Accordingly, the cloud server 30 can respond to the request of the terminal device 20, execute the task, and return the execution result to the terminal device 20, so that the terminal device 20 updates the running status of the first program, such as refreshing the interface, etc. In this way, the cloud-side computing power becomes part of the end-side computing power, realizing the coordinated scheduling of end-side and cloud-side tasks, solving the problem of insufficient end-side computing power while avoiding putting all logic and computing power on the cloud side, resulting in idle end-side computing power and high cloud-side costs.

[0091] In the end-cloud computing power collaborative scheduling system, the functions of the terminal device 20 and the cloud server 30 and the executed processes can also be referred to in the following Fig.11 Related content in the method embodiment.

[0092] like Figure 5 As shown, the terminal device 20 may include: a scheduling module 201 and a task execution engine module 203. The cloud server 30 may include: an access module 301 and a task scheduling execution module 303.

[0093] like Figure 5 As shown in the figure, the key processes in the end-cloud computing power collaborative scheduling system may include:

[0094] 1. The task execution engine module 203 in the terminal device 20 executes the task.

[0095] 2. When the task is running, the scheduling module 201 in the terminal device 20 determines whether the task is executed on the terminal side or the cloud side.

[0096] 3. If the task meets the cloud-side execution conditions, the scheduling module 201 in the terminal device 20 sends the task to the cloud server 30.

[0097] 4. The access module 301 in the cloud server 30 is responsible for receiving tasks from the terminal device 20 and for load balancing the tasks. If the terminal device 20 is accessing for the first time, an operating environment is allocated to the terminal device 20. All subsequent tasks received from the terminal device 20 are routed to the previously allocated operating environment.

[0098] 5. The task scheduling execution module 303 in the cloud server 30 allocates cloud-side resources such as CPU, NPU or GPU according to the computing power requirements of the task and executes the task.

[0099] The following is a detailed description of the various components of the end-cloud computing power collaborative scheduling system.

[0100] Scheduling Module 201

[0101] The scheduling module 201 may include two services: scheduling strategy and communication agent.

[0102] Scheduling Strategy Service

[0103] The scheduling policy service can be used to execute the scheduling policy, and select whether to schedule the task in the first program to be executed on the cloud side or on the terminal side, or to schedule part of the task to be executed on the terminal side and the other part to be executed on the cloud side.

[0104] The scheduling strategy can be formulated based on but not limited to one or more of the following factors: task type, power consumption of the terminal device, temperature of the terminal device, network status between the terminal device and the server, and cloud-side computing load.

[0105] The scheduling policy can be the default scheduling policy or customized by the program developer.

[0106] In the implementation of this application, the scheduling strategy may include but is not limited to one or more of the following:

[0107] 1. Task type strategy

[0108] Implement task scheduling based on task type. That is, choose to schedule end-side tasks to the end-side for execution, and choose to schedule cloud-side tasks to the cloud-side for execution. For end-cloud tasks, you can choose to schedule them to the end-side for execution, or you can choose to schedule them to the cloud-side for execution.

[0109] Different task types reflect different computing power requirements. Program developers generally declare light-load tasks as end-side tasks and heavy-load tasks as cloud-side tasks. Program developers can also declare heavy-load tasks as end-cloud tasks, and light-load tasks as end-cloud tasks.

[0110] The program developer can declare the task type of each task in the first program through the end-cloud collaborative programming framework. In the software package of the first program, the machine code of the cloud-side task and the machine code of the end-side task have their own characteristics, so the terminal device can identify tasks of different task types. The data characteristics of the software package of the first program will be explained later, so I will not expand on them here.

[0111] 2. Whitelist strategy

[0112] Perform whitelist verification, such as verifying whether the model of the terminal device belongs to the preset model, whether the type of the application belongs to the preset type, whether the terminal device is currently in the preset area, etc. If the whitelist verification passes, choose to schedule the task to the cloud side for execution, otherwise choose to schedule the task to the end side for execution. In some embodiments, only the end whitelist policy can be applied to the scheduling of cloud-side tasks and end-cloud tasks. End-side tasks can be scheduled to the end side for execution based on the task type policy.

[0113] 3. Client-side power consumption strategy

[0114] Considering the power consumption of the terminal device, if the remaining power of the terminal device is lower than the first power value, such as 30%, the end-cloud task is scheduled to the cloud side for execution to save the end-side power consumption. In some embodiments, only the end-side power consumption policy can be applied to the scheduling of end-cloud tasks. The scheduling of cloud-side tasks and end-side tasks can be considered based on task type policies, etc.

[0115] 4. Device-side temperature control strategy

[0116] Considering the temperature of the terminal device, if the temperature of the terminal device is higher than the first temperature value, such as 38°C, the task is scheduled to be executed on the cloud side to alleviate the problem of excessive temperature on the terminal side. In some embodiments, only the terminal side temperature control policy can be applied to the scheduling of terminal cloud tasks. The scheduling of cloud side tasks and terminal side tasks can be considered based on task type policies, etc.

[0117] 5. Network Status Strategy

[0118] Considering the communication quality between the terminal device and the cloud server, if the communication quality is high, the task is scheduled to the cloud side for execution, otherwise the task is scheduled to the end side for execution. Among them, the communication quality may include but is not limited to: bandwidth, delay, etc. The standard of high communication quality may include but is not limited to: the communication bandwidth is greater than the first bandwidth (such as 200Mbps), and the communication delay is lower than the first delay (such as 1 millisecond). In some embodiments, the network status policy can be applied only to the scheduling of cloud-side tasks and end-cloud tasks. End-side tasks can be scheduled to the end side for execution based on the task type policy.

[0119] 6. Cloud-side computing load strategy

[0120] Consider the cloud-side computing load. If the cloud-side computing load is lower than a certain percentage, such as 60%, the task is scheduled to the cloud side for execution. Otherwise, the task is scheduled to the end-side for execution. Cloud-side computing load = cloud-side computing resource usage / total cloud-side computing resources. In some embodiments, only the network status policy can be applied to the scheduling of cloud-side tasks and end-cloud tasks. End-side tasks can be scheduled to the end-side for execution based on the task type policy.

[0121] 7. Dynamic Factor Strategies

[0122] For task scheduling, the available bandwidth between the terminal device and the cloud server is dynamically detected, and further consideration is given to whether the task transmission time meets the task latency requirement, such as the latency requirement indicated by the parameter deadline in the submit() method. If the latency requirement is met, the task is scheduled to the cloud side for execution, otherwise it is scheduled to the client side for execution.

[0123] The transmission time of a task can be estimated based on the data packet size of the task and the real-time available bandwidth. The data packet size of a task mainly depends on the size of the machine code of the task. In some embodiments, the dynamic factor strategy can be applied only to the scheduling of cloud-side tasks and end-cloud tasks. End-side tasks can be scheduled to the end-side for execution based on the task type strategy.

[0124] By formulating a scheduling strategy, we can consider factors such as the computing power requirements of each task, end-side power consumption, end-side temperature control, network status, and cloud-side computing power load to implement task scheduling, so that task scheduling can be more flexible to adapt to different scenarios.

[0125] In addition to the above scheduling strategies, program developers can also customize scheduling strategies according to the business characteristics of the program. For example, judging the location of the user number, if the user is overseas, the task is determined not to be executed on the cloud side.

[0126] The scheduling policy may be recorded in a scheduling policy file, and the scheduling policy file may be packaged in the software package of the first program. The terminal device may obtain the scheduling policy file after downloading the software package of the first program.

[0127] For example, a scheduling policy file (such as taskDispatchPolicy.json) can record the scheduling policy as follows:

[0128]

[0129]

[0130] In the taskDispatchPolicy.json example above, the array policy[] records the numbers of the two scheduling policies "temperaturePolicy" and "userDefinedPolicy" (see the assignment of the "NO." parameter), key parameter values ​​(see the assignment of the "value" parameter), and scheduling targets (see the assignment of the "run" parameter).

[0131] The scheduling policy "temperaturePolicy" can be the terminal-side temperature control policy mentioned above. Its key parameter value is "38+" and the scheduling target is "cloud", which specifically indicates the following scheduling policy: when the terminal-side temperature is above 38°C, the task is scheduled to the cloud side for execution to alleviate the problem of excessively high terminal-side temperature.

[0132] Among them, the scheduling policy "userDefinedPolicy" can be the custom policy of the program developer mentioned above. Its key parameters are the business policy "appPolicy" customized by the program developer to determine whether to schedule the task to the cloud side for execution, and the scheduling target is "cloud", where "appPolicy" can be a string describing the custom business policy. This can specifically represent the following scheduling policy: determine whether to schedule the task to the cloud side for execution based on the custom "appPolicy".

[0133] Communication agent service

[0134] The communication proxy service can be used to establish a communication connection between the terminal device 20 and the cloud server 30, providing high-performance, low-latency communication services for scheduling tasks to be executed on the cloud side.

[0135] When the scheduling policy service determines to schedule the cloud-side task in the first program to the cloud-side for execution, the communication proxy service can send the machine code of the cloud-side task to the cloud server 30 through the communication connection and request the cloud server 30 to execute the task. The communication proxy service can also be used to receive the task execution result returned by the cloud server 30 through the communication connection, such as rendered image data.

[0136] Compared with traditional hypertext transfer protocol (HTTP) communication, the communication proxy service is optimized in the following aspects:

[0137] 1. Protocol optimization: Some new fields are added to the protocol header of the Computer As A Service (Caas) protocol. Specifically, they are:

[0138] The following fields can be added to the message header of the request message based on the Caas protocol:

[0139] Field "ProjectId": program development project number, which can correspond to the cloud-side resource number and is used for cloud-side tenant isolation.

[0140] Field "AppId": program number.

[0141] Field "AppVersion": program version number.

[0142] Field "DeviceId": the device ID of the terminal device, which can be a universally unique identifier (UUID).

[0143] Field "TaskDAGs": string array describing the dependencies between tasks.

[0144] The request message may be sent by the terminal device 20 to the cloud server 30 to request the cloud server 30 to perform a task. The message body of the request message may carry the machine code of the task. For example, the message body CaasMsgBody of the request message of the Caas protocol may include a byte array, and the machine code of the task may be serialized and stored in the byte array.

[0145] The following fields can be added to the message header of the response message based on the Caas protocol:

[0146] Field "TaskDAGs": string array describing the dependencies between tasks.

[0147] Field "ErrorCode": error code, a specific value (such as 0) can indicate success.

[0148] Field "ErrorMsg": error description information.

[0149] The response message may be sent by the cloud server 30 to the terminal device 20 to feedback the execution result of the task to the terminal device 20. The message body of the response message may carry the execution result of the task. For example, the message body CaasMsgBody of the response message of the Caas protocol may include a byte array, and the execution result of the task may be serialized and stored in the byte array.

[0150] In the message header of the response message, the field "TaskDAGs" is optional.

[0151] 2. The communication proxy service can detect the available bandwidth between the terminal device 20 and the cloud server 30, and provide the real-time detected available bandwidth to the scheduling strategy service as a judgment condition for whether to schedule the cloud-side service to be executed on the cloud server 30, such as the "dynamic factor strategy" mentioned above.

[0152] 3. The communication proxy service can detect the cloud-side computing load and provide the real-time detected cloud-side computing load to the scheduling strategy service as a judgment condition for whether to schedule the cloud-side service to the cloud server 30 for execution, such as the "cloud-side computing load strategy" mentioned above.

[0153] In the specific implementation, the communication proxy service can send a heartbeat message to the cloud server 30 every period T, and receive information returned by the cloud server 30 such as the current occupancy of cloud-side computing resources such as NPU and GPU, the execution time of cloud-side tasks in the previous period, etc., so as to estimate the cloud-side computing load based on this information and assist in deciding whether the cloud-side tasks in the next period can be scheduled to the cloud side for execution.

[0154] Task execution engine module 203

[0155] The task execution engine module 203 can be used to execute the terminal-side task using the terminal-side computing resources. When the scheduling policy service determines to schedule the cloud-side task or the terminal-cloud task in the first program to the terminal-side for execution, the terminal-side computing resources can also be used to execute the cloud-side task or the terminal-cloud task.

[0156] The task execution engine module 203 can also be used to manage the life cycle of tasks when running the tasks in the first program, such as creating tasks, establishing task queues, and managing task status. Figure 6As shown in the figure, a task can include the following states in its life cycle: pending, ready, executing, finished, and blocked. After a task is created, it enters the waiting state, and enters the ready state after the running conditions are met. Tasks in the ready state can be scheduled for execution on the client side or the cloud side, tasks in execution are in the executing state, tasks can be suspended during execution and are in the blocked state, and tasks are in the finished state after they are completed.

[0157] The task execution engine module 203 can also be used to extract the dependency arrangement file from the software package of the first program, so as to provide the dependency between the tasks to the cloud server 30 through the communication proxy service, so that the cloud server 30 executes the task according to the dependency. In some embodiments, the terminal device 20 does not need the dependency when executing the task, because the task call on the terminal side is a code-level call, and no additional configuration file is required.

[0158] In the embodiments of the present application, the dependency relationship between tasks can be used to describe: the execution order between tasks, the input parameters and output parameters of each task (also known as the context data of the task). The dependency of task a on task b can be reflected in: the execution order of task b is earlier than the execution order of task a, and the input parameters of task a include the output parameters (execution results) of task b.

[0159] In one implementation, the dependency orchestration file (such as taskDAGs.json) can record the dependencies between tasks as follows:

[0160]

[0161] In the taskDAGs.json of the above example, the array DAGs[] records the task types (see the assignment of the "type" parameter), execution order (see the assignment of the "seq" parameter), input parameters (see the assignment of the "in" parameter), and output parameters (see the assignment of the "out" parameter) of the three tasks "xxxMobileTask", "yyyCloudTask", and "zzzMobileCloudTask".

[0162] Among them, the execution order of tasks, input parameters and output parameters of tasks can be used to determine the dependencies between tasks.

[0163] In the above example, the execution order of "xxxMobileTask" is before that of "yyyCloudTask", and the execution order of "yyyCloudTask" is before that of "zzzMobileCloudTask", and the input parameter of "yyyCloudTask" is the output parameter of "xxxMobileTask". Therefore, it can be determined that "zzzMobileCloudTask" depends on "yyyCloudTask", and "yyyCloudTask" depends on "xxxMobileTask".

[0164] In some embodiments, in order to pass the dependencies between tasks to the cloud server 30, the terminal device 20 can serialize the dependencies recorded in the dependency orchestration file (such as taskDAGs.json) into a byte array and put it in the message header of the request message (such as a task execution request). In this way, after receiving the request message, the cloud server 30 can parse the dependencies from the message header. The dependency placed in the message header can be exemplified as follows: taskDAGs:byte[]Codec(in:taskDAGs.json). Among them, "Codec(in:taskDAGs.json)" means encoding the dependency orchestration file taskDAGs.json into a bytecode array.

[0165] In another implementation, the dependency arrangement file may record the dependency relationships between tasks in a graphical manner. Figure 7 As shown, the dependency relationship can be graphically recorded in a configuration file in a format such as YAML\XML, which can represent: Task 6 depends on Task 5, Task 5 depends on Task 2, Task 3 and Task 4, and Task 2, Task 3 and Task 4 depend on Task 1. One possible scenario for the dependency relationship is: the execution order of Task 1 is before the execution order of Task 2, Task 3 and Task 4, the execution order of Task 2, Task 3 and Task 4 is before the execution order of Task 5, and the execution order of Task 5 is before the execution order of Task 6; and the output parameter of Task 1 is the input parameter of Task 2, Task 3 and Task 4.

[0166] Access module 301

[0167] The access module 301 may be mainly responsible for authenticating the access of the end-side device and conducting session negotiation.

[0168] In addition, the access module 301 may be responsible for receiving tasks, aggregating tasks, and dispatching tasks to the task scheduling and execution module 303 for execution after the dependencies of the tasks are prepared.

[0169] like Figure 8 As shown, the access module 301 may include a task access service and a load balancing service, and may execute the following process:

[0170] The task access service can be used to: authenticate the terminal device 20 when it accesses for the first time; after authentication, allow the terminal device 20 to establish a communication connection with the cloud server, and parse the message header of the request message according to the protocol specification and field definition to obtain the dependency orchestration file, and then determine the dependency between tasks.

[0171] The task access service can receive multiple tasks through multiple paths, where tasks from the same program are transmitted through the same path. For example, tasks from APP1 (appID1) are transmitted through path 1, tasks from APP2 (appID2) are transmitted through path 2, and tasks from APP3 (appID3) are transmitted through path 3. Here, a path can refer to a session link established based on the session mechanism.

[0172] The task access service can aggregate multiple received tasks, convert them into a request message inside the cloud server 30, and dispatch them to the task execution environment through the load balancing service.

[0173] The load balancing service can be used for: When a task of a program is scheduled to the cloud side for the first time, the task scheduling execution module is triggered to allocate an operating environment for the program according to the program development project number (ProjectID) and program number (appID), and cache the mapping relationship between the combination of the development project number and the program number and the operating environment.

[0174] The operating environment may include resources such as servers or virtual machines or Docker containers that run tasks. For example, the operating environment assigned to a task may be an NPU server, a GPU server, or a CPU server.

[0175] Task scheduling execution module 303

[0176] like Fig. 9 As shown, the task scheduling execution module 303 may include two functional modules: task runtime and heterogeneous resource scheduling.

[0177] The task runtime can be used to run tasks, and it can include: the task execution engine on the cloud server 30, the drivers and class libraries corresponding to the cloud-side computing resources (such as GPU, NPU), and the framework of AI algorithms, rendering algorithms, etc. A task runtime can be an on-demand function instance process that can process task execution requests from one or more terminal devices based on the computing load.

[0178] Heterogeneous resource scheduling can be used to schedule the computing resources required for tasks, as well as to achieve rapid resource scheduling and resource recovery based on distributed scheduling, container pooling, container cold start acceleration technology (high-speed snapshot cache recovery), Fourier statistical models, etc. One reference technology implementation is to use serverless (Serverless / FaaS) technology.

[0179] In order to achieve coordinated scheduling of end-cloud computing power, an embodiment of the present application provides an end-cloud collaborative programming framework.

[0180] As described in the background technology section, there is currently a problem of split computing power scheduling on the terminal device side and the cloud server side. It is impossible to coordinate the computing power of the terminal side and the cloud side, and it is impossible to dynamically choose to run the program on the terminal side or the cloud side based on factors such as the load conditions on the cloud side and the terminal side. In this case, program developers often need to develop terminal-side programs and cloud-side programs, and purchase cloud-side computing power resources to deploy cloud-side programs. The development of terminal-side programs needs to consider the terminal-side operating environment, and the development of cloud-side programs needs to consider the cloud-side operating environment. They need to be developed and managed separately, and the development and maintenance costs are high.

[0181] Through the end-cloud collaborative programming framework provided in the embodiments of the present application, program developers can achieve computing power coordination between the end side and the cloud side with just one development, and can flexibly schedule tasks in the program to the cloud side or the end side for execution as needed based on factors such as the computing power load on the end side, the power consumption on the end side, the temperature control on the end side, the network status, and the computing power load on the cloud side, thereby turning the computing power on the cloud side into a part of the computing power on the end side.

[0182] Fig.10 The end-cloud collaborative programming framework provided by an embodiment of the present application is shown.

[0183] Defining Task Types

[0184] like Fig.10 As shown, the end-cloud collaborative programming framework may include interfaces for declaring task types: "MobileTask" and "CloudTask". The interface "MobileTask" can be used to declare that the task type of a task is an end-side task, and the interface "CloudTask" can be used to declare that the task type of a task is a cloud-side task.

[0185] The interfaces "MobileTask" and "CloudTask" can inherit the parent interface (parent class) "DistributedTask". The parent interface "DistributedTask" can contain three methods: the initialization method "init()", the task submission method "submit(long deadline)", and the method "wait()" for waiting for the execution result.

[0186] The interface "MobileTask" can be implemented as follows: abstract MobileTask implementsDistributedTask. It implements the methods of its parent class: init(), submit(long deadline), wait(). The method submit(long deadline) can be used to submit the task to the terminal device side, and deadline can be used to indicate the submission timeout limit on the terminal side.

[0187] The interface "CloudTask" can be implemented as follows: abstract CloudTask implementsDistributedTask. It implements the methods of its parent class: init(), submit(long deadline), wait(). Among them, the method submit(long deadline) can be used to submit tasks to the cloud server side, and deadline is used to indicate the submission timeout limit on the cloud side.

[0188] In addition to end-side tasks and cloud-side tasks, the end-cloud collaborative programming framework can also include an interface for declaring the task type of end-cloud tasks: "MobileCloudTask".

[0189] Program developers can choose to inherit tasks from MobileTask, CloudTask, or MobileCloudTask according to the actual computing power requirements of each task in the program, and implement the corresponding init(), submit(longdeadline), and wait() interfaces.

[0190] For example, the code logic of a terminal-side task "renderAMobileTask" used to implement light-load rendering can be as follows:

[0191]

[0192] In the above example, the task "renderAMobileTask" inherits MobileTask, implements the terminal resource initialization in the init() method, and implements the terminal task waiting mechanism in the wait() method, thereby declaring and defining the task "renderAMobileTask" as a terminal task. The code logic of the task "renderAMobileTask" can also include the code logic of the task itself for implementing the rendering function, such as renderA(). The computing power requirement of renderA() is relatively low, so its task can be designed as a terminal task.

[0193] For example, the code logic of a cloud-side task "reflectionCloudTask" used to implement ray tracing can be as follows:

[0194]

[0195]

[0196] In the above example, the task "reflectionCloudTask" inherits CloudTask, and implements cloud-side resource initialization (including initializing communication between the end and the cloud) in the method init(), and implements the cloud-side task timeout mechanism in the method wait(), thereby declaring and defining the task "reflectionCloudTask" as a cloud-side task. The code logic of the task "reflectionCloudTask" can also include the task's own code logic for implementing ray tracing rendering functions, such as reflection(). The computing power requirement of reflection() is relatively high, so its task can be designed as a cloud-side task.

[0197] The end-cloud collaborative programming framework may also include: a sub-interface (sub-class) "MobileCloudTask" that inherits the parent interface (parent class) "DistributedTask". "MobileCloudTask" defines an interface for tasks that can be executed on the end side or on the cloud side (i.e., end-cloud tasks), which can be implemented as follows: abstract MobileCloudTask implementsDistributedTask. It implements the methods of its parent class: init(), submit(long deadline), wait(), among which the method submit(long deadline) can be specifically used to submit tasks to the cloud server side or the end side, and deadline is used to indicate the submission timeout limit.

[0198] For example, the code logic of a certain end-cloud task "calculateMobileCloudTask" can be as follows:

[0199]

[0200] In the above example, the task "calculateMobileCloudTask" inherits MobileCloudTask, and implements the initialization of end-side resources and cloud-side resources (including initializing communication between the end and the cloud) in the method init(). In the method wait(), the cloud-side or end-side task timeout mechanism is implemented according to whether the actual execution is on the cloud side or the end side, thereby declaring and defining the task "calculateMobileCloudTask" as an end-cloud-side task. The code logic of the task "calculateMobileCloudTask" can also include the code logic of its own functions, such as calculate(). The computing power requirements of calculate() can be met by both end-side computing power resources and cloud-side computing power resources, so its task can be designed as an end-cloud-side task.

[0201] Defining Scheduling Strategies

[0202] In addition to task types, the end-cloud collaborative programming framework can also support program developers to customize scheduling policies. The end-cloud collaborative programming framework can also include: scheduling policy interfaces, such as "TaskDispatchPolicy".

[0203] The parent interface "TaskDispatchPolicy" may include the following methods: intpolicy(stringpolicyExpress) interface. The input parameter "policyExpress" of this method is used to receive a custom scheduling policy. The return result of this method may include "0" and "1", where "0" can be defined as cloud-side execution and "1" can be defined as client-side execution.

[0204] Program developers can customize scheduling policies, such as customizing the appPolicy interface to implement the scheduling policy interface "TaskDispatchPolicy", and implement the method int policy(string policyExpress), where the input parameter "policyExpress" is a customized scheduling policy.

[0205] The scheduling policy can be recorded in a scheduling policy file (such as taskDispatchPolicy.json). When the terminal device runs the application, it can read the scheduling policy in the file and perform task scheduling according to the scheduling policy. The file can be packaged in the software package of the application. The terminal device can obtain the file after downloading the software package of the application.

[0206] Through the end-cloud collaborative programming framework provided in the embodiment of the present application, program developers can use the interface provided by the framework to schedule tasks to the end side or cloud side for execution on demand, achieving the ability to develop once and run on the end side or cloud side at the same time, significantly improving program development efficiency and shielding the differences in the operating environments of the end side and the cloud side.

[0207] Fig.11 The overall process of the end-cloud computing power collaborative scheduling method provided in the embodiment of the present application is exemplified. The method can be applied to the end-cloud computing power collaborative scheduling system including the terminal device 20 and the cloud server 30. Fig.11 As shown, the method may include the following process:

[0208] S11. The terminal device 20 downloads the software package of the first program.

[0209] The first program may be developed by a program developer based on the end-cloud collaborative programming framework. The first program may be a photo-taking application, a game application, etc. The embodiment of the present application does not limit the type of the first program and the functions provided. The first program can be subdivided into multiple tasks. The task types of tasks may include cloud-side tasks and end-side tasks. Among them, end-side tasks may be tasks scheduled to be executed on the terminal device 20, and cloud-side tasks may be tasks scheduled to be executed on the cloud server 30. The task types of tasks may also include end-cloud tasks, which can be scheduled to be executed on the terminal device 20 or on the cloud server 30.

[0210] When writing the first program, the program developer can declare tasks with higher computing power requirements as cloud-side tasks, and declare tasks with lower computing power requirements as terminal-side tasks.

[0211] The software package of the first program may include: program code, and a dependency arrangement file (such as taskDAGs.json).

[0212] The program code of the first program may include the code of the terminal-side task and the code of the cloud-side task. Here, the code may refer to machine code (or bytecode), which is generated after the source code is compiled and can be directly read and executed by the processor with a fast execution speed.

[0213] In the software package of the first program, the code of the cloud-side task and the code of the terminal-side task have their own characteristics: the code of the terminal-side task may include a first code snippet, which may be a piece of machine code converted from a source code (such as "extends MobileTask") used to declare that the task type is a terminal-side task. The code of the cloud-side task may include a second code snippet, which may be a piece of machine code converted from a source code (such as "extendsCloudTask") used to declare that the task type is a cloud-side task. Therefore, the terminal device can identify the task type based on the machine code of the task. The data characteristics of the software package of the first program will be explained later, so they will not be expanded here.

[0214] The program code of the first program may also include the code of the end-cloud task. The code of the end-cloud task may include a third code segment, which may be a machine code converted from a source code (such as "extendsMobileCloudTask") for declaring that the task type is an end-cloud task.

[0215] Among them, the dependency arrangement file can be used to indicate the dependency relationship between various tasks in the first program. For the specific description of this file, please refer to the previous text and will not be repeated here.

[0216] The program code of the first program may also include: a scheduling policy file (such as taskDispatchPolicy.json), which can be used to record the scheduling policy based on which the terminal device performs task scheduling. For the specific description of this file, please refer to the previous text and will not be repeated here. In some embodiments, the scheduling policy may not be recorded in the scheduling policy file, but may be reflected in the code logic of the scheduling module. For example, the scheduling module can implement the following scheduling strategy based on its code logic: perform task scheduling according to the task type, specifically schedule cloud-side tasks to the cloud-side for execution, and schedule end-side tasks to the end-side for execution.

[0217] Downloading is only one way for the terminal device 20 to obtain the first program software package. Without limitation, the terminal device 20 may also obtain the first program software package from another device through communication methods such as Bluetooth, data cable, satellite communication, etc., or may obtain the first program software package by reading an external memory. The embodiment of the present application does not limit the way in which the terminal device 20 obtains the first program software package.

[0218] S12. After obtaining the software package of the first program, the terminal device 20 may parse the software package of the first program to obtain a dependency arrangement file, and install the first program.

[0219] The dependency relationship between tasks recorded in the dependency arrangement file can be sent to the cloud server 30 for the cloud server 30 to execute the task accordingly. The scheduling policy file can be used by the terminal device 20 to determine how to schedule the tasks in the first program. This will be described in detail later and will not be expanded here.

[0220] The first program may also be a program pre-installed in the terminal device 20 when it leaves the factory, in which case S11-S12 are optional steps. The terminal device 20 may obtain the dependency arrangement file and the scheduling strategy file from the installation directory of the first program.

[0221] S13. Before executing the task to be executed in the first program, the terminal device 20 may determine whether the task to be executed is scheduled to be executed on the terminal side or on the cloud side according to the scheduling policy. If it is determined that the task to be executed is scheduled to be executed on the terminal side, the terminal device 20 may execute S14; if it is determined that the task to be executed is scheduled to be executed on the cloud side, the terminal device 20 may execute S15-S18.

[0222] The task to be executed may be a task to be executed triggered by a user operation. For example, for a camera program, the task to be executed triggered by the user operation of pressing the shutter button is a photo capture task, which is used to save the image captured at the moment of the shutter as a photo. The task to be executed may also be a task to be executed triggered by the first program according to its own operation logic.

[0223] The scheduling strategy may include but is not limited to one or more of the following: "task type strategy", "end-side power consumption strategy", "end-side temperature control strategy", "network status strategy", "cloud-side computing load strategy", etc.

[0224] For example, if the scheduling policy is "task type policy", the terminal device 20 can choose to schedule the task to the end side or the cloud side according to the task type of the task. When a task is an end-side task, you can choose to schedule the task to the end side for execution; when a task is a cloud-side task, you can choose to schedule the task to the cloud side for execution. When a task is an end-cloud task, you can choose to schedule the task to the cloud side for execution, or you can choose to schedule the task to the end side for execution.

[0225] For detailed description of the scheduling strategy, please refer to the previous article, which will not be repeated here.

[0226] S14. The terminal device 20 uses the terminal-side computing resources to execute the task to be executed.

[0227] S15. The terminal device 20 sends a request message to the cloud server 30 to request the cloud server 30 to execute the task to be executed. Accordingly, the cloud server 30 may receive the request message sent by the terminal device 20.

[0228] The request message may include: a message header and a message body. The message header may carry a dependency orchestration file, and the message body may carry the machine code of the task to be executed. The dependency orchestration file may be serialized into a byte array and placed in the message header of the request message. The machine code of the task to be executed may be organized into a binary file, such as a .so file. The message body may be a byte array, which may be used to store a file serialized into a byte array (such as a .so file).

[0229] The request message may also carry: input parameters of the task to be executed. The input parameters may be specifically stored in the message body. The input parameters of a task may be output parameters (such as execution results) of other tasks on which it depends. Assuming that task a is an image rendering task, the input parameter of task a may be a RAW image captured by the camera, which may be an output parameter of an image capture task on which task a depends, which may be encoded into a byte array and stored in the message body of the request message to be sent to the cloud server 30 for rendering processing.

[0230] For the description of the message header and message body of the request message, please refer to the previous article and will not be repeated here.

[0231] In some embodiments, during the session between the cloud server 30 and the terminal device 20, the cloud server 30 may save the received dependency orchestration file and the code of the task.

[0232] Thus, during the session, for the first program, if its dependency orchestration file has been transmitted to the cloud server 30, then if the terminal device has another task scheduled to be executed on the cloud side, the dependency orchestration file may no longer be transmitted in the message header of the request message, so as to save network resources. That is, during the session, the dependency orchestration file may be transmitted through the request message only when a task is scheduled to be executed on the cloud side for the first time in the first program.

[0233] Thus, during the session, for a task, if its code has been transmitted to the cloud server 30, then if the terminal device schedules the task to be executed on the cloud side later, the code of the task may no longer be transmitted in the message body of the request message, so as to save network resources. That is, during the session, the code that depends on the task may be transmitted through the request message only when the task is first scheduled to be executed on the cloud side.

[0234] S16. The cloud server 30 may extract the task code from the request message.

[0235] In some embodiments, if the request message carries a dependency orchestration file, the cloud server 30 may also extract the dependency orchestration file from the request message.

[0236] S17. Based on the task code, the cloud server 30 uses cloud-side computing resources to execute the task.

[0237] Specifically, according to the computing power requirements of the task, cloud-side computing power resources such as CPU, NPU or GPU are allocated and the task is executed. The computing power requirements of a task can be determined according to the task type. Specifically, if a task is declared as a terminal-side task, it means that the computing power requirement of the task is low; if a task is declared as a cloud-side task, it means that the computing power requirement of the task is high.

[0238] If the terminal device 20 is accessed for the first time, an operating environment is allocated to the terminal device 20. All subsequent tasks received from the terminal device 20 are routed to the previously allocated operating environment.

[0239] S18. The cloud server 30 returns a response message to the terminal device 20 to feed back the execution result of the task to the terminal device 20. Accordingly, the terminal device 20 can receive the response message sent by the cloud server 30 and extract the execution result of the task from the response message.

[0240] The response message may include a message header and a message body. The message header may carry a dependency orchestration file, and the message body may carry the execution result of the task. The dependency orchestration file may be serialized into a byte array and placed in the message header of the response message. The message body may be a byte array, which may be used to store the execution result of the task serialized into a byte array.

[0241] The execution result of a task can be the input parameter of other tasks that depend on it. Assuming that task a is an image rendering task, the execution result of task a can be a rendered image, which can be encoded into a byte array and stored in the message body of the response message to be fed back to the terminal device 20. The terminal device 20 can use the rendered image as an input parameter of the display task to display the rendered image. The display task depends on task a.

[0242] For the description of the message header and message body of the response message, please refer to the previous article and will not be repeated here.

[0243] In the embodiment of the present application, the cloud server 30 may receive multiple request messages sent by the terminal device 20. These multiple request messages are respectively used to request the cloud server 30 to perform multiple different tasks. After receiving these multiple request messages, the cloud server 30 may restore the execution order of these multiple tasks according to the dependency arrangement file, and cache these multiple tasks in the task queue according to the task execution order described in the dependency arrangement file to record the execution order of these multiple tasks. In this way, when the cloud server 30 wants to run these tasks, it can take out the tasks from the task queue in order and execute them to ensure that the task execution order is correct.

[0244] Through the end-cloud computing power collaborative scheduling method provided in the embodiment of the present application, it is possible to make the cloud-side computing power a part of the end-side computing power, and realize the collaborative scheduling of end-side and cloud-side tasks. While solving the problem of insufficient computing power on the end-side, it also avoids putting all logic and computing power on the cloud side, resulting in idle computing power on the end-side and high cloud-side costs. Moreover, the task scheduling strategy can be customized by the program developer. In addition to the computing power requirements of each task, factors such as end-side power consumption, end-side temperature control, network status, and cloud-side computing power load can also be considered to implement task scheduling, so that task scheduling can be more flexible to adapt to different scenarios.

[0245] The embodiment of the present application also provides a transmission rate adjustment method, which can optimize the communication between the end and the cloud, effectively improve the end-cloud communication bandwidth, and reduce the task transmission delay, so that more tasks that meet the conditions can be scheduled to the cloud side for execution.

[0246] Existing network transmission problems include: the initial rate of the HTTP network transmission protocol is slow to climb, and wide area network transmission faces complex weak network environments (such as random packet loss, RTT jitter, bandwidth fluctuations, etc.). The existing HTTP transmission protocol is prone to large and small fluctuations in available bandwidth assessment when faced with weak network signals (signals indicating poor network conditions), which in turn leads to low transmission efficiency (network bandwidth utilization <40%).

[0247] The communication method provided in the embodiment of the present application can provide optimization effects in the following aspects:

[0248] 1. User experience-aware network QoS scheduling: perceive the differentiated demands of different tasks for accessing data and processing time (deadline delay, task priority, etc.), and accurately deliver the network QoS requirements for experience demand conversion end-to-end.

[0249] 2. Transmission optimization: Fast start-up and accurate bandwidth assessment against weak networks ensure high bandwidth utilization > 80% under WAN communication; adaptive learning of the real-time network environment of the WAN (bandwidth changes, packet loss, jitter, etc.), targeted resistance to various types of network loss interference in the access network, packet loss rate > 3% still ensures bandwidth utilization > 80%; multi-path transmission solves the problem of insufficient bandwidth of a single path or poor network quality, and different tasks take different paths; the transmission protocol obtains the maximum available bandwidth through active detection and reports it to the scheduling module.

[0250] like Fig.12As shown, the communication proxy service on the terminal device may further include: a statistical module, a decision module and a network scheduling module. Among them, the statistical module can be used to create a network statistical model for the task according to the identification information of the task (such as the task ID), and to count the data packet size of the task, the maximum delay of the task, the transmission time of the task (i.e., the transmission time), etc., and determine the available upload network bandwidth K and the network bandwidth utilization rate R allocated to the task in the next cycle based on the historical transmission rate and the current available bandwidth. The decision module can be used to decide which transmission method to use, whether the task should be unpacked, etc. according to information such as the data packet size, the maximum delay of the task, the transmission time of the task (i.e., the transmission time), and inform the network scheduling module of the decision result. The network scheduling module can be used to allocate and adjust the network bandwidth for the task according to the delay requirements of the task, the data packet size of the task, the available network bandwidth, etc.

[0251] Specifically, the transmission rate adjustment provided in the embodiment of the present application may include S21 and S22.

[0252] S21. The communication proxy service provides the scheduling strategy service with cloud-side computing load information, which is used by the scheduling strategy service to evaluate the network status and decide whether to schedule tasks to the cloud side for execution. The specific implementation process is as follows: Fig.13 As shown, it may include S31-S33, wherein:

[0253] S31. The scheduling policy service may send a query request to the statistics module, and the query request may carry the identification information of the task to query the available upload network bandwidth and network bandwidth usage rate allocated to the task in the next cycle.

[0254] S32. The statistics module can determine the available upload network bandwidth K and the network bandwidth utilization rate R allocated to the task in the next cycle based on the historical transmission rate and the current available bandwidth, and return them to the scheduling strategy service.

[0255] S33. According to the data packet size S of the task, the available upload network bandwidth K and the network bandwidth utilization rate R, the scheduling strategy service can calculate the transmission time T = S / (K*R). If the transmission time T is less than the maximum delay of the task, the scheduling strategy service can schedule the task to the cloud side for execution. The maximum delay of the task refers to the maximum delay that the task can tolerate, which reflects the delay requirement of the task.

[0256] S22. provides a high-performance and reliable protocol transmission framework for the task execution engine module on the end side. The specific implementation process is as follows: Fig.14 As shown, it may include S41-S45, wherein:

[0257] S41. The task execution engine module transmits the task cloud-side execution request to the statistics module.

[0258] S42. The statistical module creates a network statistical model for the task according to the identification information of the task (such as the task ID), and can count the data packet size of the task, the maximum delay of the task, the transmission time of the task (ie, the transmission time), etc.

[0259] S43. The statistics module may further transmit a cloud-side execution request for the task to the decision module, and the request may carry information such as the data packet size of the task, the maximum delay of the task, and the transmission time of the task (i.e., the transmission time).

[0260] S44. The decision module can decide which transmission method to use, whether to unpack the task, etc. based on information such as the data packet size, the maximum delay of the task, the transmission time of the task (i.e., the transmission time), and inform the network scheduling module of the decision result.

[0261] S45. The network scheduling module adjusts the transmission rate for the task according to the task's latency requirements, the task's data packet size, the available network bandwidth, etc. The key technologies used may include one or more of the following:

[0262] Fast link building method

[0263] The existing transmission protocol has a low initial rate and needs to go through a slow start process to slowly increase the rate, which is not conducive to reducing network transmission delay. The fast link establishment method provided in the embodiment of the present application can improve this problem. Its core idea includes: the first round trip time (RTT) uses a set high rate (based on business expectations or connection history cache information) as the test rate, and completes the initial rate calculation within 1 RTT based on network feedback.

[0264] Specifically, Fig.15 As shown, the fast link establishment method provided in the embodiment of the present application may include:

[0265] S51. At the beginning of transmitting the task data packet (mainly including the task code and context data) to the cloud server (such as the first RRT of the data packet transmission), the terminal device transmits data at the test rate (selected rate) v_test.

[0266] v_test is a larger transmission rate, and its value is greater than v_start mentioned later.

[0267] In some embodiments, the test rate v_test selected for a task can be determined based on business expectations: v_test = data_size / Tmax, where data_size represents the size of the data packet of the task, and Tmax represents the maximum latency of the task. Specifically, the end-cloud collaborative programming framework can provide an interface, and program developers can configure the latency requirements of each task by implementing the interface.

[0268] In some embodiments, the test rate v_test selected for a task is also determined based on the historical cache information. Here, the historical cache information can be the test rate once selected for the historically transmitted task.

[0269] S52. The terminal device receives the confirmation (acknowledgement, ACK) returned by the cloud server for the first round of data transmission.

[0270] S53. The terminal device estimates the data receiving rate v_recv of the cloud server based on the reception time of the confirmation (ACK), and uses it as the starting rate v_start for the next data transmission.

[0271] Specifically, the data receiving rate v_recv can be estimated as follows: v_recv = m / Δt, where m represents the amount of data sent in the first round, such as m KB, and m is a positive integer; Δt = t2-t1, where t1 represents the start transmission time of the first round of data sent, and t2 represents the reception time of the ACK for the first round of data sent. That is, v_start = m / Δt.

[0272] If the ACK is returned to the terminal device before the first round of data (m KB) is sent, the terminal device can stop sending the remaining part of the first round of data with v_test in advance. In this way, the transmission rate can be adjusted to v_start earlier and adapt to the network faster.

[0273] If the terminal device has not received ACK after the first round of data transmission (m KB) is completed, the terminal device can stop sending subsequent data and wait for ACK to return. In this way, network congestion and packet loss can be avoided.

[0274] S54. The terminal device adjusts the transmission rate to the starting rate v_start.

[0275] The rate adjustment of subsequent data transmission can be performed by a congestion control method. Specifically, the terminal device can use a congestion control algorithm to adjust the transmission rate starting from the second RTT.

[0276] Congestion Control Methods

[0277] Existing congestion control methods can be Fig.16 As shown, based on the delay information, packet loss rate, receiving rate, data packet ACK and other information sent by the network information sampling and processing module, the rate adjustment module driven by network feedback can determine the available bandwidth B to instruct the service rate to be transmitted according to the available bandwidth B.

[0278] However, WAN transmission faces complex weak network environments (such as random packet loss, RTT jitter, bandwidth fluctuations, etc.), weak network signals are prone to occur, and cannot accurately reflect the true status of the network. Fig.16 The existing congestion control method will cause the evaluation of the available bandwidth B to fluctuate greatly and be too small, which in turn leads to the problem of low transmission efficiency.

[0279] The congestion control method provided in the embodiment of the present application can improve this problem. A new weak network identification and information correction module is introduced to resist the interference of weak network noise in the wide area network, and a traffic feature-aware congestion control module is introduced to adjust the rate in time according to the traffic characteristics of the application to ensure transmission efficiency.

[0280] Specifically, Fig.17 and Fig.18 As shown, the fast link establishment method provided in the embodiment of the present application may include:

[0281] Congestion control at RTT granularity (S61-S63)

[0282] S61. The terminal device can obtain delay information, packet loss rate, receiving rate, data packet ACK and other data through the network information sampling and processing module, and can analyze these data through the weak network identification and information correction module to identify the noise intensity mixed with the weak network signal, such as through adaptive dynamic learning of RTT jitter and random packet loss noise intensity. The greater the noise intensity, the less reliable the conclusion of weak network signal is, and correction is needed to make the output of available bandwidth B closer to the actual network status. For example, if only one or two data packets are lost in the continuous transmission of 100 data packets, the noise intensity of the packet loss is large, and it cannot reflect that the network status is poor. For another example, if 100 data packets are transmitted continuously, and only one data packet has an abnormally large RTT, then the RTT jitter cannot reflect that the network status is poor.

[0283] S62. If the noise intensity is greater than the first threshold, the terminal device can correct the abnormal data causing the noise through the weak network identification and information correction module, and the abnormal data is close to the average level after correction.

[0284] S63. The terminal device can determine the available bandwidth B based on the corrected delay information, packet loss rate, receiving rate, data packet ACK and other data through the rate adjustment module driven by network feedback.

[0285] Traffic feature-aware congestion control (S64-S67)

[0286] S64. The terminal device may record the available bandwidth B determined in S63 through a historical bandwidth cache module.

[0287] S65. The terminal device can identify the traffic valley period and traffic peak period during the running of the first program through the traffic feature recognition module.

[0288] Among them, the traffic trough period may refer to the stage when the terminal device transmits data to the cloud server at a low rate, such as when transmitting a task with a small amount of data, and the traffic peak period may refer to the stage when the terminal device transmits data to the cloud server at a high rate, such as when transmitting a task with a large amount of data. In other words, the traffic trough period may refer to the stage when the terminal device transmits the code of the first task to the server, and the data volume of the code of the first task is greater than the first data volume; the traffic peak period refers to the stage when the terminal device transmits the code of the second task to the server, and the data volume of the code of the second task is less than the second data volume; the second data volume is less than or equal to the first data volume.

[0289] In some embodiments, the terminal device can traverse the data volume of each task code in the first program, and determine the transmission period of the task code with a large data volume as the traffic peak period, and determine the transmission period of the task code with a small data volume as the traffic valley period. In other embodiments, the end-cloud collaborative programming framework can provide an interface, and program developers can write the data volume of each task to a specified variable or execution storage area by implementing the interface. In this way, once the terminal device can read the data volume of each task from the specified variable or execution storage area, it can determine which tasks are transmitted during the traffic valley period.

[0290] S66. During the traffic trough period, the terminal device can detect the network bandwidth through the bandwidth fast upward detection module, and the detected network bandwidth is recorded as the upward detection bandwidth.

[0291] Specifically, the terminal device can fill patch data (such as data "0" or other data) in the middle of low-speed data to form high-speed data transmission. If timely feedback is received from the network, the filling is further increased until the network feedback times out. The highest rate that receives timely feedback from the network can be determined as the upward detection bandwidth. This process can also be called the process of "upward detection of network bandwidth".

[0292] S67. During traffic peaks, such as when burst traffic arrives, the terminal device can adjust the transmission rate based on the upward detection bandwidth and the maximum cached available bandwidth B (also known as the historical maximum available bandwidth) through the burst traffic sending rate calculation module. The adjusted transmission rate can be the smaller of the upward detection bandwidth and the maximum cached available bandwidth B.

[0293] Fig.19 A terminal device 200 provided in an embodiment of the present application is shown.

[0294] In the embodiment of the present application, the device type of the terminal device 200 can be a mobile phone, a tablet computer, a handheld computer, a desktop computer, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a cellular phone, a personal digital assistant (PDA), as well as smart home devices such as smart large screens and smart speakers, wearable devices such as smart bracelets, smart watches, and smart glasses, augmented reality (AR), virtual reality (VR), mixed reality (MR) and other extended reality (XR) devices, vehicle-mounted devices or smart city devices, and the like.

[0295] like Fig.19 As shown, the terminal device 200 may include: a processor 110, a memory 120, a display screen 130, a display driver integrated circuit (DDIC), an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, and a subscriber identification module (SIM) card interface 195, etc. Among them, the sensor module 180 may include a gyroscope sensor 180B, an acceleration sensor 180E, and a touch sensor 180K, etc. The various components in the terminal device 200 may be connected through a bus.

[0296] The processor 110 may be one or more, and they may be integrated into an integrated circuit of a system on chip (SOC). SOC is a system-on-chip. The processor 110 may include a central processing unit (CPU), a graphics processing unit (GPU), and a display driver integrated circuit (DDIC). The CPU may be an application processor (AP). The CPU and GPU may be used to render and synthesize images to be sent to the display screen 130. The processor 110 may also include a neural-network processing unit (NPU), a modem processor, and the like.

[0297] The processor 110 may include one or more interfaces, such as an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general purpose input output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0298] The processor 110 may be provided with a cache memory for storing instructions or data just used or cyclically used by the processor 110. If the processor 110 needs to use the instruction or data again, it can be directly called from the cache memory, which can reduce the waiting time of the processor 110 and improve the program running efficiency.

[0299] The memory 120 may include a program storage area and a user data storage area, wherein the program storage area may store an operating system and one or more application programs (such as game applications), and the data storage area may store data (such as photos, contacts) created by the user in the process of using the terminal device 200. The memory 120 may be a high-speed random access memory or a non-volatile memory, such as a disk, a flash memory, a universal flash storage (UFS), etc. The memory 120 may also be an external memory card, such as a Micro SD card.

[0300] The memory 120 may also store code instructions for the display partition refresh method provided in the embodiment of the present application. When the processor 110 reads the code instructions from the memory 120 and runs the code instructions, the terminal device 200 may execute the method.

[0301] The memory 120 may also be integrated with the processor 110 in an integrated circuit of a SOC.

[0302] The terminal device 200 can realize the display function through SOC, DDIC, and display screen 130, etc.

[0303] The display screen 130 may have a partition frequency conversion function, and may be the display screen 130 described in the aforementioned embodiment. The display screen 130 may include a display panel, a timing controller (TCON), etc., wherein the display panel may include a plurality of pixel units, and may be an organic light emitting diode (OLED) display panel, each pixel unit including an OLED. The TCON is mainly used to connect the GPU or SOC with the display panel, and after receiving the image data or control signal from the GPU or SOC, the timing of the control signal and the data signal is set according to the relevant data or signal, and the control signal and the data signal are transmitted to the display driving circuit according to the timing, so as to achieve the purpose of driving the display panel to display graphics.

[0304] Among them, the display driver integrated circuit (DDIC) can be used as the control core of the display screen 130, driving the display screen 130 to work and receiving data from the SOC (processor 110), such as image data and some instructions. The DDIC can send driving signals and data to the display panel of the display screen 130 in the form of electrical signals, and then realize the control of the screen brightness and color, so that image information such as letters and pictures can be displayed on the screen, completing the screen refresh.

[0305] The image data of the screen to be displayed sent by the SOC to the DDIC can be sent to the frame buffer for storage to complete the display (or image sending). Then, the DDIC takes out the image data from the frame buffer and drives the display screen 130 to display.

[0306] The wireless communication function of the terminal device 200 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0307] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in terminal device 200 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0308] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied to the terminal device 200. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0309] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display screen 130. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0310] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the terminal device 200. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0311] In some embodiments, the antenna 1 of the terminal device 200 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the terminal device 200 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0312] The terminal device 200 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 130 and application processor.

[0313] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.

[0314] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the terminal device 200 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0315] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the terminal device 200 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0316] Video codecs are used to compress or decompress digital videos. The terminal device 200 may support one or more video codecs. Thus, the terminal device 200 may play or record videos in multiple coding formats, such as moving picture experts group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0317] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of the terminal device 200 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0318] The terminal device 200 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0319] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

[0320] The speaker 170A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The terminal device 200 can listen to music or listen to a hands-free call through the speaker 170A.

[0321] The receiver 170B, also called a "handset", is used to convert audio electrical signals into sound signals. When the terminal device 200 receives a call or voice message, the voice can be received by placing the receiver 170B close to the ear.

[0322] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to the microphone 170C to input the sound signal into the microphone 170C. The terminal device 200 can be provided with at least one microphone 170C. In other embodiments, the terminal device 200 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the terminal device 200 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify the sound source, realize directional recording function, etc.

[0323] The earphone interface 170D is used to connect a wired earphone and can be a USB interface 130 or a 3.5 mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0324] The button 190 includes a power button, a volume button, etc. The button 190 may be a mechanical button. It may also be a touch button. The terminal device 200 may receive the button input and generate a key signal input related to the user settings and function control of the terminal device 200. The motor 191 may generate a vibration prompt. The SIM card interface 195 is used to connect the SIM card. The SIM card may be connected to or separated from the terminal device 200 by inserting the SIM card interface 195 or pulling it out from the SIM card interface 195.

[0325] Fig.19 The illustrated structure does not constitute a specific limitation on the terminal device 200. The terminal device 200 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0326] Fig. 20FIG. 3 shows a server 300 provided in an embodiment of the present application. The server 300 may be the cloud server 30 mentioned in the above embodiment. Fig. 20 As shown, the server 300 may include: a processor 210, a memory 220, an input and output device 230, a communication module 240, etc. These components may be coupled via a bus.

[0327] The server 300 may have powerful computing resources, and the processor 210 thereon may include one or more processors with powerful computing power, such as a central processing unit (CPU), a neural network processing unit (NPU), a graphics processing unit (GPU), etc.

[0328] The processor 210 may include one or more interfaces, such as an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general purpose input output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0329] The processor 210 may be provided with a cache memory, which may be used to store instructions or data just used or cyclically used by the processor 210. If the processor 210 needs to use the instruction or data again, it may be directly called from the cache memory, which may reduce the waiting time of the processor 210 and improve the program running efficiency.

[0330] The processor 210 may also be connected to an external memory. The memory may be a high-speed random access memory or a non-volatile memory, such as a disk, a flash memory, a universal flash memory (UFS), etc. The memory may also be an external memory card, such as a Micro SD card.

[0331] The processor 210 is the computing core of the server 300 and has powerful computing capabilities. It is coupled to the memory 220 and can be used to read and execute computer-readable instructions in the memory 220, and run the operating system and various programs. Specifically, the CPU 210 can be used to call the program stored in the memory 220, such as the implementation program of the end-cloud collaborative computing power scheduling method provided in the embodiment of the present application, and execute the instructions contained in the program.

[0332] The memory 220 may include a high-speed random access memory, a non-volatile memory, such as a disk, a flash memory, or other non-volatile solid-state storage device. The memory 220 may be used to store various software programs and multiple sets of instructions. The memory 220 may store an operating system, such as an operating system such as Linux. The memory 220 may also store one or more programs, such as programs involved in patch production, such as a compiler and a linker. The memory 220 may also store an implementation program of the end-cloud collaborative computing power scheduling method provided in an embodiment of the present application.

[0333] The input and output devices 230 may include a display screen, a keyboard, a mouse and other devices, which can be used to receive user input and output program execution results to the user.

[0334] The communication module 240 may include a wired communication module and a wireless communication module. Among them, the wired communication module may support wired communication protocols, such as universal serial bus (USB), serial port, Ethernet and other protocols, and communicate with other devices through physical communication cables. The wireless communication module may include 2G / 3G / 4G / 5G and other wireless communication modules, Wi-Fi communication modules, etc. The wireless communication module receives electromagnetic waves via an antenna, modulates and filters the electromagnetic wave signals, and sends the processed signals to CPU210; the wireless communication module may also receive signals to be sent from CPU210, modulate them, amplify them, and convert them into electromagnetic waves for radiation through the antenna.

[0335] Fig. 20 The illustrated structure does not constitute a limitation on the server 300. The server 300 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0336] The steps of the method or algorithm described in conjunction with the disclosure of the embodiments of the present application can be implemented in a hardware manner, or can be implemented by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in RAM, flash memory, ROM, erasable programmable read-only memory (Erasable ProgrammableROM, EPROM), electrically erasable programmable read-only memory (Electrically EPROM, EEPROM), registers, hard disks, mobile hard disks, read-only compact disks (CD-ROMs) or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and can write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC. In addition, the ASIC can be located in a transceiver or a relay device. Of course, the processor and the storage medium can also be present in a wireless access network device or a user device as discrete components.

[0337] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the embodiments of the present application can be implemented with hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein the communication media include any media that facilitates the transmission of a computer program from one place to another. The storage medium can be any available medium that a general or special-purpose computer can access.

[0338] The above specific implementation methods further explain in detail the purpose, technical solutions and beneficial effects of the embodiments of the present application. It should be understood that the above are only specific implementation methods of the embodiments of the present application and are not used to limit the protection scope of the embodiments of the present application. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solutions of the embodiments of the present application should be included in the protection scope of the embodiments of the present application.

Claims

1. A method for scheduling computing power between end and cloud, characterized in that: The method is applied to a terminal device in a first communication system, the first communication system comprising: the terminal device and a server, the terminal device having a first program installed thereon, the software package of the first program comprising a program code, the program code comprising a cloud-side task code and a terminal-side task code; the cloud-side task code comprising a first code snippet, the first code snippet being used to declare that the task type is a cloud-side task; the terminal-side task code comprising a second code snippet, the second code snippet being used to declare that the task type is a terminal-side task; the first code snippet implements a cloud-side task interface in a first programming framework, the second code snippet implements a terminal-side task interface in the first programming framework; The method comprises: The terminal device determines whether to schedule the to-be-executed task in the first program to be executed on the server according to a scheduling strategy; the scheduling strategy is formulated based on one or more of the following factors: task type, power consumption of the terminal device, temperature of the terminal device, network status between the terminal device and the server, and cloud-side computing load; If the task is scheduled to be executed on the server, the terminal device sends the code of the task to be executed to the server, requesting the server to use the cloud-side computing resources to execute the task to be executed; otherwise, the terminal device uses the terminal-side computing resources to execute the task to be executed; the terminal-side computing resources include the processor resources on the terminal device, and the cloud-side computing resources include the processor resources on the server; The terminal device receives the execution result of the task to be executed returned by the server.

2. The method according to claim 1, characterized in that The program code includes the code of the end cloud task, and the code of the end cloud task includes a third code snippet. The third code snippet implements the end cloud task interface in the first programming framework and is used to declare that the task type is an end cloud task.

3. The method according to claim 2, characterized in that The software package of the first program also includes: a first file, wherein the first file is used to indicate the dependency relationship between various tasks in the first program.

4. The method according to claim 3, characterized in that Also includes: The terminal device sends the first file to the server, so that the server can determine the dependency relationship of the tasks to be executed.

5. The method according to claim 3, characterized in that The first file is specifically used to indicate the dependency relationship between the tasks in the first program by recording the execution order and context data of the tasks; the context data includes: input parameters and output parameters of the tasks.

6. The method according to claim 1, characterized in that The scheduling strategy includes one or more of the following: task type strategy, end-side power consumption strategy, end-side temperature strategy, network status strategy, and cloud-side computing load strategy; wherein, The type strategy includes: scheduling the cloud-side task to be executed on the server, and scheduling the terminal-side task to be executed on the terminal device; The terminal side power consumption strategy includes: if the remaining power of the terminal device is higher than the first power, scheduling the to-be-executed task to be executed on the server; otherwise, scheduling the to-be-executed task to be executed on the terminal device; The terminal-side temperature strategy includes: if the temperature of the terminal device is lower than a first temperature, scheduling the task to be executed on the server for execution; otherwise, scheduling the task to be executed on the terminal device for execution; The network status strategy includes: if the communication quality between the terminal device and the server satisfies one or more of the following: the communication bandwidth is greater than the first bandwidth, and the communication delay is less than the first delay, then scheduling the task to be executed on the server, otherwise scheduling the task to be executed on the terminal device; The cloud-side computing power load strategy includes: if the occupancy of the cloud-side computing power resources is less than the first ratio, scheduling the task to be executed on the server; otherwise, scheduling the task to be executed on the terminal device.

7. The method according to claim 2, characterized in that The software package of the first program also includes: a second file, and the second file is used to record the scheduling strategy.

8. The method according to claim 3, characterized in that The terminal device sends the code of the task to be executed to the server, specifically including: The terminal device sends a first request message to the server, wherein the message header of the first request message carries the first file serialized into a byte array, and the message body of the first request message carries the code of the task to be executed.

9. The method according to claim 8, characterized in that The first request message also carries input parameters of the task to be executed.

10. The method according to claim 9, characterized in that The input parameters of the task to be executed include: the execution result of the task on which the task to be executed depends.

11. The method according to any one of claims 1 to 10, characterized in that The terminal device sends the code of the task to be executed to the server, specifically including: The terminal device transmits data at a test rate v_test within a first round trip time RRT of sending the code of the task to be executed to the server; The terminal device receives a confirmation ACK for the first round of data transmission returned by the server; The terminal device adjusts the transmission rate of the code for sending the task to be executed to the starting rate v_start based on the reception time of the ACK, where v_start=m / Δt, where m represents the amount of data sent in the first round; Δt=t2-t1, where t1 represents the starting transmission time of the first round of data sent, and t2 represents the reception time of the ACK for the first round of data sent; The test rate v_test is greater than the start rate v_start.

12. The method according to claim 11, characterized in that The method comprises: The terminal device obtains one or more data of delay information, packet loss rate, and receiving rate, and analyzes the data to identify the noise intensity of the weak network signal; including: the noise intensity of RTT jitter and random packet loss; If the noise intensity is greater than a first threshold, the terminal device corrects the abnormal data causing the noise; The terminal device determines the available bandwidth based on one or more of the corrected delay information, packet loss rate, and receiving rate.

13. The method according to claim 12, characterized in that The method further comprises: The terminal device identifies a traffic trough period during the running of the first program, and upwardly detects the network bandwidth during the traffic trough period and obtains the upward detection bandwidth; the traffic trough period refers to a stage in which the terminal device transmits the code of the first task to the server, and the data volume of the code of the first task is greater than the first data volume; During traffic peak periods, the terminal device adjusts the transmission rate according to the upward detection bandwidth and the historical maximum available bandwidth, and the adjusted transmission rate is the smaller of the upward detection bandwidth and the historical maximum available bandwidth; the traffic peak period refers to the stage when the terminal device transmits the code of the second task to the server, and the data volume of the code of the second task is less than the second data volume; the second data volume is less than or equal to the first data volume.

14. A method for scheduling computing power between end and cloud, characterized in that: The method is applied to a server in a first communication system, the first communication system comprising: a terminal device and the server, a first program is installed on the terminal device, a software package of the first program comprises a program code, the program code comprises a code of a cloud-side task and a code of a terminal-side task; the code of the cloud-side task comprises a first code fragment, the first code fragment implements a cloud-side task interface in a first programming framework, and is used to declare that the task type is a cloud-side task; the code of the terminal-side task comprises a second code fragment, the second code fragment implements the terminal-side task interface in the first programming framework, and is used to declare that the task type is a terminal-side task; The method comprises: The server receives the code of the task to be executed in the first program sent by the terminal device; the task to be executed is scheduled by the terminal device to be executed on the server; The server executes the task to be executed by utilizing cloud-side computing resources; the cloud-side computing resources include processor resources on the server; The server sends the execution result of the task to be executed to the terminal device.

15. The method according to claim 14, characterized in that The program code includes the code of the end cloud task, and the code of the end cloud task includes a third code fragment, and the third code fragment implements the end cloud task interface in the first programming framework, and is used to declare that the task type is an end cloud task.

16. The method according to claim 14 or 15, characterized in that The software package of the first program also includes: a first file, wherein the first file is used to indicate the dependency relationship between various tasks in the first program.

17. The method according to claim 16, characterized in that Also includes: The server receives a first file sent by the terminal device, and determines the dependency relationship of the tasks to be executed according to the first file.

18. The method according to claim 16, characterized in that The first file is specifically used to indicate the dependency relationship between the tasks in the first program by recording the execution order and context data of the tasks; the context data includes: input parameters and output parameters of the tasks.

19. The method of claim 14, wherein: The decision is made based on a scheduling strategy; the scheduling strategy includes one or more of the following: task type strategy, end-side power consumption strategy, end-side temperature strategy, network status strategy, and cloud-side computing load strategy; wherein, The type strategy includes: scheduling the cloud-side task to be executed on the server, and scheduling the terminal-side task to be executed on the terminal device; The terminal side power consumption strategy includes: if the remaining power of the terminal device is higher than the first power, scheduling the to-be-executed task to be executed on the server; otherwise, scheduling the to-be-executed task to be executed on the terminal device; The terminal-side temperature strategy includes: if the temperature of the terminal device is lower than a first temperature, scheduling the task to be executed on the server for execution; otherwise, scheduling the task to be executed on the terminal device for execution; The network status strategy includes: if the communication quality between the terminal device and the server satisfies one or more of the following: the communication bandwidth is greater than the first bandwidth, and the communication delay is less than the first delay, then scheduling the task to be executed on the server, otherwise scheduling the task to be executed on the terminal device; The cloud-side computing power load strategy includes: if the occupancy of the cloud-side computing power resources is less than the first ratio, scheduling the task to be executed on the server; otherwise, scheduling the task to be executed on the terminal device.

20. The method of claim 19, wherein: The software package of the first program also includes: a second file, and the second file is used to record the scheduling strategy.

21. The method of claim 16, wherein: The server receives the code of the task to be executed in the first program sent by the terminal device, specifically including: The server receives a first request message sent by the terminal device, wherein a message header of the first request message carries the first file serialized into a byte array, and a message body of the first request message carries a code of the task to be executed.

22. The method according to claim 21, characterized in that The first request message also carries input parameters of the task to be executed.

23. The method of claim 22, wherein: The input parameters of the task to be executed include: the execution result of the task on which the task to be executed depends.

24. A terminal device, characterized in that: It comprises a processor and a memory; wherein the memory is coupled to the processor, the memory is used to store computer program code, the computer program code comprises computer instructions, and when the processor executes the computer instructions, the terminal device executes the method as described in any one of claims 1-13.

25. A server, characterized in that: It comprises a processor and a memory; wherein the memory is coupled to the processor, the memory is used to store computer program code, the computer program code comprises computer instructions, and when the processor executes the computer instructions, the server executes the method as described in any one of claims 14-23.

26. A communication system, characterized in that: It comprises a terminal device and a server; wherein, the terminal device is the terminal device described in claim 24, and the server is the server described in claim 25.

27. A computer-readable storage medium comprising instructions, characterized in that: When the instruction is executed on the terminal device, the terminal device executes the method according to any one of claims 1 to 13.

28. A computer-readable storage medium comprising instructions, characterized in that: When the instruction is executed on the server, the server is caused to execute the method according to any one of claims 14 to 23.

Images