Terminal-cloud computing power collaborative scheduling method, related system, and device

Abstract

Embodiments of the present application provide a terminal-cloud computing power collaborative scheduling method, a related system, and a device. The method can be applied to a terminal-cloud computing power collaborative scheduling system, and the system can comprise a terminal device and a cloud server. The terminal device may execute a scheduling policy to determine whether to schedule a task in a program to a terminal side for execution or to a cloud side for execution. In this way, cloud-side computing power becomes a part of terminal-side computing power, achieving collaborative scheduling of terminal-side and cloud-side tasks, solving the problem of insufficient terminal-side computing power, and also solving the problems of an idle state of terminal-side computing power and a high cloud side cost caused by placing all logic and computing power on the cloud side.

Description

End-cloud computing power collaborative scheduling method, related systems and equipment

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on October 20, 2023, with application number 202311376083.X, and priority to the Chinese patent application entitled “End-cloud computing power collaborative scheduling method, related systems and equipment”, all contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of electronic technology, and in particular to methods, systems, and equipment for collaborative scheduling of end-cloud computing power. Background Art

[0003] With the widespread adoption of artificial intelligence (AI) technology across various industries, the deep learning models used in applications on mobile phones, tablets, and other devices are becoming increasingly complex, leading to a corresponding increase in the computing power required by these models. However, due to limitations in chip manufacturing processes, power consumption, heat generation, and cost, computing power growth on these devices has slowed, making it difficult to meet the high computing power demands of tasks like AI algorithms and graphics processing unit (GPU) rendering, hindering application innovation.

[0004] Summary of the Invention

[0005] The embodiments of the present application provide a method for collaboratively scheduling end-cloud computing power, a communication system, and related equipment, which can realize collaborative scheduling of computing power on the terminal device side and the cloud server side, and flexibly balance the performance and cost of program operation.

[0006] In a first aspect, an embodiment of the present application provides a method for scheduling computing power in a terminal-cloud collaborative manner, which can be applied to a terminal device in a first communication system. The first communication system may include: the terminal device and a server, wherein a first program is installed on the terminal device. The software package of the first program includes program code, which includes code for cloud-side tasks and code for terminal-side tasks; the code for the cloud-side task includes a first code snippet, which is used to declare that the task type is a cloud-side task; and the code for the terminal-side task includes a second code snippet, which is used to declare that the task type is a terminal-side task.

[0007] The method may include: the terminal device determines whether to schedule the pending task in the first program to be executed on the server based on the scheduling policy; if it is scheduled to be executed on the server, the terminal device sends the code of the pending task to the server, requesting the server to use cloud-side computing resources to execute the pending task; otherwise, the terminal device executes the pending task using terminal-side computing resources. The terminal-side computing resources include processor resources on the terminal device, and the cloud-side computing resources include processor resources on the server. The terminal device receives the execution result of the pending task returned by the server.

[0008] Among them, the scheduling strategy can be formulated based on but not limited to one or more of the following factors: task type, power consumption of the terminal device, temperature of the terminal device, network status between the terminal device and the server, and cloud-side computing load.

[0009] In a second aspect, an embodiment of the present application provides a method for scheduling computing power in a terminal-cloud collaborative manner, which can be applied to a server in a first communication system. The first communication system may include: the terminal device and the server, wherein a first program is installed on the terminal device. The software package of the first program includes program code, which includes code for cloud-side tasks and code for terminal-side tasks; the code for the cloud-side task includes a first code snippet, which is used to declare that the task type is a cloud-side task; the code for the terminal-side task includes a second code snippet, which is used to declare that the task type is a terminal-side task.

[0010] The method may include: receiving, by the server, code for a pending task in the first program sent by the terminal device; the pending task being scheduled by the terminal device for execution on the server; executing the pending task by the server using cloud computing resources; the cloud computing resources including processor resources on the server; and transmitting the execution result of the pending task to the terminal device.

[0011] Implementing the methods of the first and second aspects can realize the coordinated operation of programs by utilizing the computing power resources on the terminal side and the computing power resources on the cloud side, taking into account both the program operation performance and the cost.

[0012] In the first and second aspects, the tasks in the first program can be implemented in accordance with the first programming framework. The first programming framework, which is the end-cloud collaborative programming framework mentioned in subsequent embodiments, may include interfaces for declaring task types: "MobileTask" and "CloudTask". Among them, the interface "MobileTask" can be used to declare that the task type of the task is an end-side task, and the interface "CloudTask" can be used to declare that the task type of the task is a cloud-side task. In addition to end-side tasks and cloud-side tasks, the end-cloud collaborative programming framework may also include an interface for declaring the task type of end-cloud tasks: "MobileCloudTask". Program developers can choose to have tasks inherit MobileTask, CloudTask, or MobileCloudTask based on the actual computing power requirements of each task in the program. Specifically, the end-side tasks in the first program inherit (or implement) the MobileTask interface, the cloud-side tasks inherit (or implement) the CloudTask interface, and the end-cloud tasks inherit (or implement) the MobileCloudTask interface.

[0013] In the software package of the first program, the code for the cloud-side task and the code for the terminal-side task each have their own characteristics, namely: the code for the terminal-side task may include a first code snippet, and the code for the cloud-side task may include a second code snippet. Therefore, the terminal device can identify the task type based on the machine code of the task. Among them, the first code snippet can be a piece of machine code compiled from the source code (such as "extends MobileTask") used to declare that the task type is a terminal-side task, and the second code snippet can be a piece of machine code compiled from the source code (such as "extends CloudTask") used to declare that the task type is a cloud-side task.

[0014] In this way, through the end-cloud collaborative programming framework provided by the embodiment of the present application, program developers can achieve computing power collaboration between the end side and the cloud side with one development, and can flexibly schedule tasks in the program to the cloud side or end side for execution on demand based on factors such as the end side computing power load, end side power consumption, end side temperature control, network status, and cloud side computing power load, thereby turning the cloud side computing power into part of the end side computing power.

[0015] In conjunction with the first aspect, in some embodiments, the program code of the first program may further include code for a mobile-cloud-side task. The mobile-cloud-side task code may include a third code snippet, which may be a segment of machine code converted from source code (e.g., "extends MobileCloudTask") that declares the task type as a mobile-cloud-side task.

[0016] In combination with the first aspect or the second aspect, in some embodiments, the software package of the first program may further include: a first file, the first file being used to indicate the dependency relationship between the various tasks in the first program. The first file is the dependency orchestration file mentioned in the subsequent embodiments. The terminal device can extract the first file from the software package of the first program to provide the dependency relationship between the tasks to the cloud server through the communication proxy service, so that the cloud server executes the task according to the dependency relationship. In some embodiments, the terminal device does not need the dependency relationship when executing the task, because the task call on the terminal side is a code-level call, and no additional configuration file is required.

[0017] In combination with the first aspect or the second aspect, in some embodiments, the method of the first aspect further includes: the terminal device sends a first file to the server, so that the server determines the dependency relationship of the task to be executed.

[0018] Specifically, the terminal device 20 can serialize the dependency relationship recorded in the first file into a byte array and place it in the message header of the request message. In this way, after receiving the request message, the cloud server can parse the dependency relationship from the message header. The dependency relationship placed in the message header can be exemplified as follows: taskDAGs:byte[]Codec(in:taskDAGs.json). Here, "Codec(in:taskDAGs.json)" indicates that the dependency relationship orchestration file taskDAGs.json is encoded into a bytecode array.

[0019] In conjunction with the first aspect, in some embodiments, the first file is specifically used to indicate the dependency relationships between tasks in the first program by recording the execution order and context data of each task; the context data includes: the input parameters and output parameters of the task. In other words, the dependency relationships between tasks can be used to describe: the execution order between tasks, the input parameters and output parameters of each task. Task a's dependency on task b can be reflected in: the execution order of task b is earlier than the execution order of task a, and the input parameters of task a include the output parameters (execution results) of task b.

[0020] In combination with the first aspect or the second aspect, in some embodiments, the scheduling strategy may include but is not limited to one or more of the following:

[0021] 1. Task type strategy

[0022] Task scheduling is performed based on the task type. For example, you can schedule device-side tasks to the device for execution, and cloud-side tasks to the cloud for execution. For device-cloud tasks, you can choose to schedule them to the device or the cloud.

[0023] Different task types reflect different computing power requirements. Developers typically declare light-load tasks as client-side tasks and heavy-load tasks as cloud-side tasks. Developers can also declare heavy-load tasks as client-cloud tasks, or light-load tasks as client-cloud tasks.

[0024] Program developers can use the device-cloud collaborative programming framework to declare the task type for each task in the first program. In the first program's software package, the machine code for cloud-side tasks and device-side tasks have distinct characteristics, allowing the terminal device to distinguish between tasks of different types. The data characteristics of the first program's software package will be explained later, so we won't elaborate on them here.

[0025] 2. Whitelist strategy

[0026] Perform whitelist verification, such as verifying whether the terminal device model is a preset model, whether the application type is a preset type, and whether the terminal device is currently in a preset region. If the whitelist verification passes, the task is scheduled to the cloud side for execution; otherwise, the task is scheduled to the device side for execution. In some embodiments, only the device whitelist policy can be applied to the scheduling of cloud-side tasks and device-cloud tasks. Device-side tasks can be scheduled to the device side for execution based on the task type policy.

[0027] 3. End-side power consumption strategy

[0028] Considering the power consumption of the terminal device, if the remaining power of the terminal device is less than a first power value, such as 30%, the end-cloud task is scheduled to the cloud side for execution to save end-side power consumption. In some embodiments, only the end-side power consumption policy can be applied to the scheduling of end-cloud tasks. The scheduling of cloud-side and end-side tasks can be considered based on task type policies, etc.

[0029] 4. Device-side temperature control strategy

[0030] Considering the temperature of the terminal device, if the terminal device temperature is higher than a first temperature, such as 38°C, the task is scheduled to the cloud side to alleviate the problem of excessive temperature on the terminal side. In some embodiments, the terminal-side temperature control policy can be applied only to the scheduling of terminal-cloud tasks. The scheduling of cloud-side and terminal-side tasks can be based on factors such as task type policy.

[0031] 5. Network Status Strategy

[0032] Considering the communication quality between the terminal device and the cloud server, if the communication quality is high, the task will be scheduled to the cloud side for execution, otherwise the task will be scheduled to the end side for execution. Among them, the communication quality may include but is not limited to: bandwidth, delay, etc. The standard of high communication quality may include but is not limited to: the communication bandwidth is greater than the first bandwidth (such as 200Mbps) and the communication delay is lower than the first delay (such as 1 millisecond). In some embodiments, the network status policy can be applied only to the scheduling of cloud-side tasks and end-cloud tasks. End-side tasks can be scheduled to the end side for execution based on the task type policy.

[0033] 6. Cloud-side computing load strategy

[0034] Considering the cloud-side computing load, if the cloud-side computing load is below a first ratio, such as 60%, the task is scheduled to the cloud for execution; otherwise, the task is scheduled to the device for execution. Cloud-side computing load = cloud-side computing resource usage / total cloud-side computing resource volume. In some embodiments, only the network status policy can be applied to the scheduling of cloud-side and device-cloud tasks. Device-side tasks can be scheduled to the device for execution based on the task type policy.

[0035] 7. Dynamic Factor Strategies

[0036] For task scheduling, the available bandwidth between the terminal device and the cloud server is dynamically detected. Furthermore, the task's transmission time is considered to meet the task's latency requirements, such as the latency requirement specified by the deadline parameter in the submit() method. If the latency requirement is met, the task is scheduled to the cloud for execution; otherwise, it is scheduled to the client for execution.

[0037] The transmission time of a task can be estimated based on the task's packet size and the real-time available bandwidth. The packet size of a task primarily depends on the size of its machine code. In some embodiments, dynamic factor policies can be applied only to the scheduling of cloud-side and end-cloud tasks. End-side tasks can be scheduled for execution on the end-side based on the task type policy.

[0038] By formulating a scheduling strategy, we can consider factors such as the computing power requirements of each task, end-side power consumption, end-side temperature control, network status, and cloud-side computing power load to implement task scheduling, so that task scheduling can be more flexible and adapt to different scenarios.

[0039] The scheduling strategies described above are not limited to those described above. Application developers can also customize scheduling strategies based on the application's business characteristics. For example, if the user's number is located overseas, the task will not be executed on the cloud side.

[0040] In conjunction with the first or second aspect, in some embodiments, the software package of the first program may further include: a second file for recording a scheduling policy. This second file is the scheduling policy file mentioned in subsequent embodiments. When the terminal device runs the application, it can read the scheduling policy in this file and perform task scheduling according to the scheduling policy. This file can be packaged in the software package of the application. The terminal device can obtain this file after downloading the software package of the application.

[0041] In combination with the first aspect or the second aspect, in some embodiments, the terminal device sends the code of the task to be executed to the server, specifically including: the terminal device sends a first request message to the server, the message header of the first request message carries a first file serialized into a byte array, and the message body of the first request message carries the code of the task to be executed. The machine code of the task to be executed can be organized into a binary file, such as a .so file. The message body can be a byte array that can be used to store a file serialized into a byte array (such as a .so file).

[0042] In combination with the first aspect or the second aspect, in some embodiments, the first request message also carries the input parameters of the task to be executed. The input parameters can be specifically stored in the message body of the request message. The input parameters of a task can be the output parameters (such as execution results) of other tasks on which it depends. Assuming that task a is an image rendering task, the input parameter of task a can be a RAW image captured by the camera, which can be the output parameter of the image capture task on which task a depends, and it can be encoded into a byte array and stored in the message body of the request message to be sent to the cloud server 30 for rendering processing.

[0043] In combination with the first aspect or the second aspect, in some embodiments, the terminal device may use a fast link establishment method to adjust the transmission rate of the task to be executed, thereby reducing the network transmission delay.

[0044] Specifically, the terminal device sending the code of the task to be executed to the server may specifically include: within the first round-trip time RRT of sending the code of the task to be executed to the server, the terminal device transmits data at a test rate v_test; the terminal device receives the confirmation ACK returned by the server for the first round of data sent; the terminal device adjusts the transmission rate of the code for sending the task to be executed to the starting rate v_start based on the reception time of the ACK, v_start = m / △t, where m represents the amount of data sent in the first round; △t = t2-t1, where t1 represents the starting transmission time of the first round of data sending, and t2 represents the reception time of the ACK for the first round of data sending; wherein, the test rate v_test is greater than the starting rate v_start.

[0045] In combination with the first aspect or the second aspect, in some embodiments, when the terminal device sends the code of the task to be executed to the server, the terminal device can also execute the congestion control method with RTT granularity, which can resist the interference of weak network noise in the wide area network and more accurately estimate the available bandwidth. Specifically, the method may include: the terminal device obtains one or more data of delay information, packet loss rate, and receiving rate, and analyzes the data to identify the noise intensity of the weak network signal; including: the noise intensity of RTT jitter and random packet loss; if the noise intensity is greater than the first threshold, the terminal device corrects the abnormal data causing the noise; the terminal device determines the available bandwidth based on one or more data of the corrected delay information, packet loss rate, and receiving rate. In this way, it is possible to resist the interference of weak network noise in the wide area network, and introduce a traffic feature-aware congestion control module to adjust the rate in time according to the traffic characteristics of the application to ensure transmission efficiency.

[0046] In combination with the first aspect or the second aspect, in some embodiments, when the terminal device sends the code of the task to be executed to the server, the terminal device can also perform congestion control based on traffic characteristics perception, and can adjust the rate in time according to the traffic characteristics of the application program to ensure transmission efficiency. Specifically, the method may include: the terminal device identifies the traffic trough period during the operation of the first program, and upwardly probes the network bandwidth during the traffic trough period and obtains the upward probe bandwidth; the traffic trough period refers to the stage when the terminal device transmits the code of the first task to the server, and the data volume of the code of the first task is greater than the first data volume; during the traffic peak period, the terminal device adjusts the transmission rate according to the upward probe bandwidth and the historical maximum available bandwidth, and the adjusted transmission rate is the smaller of the upward probe bandwidth and the historical maximum available bandwidth; the traffic peak period refers to the stage when the terminal device transmits the code of the second task to the server, and the data volume of the code of the second task is less than the second data volume; the second data volume is less than or equal to the first data volume.

[0047] In a third aspect, the present application provides a terminal device comprising a processor and a memory; wherein the memory is coupled to the processor, the memory is used to store computer program code, the computer program code comprises computer instructions, and when the processor executes the computer instructions, the terminal device executes the method described in the first aspect and any possible implementation of the first aspect.

[0048] In a fourth aspect, the present application provides a server comprising a processor and a memory; wherein the memory is coupled to the processor, and the memory is used to store computer program code, and the computer program code comprises computer instructions. When the processor executes the computer instructions, the server executes the method described in the second aspect and any possible implementation of the second aspect.

[0049] In the fifth aspect, an embodiment of the present application provides a chip system, which is applied to a terminal device, and the chip system includes one or more processors, which are used to call computer instructions to enable the terminal device to execute the method described in the first aspect and any possible implementation method of the first aspect.

[0050] In the sixth aspect, an embodiment of the present application provides a chip system, which is applied to a server, and the chip system includes one or more processors, which are used to call computer instructions to enable the server to execute the method described in the second aspect and any possible implementation method of the second aspect.

[0051] In a seventh aspect, the present application provides a computer-readable storage medium comprising instructions, which, when executed on a terminal device, enable the terminal device to execute the method described in the first aspect and any possible implementation of the first aspect.

[0052] In an eighth aspect, the present application provides a computer program product comprising instructions, which, when the computer program product is run on a server, enables the server to execute the method described in the second aspect and any possible implementation of the second aspect.

[0053] In a ninth aspect, the present application provides a communication system, which may include a terminal device and a server; wherein the terminal device may be the terminal device in the first aspect and any possible implementation of the first aspect, and the server may be the server in the second aspect and any possible implementation of the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the background technology, the drawings required for use in the embodiments of the present application or the background technology will be described below.

[0055] FIG1 shows a workflow of a photo taking application;

[0056] Figure 2 shows a cloud gaming workflow;

[0057] FIG3 exemplarily illustrates a method of collaborative scheduling of end-cloud computing power implemented in an embodiment of the present application;

[0058] 4A-4C exemplarily illustrate several task scheduling scenarios;

[0059] FIG5 exemplarily shows a system for coordinating computing power between devices and clouds provided in an embodiment of the present application;

[0060] Figure 6 shows the different states that a task in a program presents during its life cycle;

[0061] FIG7 exemplarily shows that the dependency relationship is graphically recorded in a configuration file in a YAML\XML format;

[0062] FIG8 exemplarily shows the functions of the access module in the cloud server;

[0063] FIG9 exemplarily shows the functions of the task scheduling execution module in the cloud server;

[0064] FIG10 shows a terminal-cloud collaborative programming framework provided by an embodiment of the present application;

[0065] FIG11 exemplarily shows the overall process of the end-cloud computing power collaborative scheduling method provided in an embodiment of the present application;

[0066] FIG12 shows the functionality of the communication proxy service in the terminal device;

[0067] FIG13 shows the functions implemented by the communication agent service and the scheduling policy service in the terminal device;

[0068] FIG14 shows a high-performance, reliable protocol transmission framework implemented on the terminal device side;

[0069] FIG15 exemplarily shows the process of the fast link establishment method provided in an embodiment of the present application;

[0070] FIG16 shows an existing congestion control method;

[0071] FIG17 exemplarily shows an architecture for implementing congestion control provided by an embodiment of the present application;

[0072] FIG18 exemplarily shows the process of the congestion control method provided in an embodiment of the present application;

[0073] FIG19 shows a terminal device provided in an embodiment of the present application;

[0074] Figure 20 shows the server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0075] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application.

[0076] In order to meet the high computing power requirements of tasks such as AI algorithms and graphics processing unit (GPU) graphics rendering, the industry mainly has two solutions to solve this problem.

[0077] One solution is to optimize the AI ​​model used on the terminal device side, trim the AI ​​model to make it as small as possible, and reduce its computing power requirements. However, the problem caused by the simplification of the model is that the AI ​​processing effect is poor and cannot achieve the expected results.

[0078] Taking the camera function of a mobile phone as an example, as shown in Figure 1, after the original image (RAW image) is generated by the image sensor and image signal processor (ISP), the mobile phone will execute some camera AI algorithms to perform noise reduction, beautification, super resolution (SR) and other processing for scenes such as portraits and macros to generate photos with higher image quality. Users can see the AI-processed images through applications such as the gallery. Compared with the original images, the AI-processed images can generally improve the image quality by about 20%. However, since camera performance is a key competitive point for major mobile phone manufacturers, the AI ​​algorithms for camera functions are becoming more and more complex, and the computing power of mobile phones is insufficient, resulting in poor execution of image processing algorithms. For example, in portrait shooting scenes such as telephoto shooting, group photos, and motion capture, there is a problem of image blur, which fails to meet user expectations.

[0079] Another approach is to offload all terminal application tasks to cloud servers, leveraging the cloud server's high computing power to compensate for the lack of computing power on the terminal device. For example, all computing tasks for cloud gaming are offloaded to the cloud server, with the user's terminal device serving only as an input / output device to receive user input, provide display, and output sound. However, this approach requires the application provider to pay high cloud server usage fees and also results in idle computing power on the terminal device.

[0080] For example, as shown in Figure 2, all computing tasks for cloud gaming are performed on cloud servers, while terminal devices such as Mobile Phone 1 and Mobile Phone 2 are only used to receive user input and display the game video stream. Leveraging the computing power of cloud servers, cloud gaming can significantly surpass its performance on mobile phones. However, cloud computing resources must be allocated to each game opened on a mobile phone. Furthermore, the high cost of cloud server computing means that cloud gaming providers must incur high cloud server usage fees, hindering the large-scale commercial use of cloud gaming.

[0081] The embodiments of the present application provide an end-cloud computing power collaborative scheduling system, an end-cloud computing power collaborative scheduling method and related equipment, which can realize the collaborative operation of programs by utilizing end-side computing power resources and cloud-side computing power resources, taking into account both program operation performance and cost.

[0082] FIG3 exemplarily shows a collaborative scheduling of end-cloud computing power implemented in an embodiment of the present application.

[0083] As shown in Figure 3, the applications on the terminal device can be divided into multiple tasks. Light-load tasks can be scheduled to the terminal device for execution, using the terminal-side computing resources, such as the central processing unit (CPU), graphics processing unit (GPU), neural processing unit (NPU) and other computing resources on the terminal device. Heavy-load tasks can be scheduled to the cloud server for execution, using the cloud-side computing resources, such as the central processing unit (CPU), graphics processing unit (GPU), neural processing unit (NPU) and other computing resources on the cloud server.

[0084] Through the coordinated scheduling of end-cloud computing power, the computing power requirements of heavy-loaded tasks can be guaranteed, improving the program running effect, and the end-side computing power resources can be used to run light-loaded tasks, avoiding the entire program running being deployed on the cloud side, which can reduce the program running cost.

[0085] For example, in the field of graphics rendering, GPU rendering tasks are divided into heavy-loaded rendering tasks (such as ray tracing) and light-loaded rendering tasks. Heavy-loaded rendering tasks are scheduled to be executed on the cloud server, while light-loaded rendering tasks are scheduled to be executed on the terminal device. After the cloud server completes the execution of the heavy-loaded rendering task, it returns the image processing results such as video streams / pictures to the terminal device. In this way, the GPU rendering effect can be improved, such as improving the picture effect of ray tracing, without causing idle computing power on the terminal device side, thereby reducing the cloud-side computing power usage cost of graphics rendering.

[0086] For example, in the field of general computing, AI algorithms are divided into heavy-load algorithms and light-load algorithms. Heavy-load algorithms are scheduled for execution on cloud servers, while light-load algorithms (such as those for recognizing scenes like portraits and food) are scheduled for execution on terminal devices. This improves the performance of AI algorithms while preventing idle computing power on terminal devices, thereby reducing the cost of cloud-side computing power for AI algorithms.

[0087] In the embodiments of the present application, heavy-load tasks refer to tasks with high computing power requirements, and their computing power requirements generally need to be guaranteed by cloud-side computing power; light-load tasks refer to tasks with low computing power requirements, and their computing power requirements can be met by terminal-side computing power.

[0088] In the embodiments of this application, a task is a functional module of a program. Tasks can be used to divide a program into multiple functional modules. A task can be a function, or it can be composed of multiple functions that perform the same functional unit. A function consists of a function header and a function body. The function header includes the function name, return value type, and parameter list, and the function body includes the statement block executed by the function.

[0089] Figures 4A-4C illustrate several task scheduling scenarios. If all tasks (tasks 1 to 4) in the application shown in Figure 4A are light-load tasks, then when the computing power resources on the terminal side are sufficient, all tasks in the application can be scheduled for execution on the terminal device; if all tasks (tasks 1 to 4) in the application shown in Figure 4B are heavy-load tasks, then all tasks in the application can be scheduled for execution on the cloud server; if some tasks (task 1 and task 4) in the application shown in Figure 4C are light-load tasks and the other tasks (task 2 and task 3) are heavy-load tasks, then the light-load tasks can be scheduled for execution on the terminal device, and the heavy-load tasks can be scheduled for execution on the cloud server.

[0090] FIG5 exemplarily shows an end-cloud computing power collaborative scheduling system provided in an embodiment of the present application.

[0091] As shown in Figure 5, the system includes a terminal device 20 and a cloud server 30. Among them, a first program can be installed on the terminal device 20. The first program is developed by the program developer based on the end-cloud collaborative programming framework. The framework will be introduced later and will not be expanded here. The tasks in the first program can include the following types: cloud-side tasks and end-side tasks. End-side tasks can be tasks scheduled to be executed on the terminal device 20, and cloud-side tasks can be tasks scheduled to be executed on the cloud server 30. These multiple tasks can also include end-cloud tasks, which are tasks that can be scheduled to be executed on both the terminal device 20 and the cloud server 30.

[0092] The terminal device 20 can be used to send cloud-side tasks or end-cloud tasks to the cloud server 30 and request the cloud server 30 to execute the task. Accordingly, the cloud server 30 can respond to the request of the terminal device 20, execute the task, and return the execution result to the terminal device 20, so that the terminal device 20 can update the running status of the first program, such as refreshing the interface. In this way, the cloud-side computing power becomes part of the end-side computing power, realizing the coordinated scheduling of end-side and cloud-side tasks. While solving the problem of insufficient end-side computing power, it also avoids placing all logic and computing power on the cloud side, resulting in idle end-side computing power and high cloud-side costs.

[0093] In the end-cloud computing power collaborative scheduling system, the functions of the terminal device 20 and the cloud server 30 and the executed processes can also refer to the relevant content in the subsequent method embodiment of Figure 11.

[0094] As shown in FIG5 , the terminal device 20 may include: a scheduling module 201 and a task execution engine module 203 . The cloud server 30 may include: an access module 301 and a task scheduling execution module 303 .

[0095] As shown in Figure 5, the key processes in the end-cloud computing power collaborative scheduling system may include:

[0096] 1. The task execution engine module 203 in the terminal device 20 executes the task.

[0097] 2. When the task is running, the scheduling module 201 in the terminal device 20 determines whether the task is executed on the terminal side or the cloud side.

[0098] 3. If the task meets the cloud-side execution conditions, the scheduling module 201 in the terminal device 20 sends the task to the cloud server 30.

[0099] 4. The access module 301 in the cloud server 30 is responsible for receiving tasks from the terminal device 20 and load balancing the tasks. If the terminal device 20 is accessing the cloud server for the first time, an operating environment is allocated to the terminal device 20. Subsequent tasks received from the terminal device 20 are routed to the previously allocated operating environment.

[0100] 5. The task scheduling execution module 303 in the cloud server 30 allocates cloud-side resources such as CPU, NPU or GPU according to the computing power requirements of the task and executes the task.

[0101] The following is a detailed description of the various components of the end-cloud computing power collaborative scheduling system.

[0102] Scheduling Module 201

[0103] The scheduling module 201 may include two services: a scheduling strategy and a communication agent.

[0104] Scheduling Strategy Service

[0105] The scheduling policy service can be used to execute the scheduling policy, and choose whether to schedule the task in the first program to be executed on the cloud side or on the terminal side, or to schedule part of the task to be executed on the terminal side and the other part to be executed on the cloud side.

[0106] The scheduling strategy can be formulated based on, but not limited to, one or more of the following factors: task type, power consumption of the terminal device, temperature of the terminal device, network status between the terminal device and the server, and cloud-side computing load.

[0107] The scheduling policy can be the default scheduling policy or customized by the program developer.

[0108] In the implementation of this application, the scheduling strategy may include but is not limited to one or more of the following:

[0109] 1. Task type strategy

[0110] Task scheduling is performed based on the task type. For example, you can schedule device-side tasks to the device for execution, and cloud-side tasks to the cloud for execution. For device-cloud tasks, you can choose to schedule them to the device or the cloud.

[0111] Different task types reflect different computing power requirements. Developers typically declare light-load tasks as client-side tasks and heavy-load tasks as cloud-side tasks. Developers can also declare heavy-load tasks as client-cloud tasks, or light-load tasks as client-cloud tasks.

[0112] Program developers can use the device-cloud collaborative programming framework to declare the task type for each task in the first program. In the first program's software package, the machine code for cloud-side tasks and device-side tasks have distinct characteristics, allowing the terminal device to distinguish between tasks of different types. The data characteristics of the first program's software package will be explained later, so we won't elaborate on them here.

[0113] 2. Whitelist strategy

[0114] Perform whitelist verification, such as verifying whether the terminal device model is a preset model, whether the application type is a preset type, and whether the terminal device is currently in a preset region. If the whitelist verification passes, the task is scheduled to the cloud side for execution; otherwise, the task is scheduled to the device side for execution. In some embodiments, only the device whitelist policy can be applied to the scheduling of cloud-side tasks and device-cloud tasks. Device-side tasks can be scheduled to the device side for execution based on the task type policy.

[0115] 3. End-side power consumption strategy

[0116] Considering the power consumption of the terminal device, if the remaining power of the terminal device is less than a first power value, such as 30%, the end-cloud task is scheduled to the cloud side for execution to save end-side power consumption. In some embodiments, only the end-side power consumption policy can be applied to the scheduling of end-cloud tasks. The scheduling of cloud-side and end-side tasks can be considered based on task type policies, etc.

[0117] 4. Device-side temperature control strategy

[0118] Considering the temperature of the terminal device, if the terminal device temperature is higher than a first temperature value, such as 38°C, the task is scheduled to the cloud side to alleviate the problem of excessive temperature on the terminal side. In some embodiments, the terminal-side temperature control policy can be applied only to the scheduling of terminal-cloud tasks. The scheduling of cloud-side and terminal-side tasks can be based on task type policies, etc.

[0119] 5. Network Status Strategy

[0120] Considering the communication quality between the terminal device and the cloud server, if the communication quality is high, the task will be scheduled to the cloud side for execution, otherwise the task will be scheduled to the end side for execution. Among them, the communication quality may include but is not limited to: bandwidth, delay, etc. The standard of high communication quality may include but is not limited to: the communication bandwidth is greater than the first bandwidth (such as 200Mbps) and the communication delay is lower than the first delay (such as 1 millisecond). In some embodiments, the network status policy can be applied only to the scheduling of cloud-side tasks and end-cloud tasks. End-side tasks can be scheduled to the end side for execution based on the task type policy.

[0121] 6. Cloud-side computing load strategy

[0122] Consider the cloud-side computing load. If the cloud-side computing load is below a certain percentage, such as 60%, the task is scheduled to the cloud for execution; otherwise, the task is scheduled to the device for execution. Cloud-side computing load = cloud-side computing resource usage / total cloud-side computing resource availability. In some embodiments, only the network status policy can be applied to the scheduling of cloud-side and device-cloud tasks. Device-side tasks can be scheduled to the device for execution based on the task type policy.

[0123] 7. Dynamic Factor Strategies

[0124] For task scheduling, the available bandwidth between the terminal device and the cloud server is dynamically detected. Furthermore, the task's transmission time is considered to meet the task's latency requirements, such as the latency requirement specified by the deadline parameter in the submit() method. If the latency requirement is met, the task is scheduled to the cloud for execution; otherwise, it is scheduled to the client for execution.

[0125] The transmission time of a task can be estimated based on the task's packet size and the real-time available bandwidth. The packet size of a task primarily depends on the size of its machine code. In some embodiments, dynamic factor policies can be applied only to the scheduling of cloud-side and end-cloud tasks. End-side tasks can be scheduled for execution on the end-side based on the task type policy.

[0126] By formulating a scheduling strategy, we can consider factors such as the computing power requirements of each task, end-side power consumption, end-side temperature control, network status, and cloud-side computing power load to implement task scheduling, so that task scheduling can be more flexible and adapt to different scenarios.

[0127] The scheduling strategies described above are not limited to those described above. Application developers can also customize scheduling strategies based on the application's business characteristics. For example, if the user's number is located overseas, the task will not be executed on the cloud side.

[0128] The scheduling policy can be recorded in a scheduling policy file, which can be packaged in the software package of the first program. The terminal device can obtain the scheduling policy file after downloading the software package of the first program.

[0129] For example, a scheduling policy file (such as taskDispatchPolicy.json) can record the scheduling policy as follows:

[0130] In the taskDispatchPolicy.json example above, the policy[] array records the numbers of the two scheduling policies, "temperaturePolicy" and "userDefinedPolicy" (see the assignment of the "NO." parameter), key parameter values ​​(see the assignment of the "value" parameter), and scheduling targets (see the assignment of the "run" parameter).

[0131] The "temperaturePolicy" scheduling policy can be the device-side temperature control policy mentioned above. Its key parameter value is "38+" and the scheduling target is "cloud." This specifically indicates the following scheduling policy: when the device-side temperature is above 38°C, the task is scheduled to the cloud to alleviate the problem of excessive temperature on the device.

[0132] The "userDefinedPolicy" scheduling policy can be the custom policy defined by the application developer mentioned above. Its key parameters are the "appPolicy" business policy defined by the application developer to determine whether to schedule tasks to the cloud for execution, and the scheduling target "cloud." "appPolicy" can be a string describing the custom business policy. Specifically, this scheduling policy can determine whether to schedule tasks to the cloud based on the custom "appPolicy."

[0133] Communication proxy service

[0134] The communication proxy service can be used to establish a communication connection between the terminal device 20 and the cloud server 30, providing high-performance, low-latency communication services for scheduling tasks to the cloud side for execution.

[0135] After the scheduling policy service determines to schedule the cloud-side task in the first program to the cloud-side for execution, the communication proxy service can send the machine code of the cloud-side task to the cloud server 30 via the communication connection and request the cloud server 30 to execute the task. The communication proxy service can also be used to receive the task execution results, such as rendered image data, returned by the cloud server 30 via the communication connection.

[0136] Compared with traditional Hypertext Transfer Protocol (HTTP) communication, the communication proxy service is optimized in the following aspects:

[0137] 1. Protocol optimization: Some new fields have been added to the protocol header of the Computer as a Service (Caas) protocol. Specifically, the following fields are added:

[0138] The following fields can be added to the message header of the request message based on the Caas protocol:

[0139] Field "ProjectId": Program development project number, which can correspond to the cloud-side resource number and is used for cloud-side tenant isolation.

[0140] Field "AppId": program number.

[0141] Field "AppVersion": program version number.

[0142] Field "DeviceId": the device ID of the terminal device, which can be a universally unique identifier (UUID).

[0143] Field "TaskDAGs": string array that describes the dependencies between tasks.

[0144] A request message can be sent from terminal device 20 to cloud server 30 to request that cloud server 30 execute a task. The body of the request message can carry the machine code for the task. For example, the CaasMsgBody body of a Caas protocol request message can include a byte array, into which the machine code for the task can be serialized and stored.

[0145] The following fields can be added to the message header of the response message based on the Caas protocol:

[0146] Field "TaskDAGs": string array that describes the dependencies between tasks.

[0147] Field "ErrorCode": error code, a specific value (such as 0) can indicate success.

[0148] Field "ErrorMsg": error description information.

[0149] The cloud server 30 may send a response message to the terminal device 20 to provide feedback on the task execution result. The response message body may carry the task execution result. For example, the CaasMsgBody message body of a Caas protocol response message may include a byte array, and the task execution result may be serialized and stored in the byte array.

[0150] In the message header of the response message, the field "TaskDAGs" is optional.

[0151] 2. The communication proxy service can detect the available bandwidth between the terminal device 20 and the cloud server 30, and provide the real-time detected available bandwidth to the scheduling strategy service as a judgment condition for whether to schedule the cloud-side service to be executed on the cloud server 30, such as the "dynamic factor strategy" mentioned above.

[0152] 3. The communication proxy service can detect the cloud-side computing load and provide the real-time detected cloud-side computing load to the scheduling strategy service as a judgment condition for whether to schedule the cloud-side service to the cloud server 30 for execution, such as the "cloud-side computing load strategy" mentioned above.

[0153] In the specific implementation, the communication proxy service can send heartbeat messages to the cloud server 30 every period T, and receive information returned by the cloud server 30 such as the current occupancy of cloud-side computing resources such as NPU and GPU, the execution time of cloud-side tasks in the previous period, etc., so as to estimate the cloud-side computing load based on this information and assist in deciding whether the cloud-side tasks in the next period can be scheduled to the cloud side for execution.

[0154] Task execution engine module 203

[0155] The task execution engine module 203 can be used to use the terminal computing power resources to execute the terminal tasks. When the scheduling policy service determines to schedule the cloud-side task or the terminal-cloud task in the first program to the terminal for execution, the terminal computing power resources can also be used to execute the cloud-side task or the terminal-cloud task.

[0156] The task execution engine module 203 can also be used to manage the life cycle of tasks when running tasks in the first program, such as creating tasks, establishing task queues, and managing task status. For example, as shown in Figure 6, a task may include the following states in its life cycle: pending, ready, executing, finished, and blocked. After the task is created, it enters the waiting state. After the running conditions are met, it enters the ready state. Tasks in the ready state can be scheduled for execution on the end side or the cloud side. Tasks in execution are in the executing state. During the execution of the task, it can be suspended and in the blocked state. After the task is completed, it is in the finished state.

[0157] The task execution engine module 203 may also be configured to extract a dependency orchestration file from the first program's software package, and to provide the dependencies between tasks to the cloud server 30 via the communication proxy service, so that the cloud server 30 executes the tasks based on the dependencies. In some embodiments, the terminal device 20 does not require these dependencies when executing tasks, because the tasks on the terminal are called at the code level, without the need for additional configuration files.

[0158] In the embodiments of the present application, the dependency relationship between tasks can be used to describe: the execution order between tasks, the input parameters and output parameters of each task (also known as the context data of the task). Task a's dependency on task b can be reflected in: the execution order of task b is earlier than the execution order of task a, and the input parameters of task a include the output parameters (execution results) of task b.

[0159] In one implementation, a dependency orchestration file (such as taskDAGs.json) can record the dependencies between tasks as follows:

[0160] In the taskDAGs.json file in the preceding example, the DAGs[] array records the task types (see the value of the "type" parameter), execution order (see the value of the "seq" parameter), input parameters (see the value of the "in" parameter), and output parameters (see the value of the "out" parameter) of the three tasks, "xxxMobileTask", "yyyCloudTask", and "zzzMobileCloudTask".

[0161] Among them, the execution order of tasks, the input parameters and output parameters of tasks can be used to determine the dependencies between tasks.

[0162] In the above example, the execution order of "xxxMobileTask" is before the execution order of "yyyCloudTask", and the execution order of "yyyCloudTask" is before the execution order of "zzzMobileCloudTask", and the input parameter of "yyyCloudTask" is the output parameter of "xxxMobileTask". Therefore, it can be determined that: "zzzMobileCloudTask" depends on "yyyCloudTask", and "yyyCloudTask" depends on "xxxMobileTask".

[0163] In some embodiments, in order to pass the dependencies between tasks to the cloud server 30, the terminal device 20 can serialize the dependencies recorded in the dependency orchestration file (such as taskDAGs.json) into a byte array and put it into the message header of the request message (such as a task execution request). In this way, after receiving the request message, the cloud server 30 can parse the dependencies from the message header. The dependency placed in the message header can be exemplified as follows: taskDAGs:byte[]Codec(in:taskDAGs.json). Among them, "Codec(in:taskDAGs.json)" means encoding the dependency orchestration file taskDAGs.json into a bytecode array.

[0164] In another implementation, the dependency orchestration file can record the dependency between tasks in a graphical manner. For example, as shown in FIG7 , the dependency can be graphically recorded in a configuration file in a format such as YAML\XML, which can indicate that: Task 6 depends on Task 5, Task 5 depends on Task 2, Task 3, and Task 4, and Task 2, Task 3, and Task 4 depend on Task 1. One possible scenario for this dependency is that the execution order of Task 1 is before the execution order of Task 2, Task 3, and Task 4, the execution order of Task 2, Task 3, and Task 4 is before the execution order of Task 5, and the execution order of Task 5 is before the execution order of Task 6; and the output parameter of Task 1 is the input parameter of Task 2, Task 3, and Task 4.

[0165] Access module 301

[0166] The access module 301 may be mainly responsible for authenticating the access of the end-side device and conducting session negotiation.

[0167] Furthermore, the access module 301 may be responsible for receiving tasks, aggregating tasks, and dispatching tasks to the task scheduling and execution module 303 for execution after the dependencies of the tasks are prepared.

[0168] As shown in FIG8 , the access module 301 may include a task access service and a load balancing service, and may execute the following process:

[0169] The task access service can be used to: authenticate the terminal device 20 when it first accesses the service; after authentication, allow the terminal device 20 to establish a communication connection with the cloud server, and parse the request message header according to the protocol specifications and field definitions to obtain the dependency orchestration file, thereby determining the dependency between tasks.

[0170] The Task Access Service can receive multiple tasks via multiple paths, with tasks from the same application being transmitted along the same path. For example, tasks from APP1 (appID1) are transmitted via path 1, tasks from APP2 (appID2) via path 2, and tasks from APP3 (appID3) via path 3. A path here refers to a session link established based on the session mechanism.

[0171] The task access service can aggregate multiple tasks received, convert the aggregated tasks into a request message inside the cloud server 30, and dispatch it to the task execution environment through the load balancing service.

[0172] The load balancing service can be used to: When a program has a task scheduled to the cloud side for the first time, it triggers the task scheduling execution module to allocate a running environment for the program based on the program development project number (ProjectID) and program number (appID), and caches the mapping relationship between the combination of development project number and program number and the running environment.

[0173] The runtime environment can include resources such as servers, virtual machines, or Docker containers that run tasks. For example, the runtime environment assigned to a task can be an NPU server, a GPU server, or a CPU server.

[0174] Task scheduling execution module 303

[0175] As shown in FIG9 , the task scheduling execution module 303 may include two functional modules: task runtime and heterogeneous resource scheduling.

[0176] The task runtime is used to run tasks and can include: the task execution engine on the cloud server 30, drivers and libraries corresponding to cloud-side computing resources (such as GPUs and NPUs), and frameworks for AI algorithms and rendering algorithms. A task runtime can be an on-demand function instance process that can handle task execution requests from one or more terminal devices based on computing load.

[0177] Heterogeneous resource scheduling can be used to schedule the computing resources required for tasks, as well as to achieve rapid resource scheduling and resource recovery based on distributed scheduling, container pooling, container cold start acceleration technology (high-speed snapshot cache recovery), Fourier statistical models, etc. One reference technology implementation is to use serverless (FaaS) technology.

[0178] In order to achieve collaborative scheduling of end-cloud computing power, an embodiment of the present application provides an end-cloud collaborative programming framework.

[0179] As mentioned in the background technology section, there's currently a split computing power scheduling issue between the terminal device side and the cloud server side. This makes it impossible to coordinate the computing power of the terminal and cloud sides, and it's impossible to dynamically choose whether to run a program on the terminal or cloud side based on factors such as the load on the cloud and terminal sides. In this situation, program developers often need to develop both terminal-side and cloud-side programs, and purchase cloud-side computing resources to deploy the cloud-side programs. The development of terminal-side programs requires consideration of the terminal-side operating environment, while the development of cloud-side programs requires consideration of the cloud-side operating environment. These two need to be developed and managed separately, resulting in high development and maintenance costs.

[0180] Through the end-cloud collaborative programming framework provided in the embodiments of the present application, program developers can achieve computing power collaboration between the end side and the cloud side with just one development, and can flexibly schedule tasks in the program to the cloud side or the end side for execution on demand based on factors such as the end-side computing power load, end-side power consumption, end-side temperature control, network status, and cloud-side computing power load, thereby turning the cloud-side computing power into part of the end-side computing power.

[0181] FIG10 shows the end-cloud collaborative programming framework provided by an embodiment of the present application.

[0182] Defining Task Types

[0183] As shown in Figure 10, the end-cloud collaborative programming framework includes interfaces for declaring task types: "MobileTask" and "CloudTask." The "MobileTask" interface can be used to declare a task as an end-side task, while the "CloudTask" interface can be used to declare a task as a cloud-side task.

[0184] The MobileTask and CloudTask interfaces can inherit from the DistributedTask interface (parent class). The DistributedTask interface can contain three methods: the initialization method "init()," the task submission method "submit(long deadline)," and the method "wait()" for waiting for the execution result.

[0185] The MobileTask interface can be implemented as follows: abstract MobileTask implements DistributedTask. It implements its parent class methods: init(), submit(long deadline), and wait(). The submit(long deadline) method is used to submit a task to the terminal device, and the deadline can be used to indicate the submission timeout limit on the terminal.

[0186] The CloudTask interface can be implemented as follows: abstract CloudTask implements DistributedTask. It implements its parent class's methods: init(), submit(long deadline), and wait(). The submit(long deadline) method is used to submit a task to the cloud server, and the deadline is used to indicate the cloud-side submission timeout limit.

[0187] In addition to end-side tasks and cloud-side tasks, the end-cloud collaborative programming framework can also include an interface for declaring the end-cloud task type: "MobileCloudTask".

[0188] Program developers can choose to inherit tasks from MobileTask, CloudTask, or MobileCloudTask based on the actual computing power requirements of each task in the program, and implement the corresponding init(), submit (long deadline), and wait() interfaces.

[0189] For example, the code logic of a mobile-side task "renderAMobileTask" used to implement light-load rendering can be as follows:

[0190] In the above example, the task "renderAMobileTask" inherits MobileTask and implements device-side resource initialization in the init() method and the device-side task waiting mechanism in the wait() method. This declares and defines the task "renderAMobileTask" as a device-side task. The code logic of the task "renderAMobileTask" can also include its own code logic for implementing rendering functions, such as renderA(). renderA() requires relatively low computing power, so it can be designed as a device-side task.

[0191] For example, the code logic of a cloud-side task "reflectionCloudTask" used to implement ray tracing can be as follows:

[0192] In the example above, the task "reflectionCloudTask" inherits CloudTask and implements cloud-side resource initialization (including initializing communication between the client and the cloud) in the init() method. It also implements a cloud-side task timeout mechanism in the wait() method, thereby declaring and defining the task "reflectionCloudTask" as a cloud-side task. The code logic of the task "reflectionCloudTask" can also include its own code logic for implementing ray tracing rendering functions, such as reflection(). Because reflection() requires relatively high computing power, it can be designed as a cloud-side task.

[0193] The end-cloud collaborative programming framework can also include: a sub-interface (subclass) "MobileCloudTask" that inherits the parent interface (parent class) "DistributedTask". "MobileCloudTask" defines an interface for tasks that can be executed on both the end side and the cloud side (i.e., end-cloud tasks). It can be implemented as follows: abstract MobileCloudTask implements DistributedTask. It implements the methods of its parent class: init(), submit(long deadline), and wait(). Among them, the method submit(long deadline) can be used to submit tasks to the cloud server side or the end side, and deadline is used to indicate the submission timeout limit.

[0194] For example, the code logic of a certain end-cloud task "calculateMobileCloudTask" can be as follows:

[0195] In the above example, the task "calculateMobileCloudTask" inherits MobileCloudTask and implements the initialization of end-side resources and cloud-side resources (including initializing communication between the end and the cloud) in the init() method. In the wait() method, the cloud-side or end-side task timeout mechanism is implemented according to whether the actual execution is on the cloud side or the end side, thereby declaring and defining the task "calculateMobileCloudTask" as an end-cloud-side task. The code logic of the task "calculateMobileCloudTask" can also include the code logic of its own functions, such as calculate(). The computing power requirements of calculate() can be met by both end-side computing power resources and cloud-side computing power resources, so its task can be designed as an end-cloud-side task.

[0196] Defining a scheduling strategy

[0197] In addition to task types, the end-cloud collaborative programming framework also supports program developers to customize scheduling policies. The end-cloud collaborative programming framework can also include scheduling policy interfaces such as "TaskDispatchPolicy".

[0198] The parent interface "TaskDispatchPolicy" can include the following method: int policy(string policyExpress) interface. This method's input parameter "policyExpress" is used to receive the customized scheduling policy. The method's return result can be "0" or "1", where "0" defines cloud-side execution and "1" defines client-side execution.

[0199] Program developers can customize scheduling policies, such as customizing the appPolicy interface to implement the scheduling policy interface "TaskDispatchPolicy" and implementing the method int policy(string policyExpress), where the input parameter "policyExpress" is the customized scheduling policy.

[0200] Scheduling policies can be recorded in a scheduling policy file (such as taskDispatchPolicy.json). When a terminal device runs an application, it can read the scheduling policy in this file and execute task scheduling according to the scheduling policy. This file can be packaged in the application software package. The terminal device can obtain this file after downloading the application software package.

[0201] Through the end-cloud collaborative programming framework provided in the embodiment of the present application, program developers can use the interface provided by the framework to schedule tasks to the end side or cloud side for execution on demand, achieving the ability to develop once and run on the end side or cloud side at the same time, significantly improving program development efficiency and shielding the differences in the operating environments of the end side and the cloud side.

[0202] FIG11 exemplarily illustrates the overall process of the end-cloud computing power collaborative scheduling method provided in an embodiment of the present application. The method can be applied to an end-cloud computing power collaborative scheduling system including a terminal device 20 and a cloud server 30. As shown in FIG11 , the method may include the following process:

[0203] S11. The terminal device 20 downloads the software package of the first program.

[0204] The first program can be developed by a program developer based on the end-cloud collaborative programming framework. The first program can be a photo-taking application, a game application, etc. The embodiment of the present application does not limit the type of the first program and the functions provided. The first program can be subdivided into multiple tasks. The task types of tasks can include cloud-side tasks and end-side tasks. Among them, end-side tasks can be tasks scheduled to be executed on the terminal device 20, and cloud-side tasks can be tasks scheduled to be executed on the cloud server 30. The task types of tasks can also include end-cloud tasks, and end-cloud tasks can be scheduled to be executed on the terminal device 20 or on the cloud server 30.

[0205] When writing the first program, the program developer can declare tasks with higher computing power requirements as cloud-side tasks, and tasks with lower computing power requirements as terminal-side tasks.

[0206] The software package of the first program may include: program code, and a dependency arrangement file (such as taskDAGs.json).

[0207] The program code of the first program may include the code for the client-side task and the code for the cloud-side task. Here, the code may refer to machine code (or bytecode). Machine code is generated after source code is compiled and can be directly read and executed by the processor, with high execution speed.

[0208] In the software package of the first program, the code for the cloud-side task and the code for the terminal-side task each have their own characteristics: the code for the terminal-side task may include a first code snippet, which may be a piece of machine code converted from a source code for declaring that the task type is a terminal-side task (such as "extends MobileTask"). The code for the cloud-side task may include a second code snippet, which may be a piece of machine code converted from a source code for declaring that the task type is a cloud-side task (such as "extends CloudTask"). Therefore, the terminal device can identify the task type based on the machine code of the task. The data characteristics of the software package of the first program will be explained later and will not be expanded here.

[0209] The program code of the first program may further include code for a mobile-cloud-side task. The code for the mobile-cloud-side task may include a third code segment, which may be a segment of machine code converted from source code (e.g., "extends MobileCloudTask") for declaring that the task type is a mobile-cloud-side task.

[0210] The dependency arrangement file may be used to indicate the dependency relationships between the various tasks in the first program. For a detailed description of this file, please refer to the previous text and will not be repeated here.

[0211] The program code of the first program may also include: a scheduling policy file (such as taskDispatchPolicy.json), which can be used to record the scheduling policy based on which the terminal device performs task scheduling. For the specific description of this file, please refer to the previous text and will not be repeated here. In some embodiments, the scheduling policy may not be recorded in the scheduling policy file, but may be reflected in the code logic of the scheduling module. For example, the scheduling module can implement the following scheduling policy based on its code logic: perform task scheduling according to the task type, specifically scheduling cloud-side tasks to the cloud-side for execution, and scheduling end-side tasks to the end-side for execution.

[0212] Downloading is only one way for terminal device 20 to obtain the first program software package. Without limitation, terminal device 20 may also obtain the first program software package from another device via Bluetooth, a data cable, satellite communication, or other communication methods, or may obtain the first program software package by reading from an external memory. This embodiment of the present application does not limit the method by which terminal device 20 obtains the first program software package.

[0213] S12. After obtaining the software package of the first program, the terminal device 20 may parse the software package of the first program to obtain a dependency arrangement file, and install the first program.

[0214] The dependencies between tasks recorded in the dependency orchestration file can be sent to the cloud server 30 for cloud server 30 to execute the tasks accordingly. The scheduling policy file can be used by the terminal device 20 to determine how to schedule tasks in the first program. This will be explained in detail later and will not be expanded upon here.

[0215] The first program may also be a program pre-installed on the terminal device 20 when it leaves the factory. In this case, S11-S12 are optional steps. The terminal device 20 may obtain the dependency arrangement file and the scheduling strategy file from the installation directory of the first program.

[0216] S13. Before executing the pending task in the first program, terminal device 20 may determine, based on the scheduling policy, whether the pending task should be scheduled for execution on the device or on the cloud. If the pending task is determined to be scheduled for execution on the device, terminal device 20 may execute S14; if the pending task is determined to be scheduled for execution on the cloud, terminal device 20 may execute S15-S18.

[0217] The pending task can be a task triggered by a user action. For example, in a camera application, the pending task triggered by the user pressing the shutter button is a photo capture task, which is used to save the image captured at the moment of the shutter release as a photo. The pending task can also be a task triggered by the first program according to its own operating logic.

[0218] Scheduling strategies may include, but are not limited to, one or more of the following: "task type strategy", "end-side power consumption strategy", "end-side temperature control strategy", "network status strategy", "cloud-side computing load strategy", etc.

[0219] For example, if the scheduling policy is "task type policy," the terminal device 20 can choose to schedule the task to the device or cloud based on the task type. If a task is a device-side task, it can be scheduled to the device for execution; if it is a cloud-side task, it can be scheduled to the cloud for execution. If a task is a device-cloud task, it can be scheduled to the cloud or the device for execution.

[0220] For detailed description of the scheduling strategy, please refer to the previous article and will not be repeated here.

[0221] S14. The terminal device 20 uses the terminal-side computing resources to execute the task to be executed.

[0222] S15. The terminal device 20 sends a request message to the cloud server 30 to request the cloud server 30 to execute the task to be executed. Accordingly, the cloud server 30 may receive the request message sent by the terminal device 20.

[0223] The request message may include a message header and a message body. The message header may contain a dependency orchestration file, and the message body may contain the machine code for the task to be executed. The dependency orchestration file may be serialized into a byte array and placed in the request message header. The machine code for the task to be executed may be organized into a binary file, such as a .so file. The message body may be a byte array, which can be used to store a file serialized into a byte array (such as a .so file).

[0224] The request message may also carry: input parameters of the task to be executed. The input parameters can be specifically stored in the message body. The input parameters of a task can be the output parameters of other tasks it depends on (such as execution results). Assuming that task a is an image rendering task, the input parameter of task a can be a RAW image captured by the camera, which can be the output parameter of the image capture task that task a depends on. It can be encoded into a byte array and stored in the message body of the request message to be sent to the cloud server 30 for rendering processing.

[0225] For the description of the message header and message body of the request message, please refer to the previous article and will not be repeated here.

[0226] In some embodiments, during the session between the cloud server 30 and the terminal device 20 , the cloud server 30 may save the received dependency orchestration file and the task code.

[0227] Thus, during this session, if the dependency orchestration file for the first program has already been transmitted to the cloud server 30, then if a task is subsequently scheduled to the cloud for execution on the terminal device, the dependency orchestration file need not be transmitted in the request message header, thereby conserving network resources. In other words, during this session, the dependency orchestration file may be transmitted via a request message only when a task is first scheduled to the cloud for execution in the first program.

[0228] Thus, during a session, if the code for a task has already been transferred to the cloud server 30, the terminal device can no longer transmit the task code in the body of a request message when it schedules the task to be executed on the cloud, thereby saving network resources. In other words, during a session, code that depends on the task can be transmitted via a request message only when the task is first scheduled for execution on the cloud.

[0229] S16. The cloud server 30 may extract the task code from the request message.

[0230] In some embodiments, if the request message carries a dependency orchestration file, the cloud server 30 may further extract the dependency orchestration file from the request message.

[0231] S17. Based on the task code, the cloud server 30 uses cloud-side computing resources to execute the task.

[0232] Specifically, cloud-side computing resources such as CPUs, NPUs, or GPUs are allocated based on the task's computing power requirements, and the task is executed. The computing power requirements of a task are determined by its type. Specifically, if a task is declared as a device-side task, it has a low computing power requirement; if a task is declared as a cloud-side task, it has a high computing power requirement.

[0233] If the terminal device 20 is accessing for the first time, an operating environment is allocated to the terminal device 20. All subsequent tasks received from the terminal device 20 are routed to the previously allocated operating environment.

[0234] S18. The cloud server 30 returns a response message to the terminal device 20 to feed back the task execution result to the terminal device 20. Accordingly, the terminal device 20 can receive the response message sent by the cloud server 30 and extract the task execution result from the response message.

[0235] The response message may include a message header and a message body. The message header may contain the dependency orchestration file, and the message body may contain the task execution results. The dependency orchestration file may be serialized into a byte array and placed in the response message header. The message body may be a byte array that can be used to store the task execution results serialized into a byte array.

[0236] The execution result of a task can be an input parameter for other tasks that depend on it. For example, if task a is an image rendering task, the execution result of task a can be a rendered image. This image can be encoded into a byte array and stored in the body of the response message to be fed back to terminal device 20. Terminal device 20 can use the rendered image as an input parameter for the display task to display the rendered image. The display task depends on task a.

[0237] For the description of the message header and message body of the response message, please refer to the previous article and will not be repeated here.

[0238] In an embodiment of the present application, the cloud server 30 may receive multiple request messages sent by the terminal device 20. These multiple request messages are respectively used to request the cloud server 30 to perform multiple different tasks. After receiving these multiple request messages, the cloud server 30 may restore the execution order of these multiple tasks according to the dependency orchestration file, and cache these multiple tasks in the task queue according to the task execution order described in the dependency orchestration file to record the execution order of these multiple tasks. In this way, when the cloud server 30 wants to run these tasks, it can retrieve the tasks from the task queue in order and execute them to ensure that the task execution order is correct.

[0239] The end-cloud computing power collaborative scheduling method provided in the embodiments of this application can make cloud-side computing power part of the end-side computing power, achieving collaborative scheduling of end-side and cloud-side tasks. While solving the problem of insufficient end-side computing power, it also avoids placing all logic and computing power on the cloud side, resulting in idle end-side computing power and high cloud-side costs. Moreover, the task scheduling strategy can be customized by the program developer. In addition to the computing power requirements of each task, factors such as end-side power consumption, end-side temperature control, network status, and cloud-side computing power load can also be considered to implement task scheduling, so that task scheduling can be more flexible and adapt to different scenarios.

[0240] The embodiment of the present application also provides a transmission rate adjustment method, which can optimize the communication between the end and the cloud, effectively improve the end-cloud communication bandwidth, and reduce the task transmission delay, so that more tasks that meet the conditions can be scheduled to the cloud side for execution.

[0241] Existing network transmission problems include: the HTTP network transmission protocol has a slow initial rate ramp-up, wide area network transmission faces complex weak network environments (such as random packet loss, RTT jitter, bandwidth fluctuations, etc.), and the existing HTTP transmission protocol is prone to large and small fluctuations in available bandwidth assessment when faced with weak network signals (signals indicating poor network conditions), which in turn leads to low transmission efficiency (network bandwidth utilization <40%).

[0242] The communication method provided in the embodiments of the present application can provide optimization effects in the following aspects:

[0243] 1. User experience-aware network QoS scheduling: Perceives the differentiated demands of different tasks for data access and processing time (deadline latency, task priority, etc.), and accurately delivers network QoS requirements for user experience transformation across the entire end-to-end path.

[0244] 2. Transmission optimization: Fast startup and accurate bandwidth assessment for weak networks ensure high bandwidth utilization >80% in WAN communications. Adaptive learning of the real-time WAN network environment (bandwidth changes, packet loss, jitter, etc.) specifically resists various types of network loss interference on the access network. Even with a packet loss rate >3%, bandwidth utilization >80% is guaranteed. Multipath transmission solves the problem of insufficient bandwidth on a single path or poor network quality, with different tasks taking different paths. The transmission protocol obtains the maximum available bandwidth through active detection and reports it to the scheduling module.

[0245] As shown in Figure 12, the communication proxy service on the terminal device may further include: a statistics module, a decision module, and a network scheduling module. The statistics module can be used to create a network statistics model for the task based on the task's identification information (such as the task ID), and to calculate the task's packet size, maximum latency, and transmission time (i.e., transmission time). Based on the historical transmission rate and the current available bandwidth, it determines the available upload network bandwidth K and network bandwidth utilization rate R allocated to the task in the next cycle. The decision module can be used to decide which transmission method to use, whether to unpack the task, etc. based on information such as packet size, maximum latency, and transmission time (i.e., transmission time), and inform the network scheduling module of the decision results. The network scheduling module can be used to allocate and adjust network bandwidth for the task based on the task's latency requirements, packet size, and available network bandwidth.

[0246] Specifically, the transmission rate adjustment provided in the embodiment of the present application may include S21 and S22.

[0247] S21. The communication proxy service provides the scheduling policy service with cloud-side computing load information, which is used by the scheduling policy service to assess the network status and decide whether to schedule tasks to the cloud side for execution. The specific implementation process is shown in Figure 13 and may include S31-S33, where:

[0248] S31. The scheduling policy service may send a query request to the statistics module. The query request may carry the identification information of the task to query the available upload network bandwidth and network bandwidth usage allocated to the task in the next cycle.

[0249] S32. The statistics module may determine the available upload network bandwidth K and the network bandwidth usage R allocated to the task in the next cycle based on the historical transmission rate and the current available bandwidth, and return them to the scheduling policy service.

[0250] S33. Based on the task's packet size S, available upload network bandwidth K, and network bandwidth utilization R, the scheduling policy service calculates the transmission time T = S / (K*R). If the transmission time T is less than the task's maximum latency, the scheduling policy service schedules the task to the cloud for execution. The task's maximum latency is the maximum latency it can tolerate, reflecting its latency requirements.

[0251] S22. Provides a high-performance, reliable protocol transmission framework for the task execution engine module on the end side. The specific implementation process, as shown in FIG14 , may include S41-S45, where:

[0252] S41. The task execution engine module transmits the task cloud-side execution request to the statistics module.

[0253] S42. The statistics module creates a network statistics model for the task based on the identification information of the task (such as the task ID), which can count the data packet size of the task, the maximum delay of the task, the transmission time of the task (i.e., the transmission time), etc.

[0254] S43. The statistics module may further transmit a cloud-side execution request for the task to the decision module. The request may carry information such as the task data packet size, the maximum delay of the task, and the transmission time of the task (i.e., the transmission time).

[0255] S44. The decision module can decide which transmission method to use, whether to unpack the task, etc. based on information such as the data packet size, the maximum delay of the task, the transmission time of the task (i.e., the transmission time), and inform the network scheduling module of the decision result.

[0256] S45. The network scheduling module adjusts the transmission rate for the task based on the task's latency requirements, task packet size, available network bandwidth, etc. The key technologies used may include one or more of the following:

[0257] Fast link building method

[0258] Existing transmission protocols have a low initial rate and require a slow start process to slowly ramp up, which is not conducive to reducing network transmission latency. The fast link establishment method provided in the embodiments of the present application can improve this problem. Its core concept includes: the first round trip time (RTT) uses a set high rate (based on service expectations or connection history cache information) as the test rate, and the initial rate calculation is completed within 1 RTT based on network feedback.

[0259] Specifically, as shown in FIG15 , the fast link establishment method provided in the embodiment of the present application may include:

[0260] S51. At the beginning of transmitting the task data packet (mainly including the task code and context data) to the cloud server (such as the first RRT of the data packet transmission), the terminal device transmits data at the test rate (selected rate) v_test.

[0261] v_test is a larger transmission rate, and its value is greater than v_start mentioned later.

[0262] In some embodiments, the test rate v_test selected for a task can be determined based on business expectations: v_test = data_size / Tmax, where data_size represents the size of the task's data packet and Tmax represents the task's maximum latency. Specifically, the end-cloud collaborative programming framework can provide an interface that program developers can implement to configure the latency requirements of each task.

[0263] In some embodiments, the test rate v_test selected for a task is also determined based on the historical buffer information. Here, the historical buffer information can be the test rate once selected for the historical transmitted tasks.

[0264] S52. The terminal device receives the confirmation (acknowledgement, ACK) returned by the cloud server for the first round of data transmission.

[0265] S53. The terminal device estimates the data receiving rate v_recv of the cloud server based on the reception time of the confirmation (ACK), and uses it as the starting rate v_start of the next data transmission.

[0266] Specifically, the data reception rate v_recv can be estimated as follows: v_recv = m / Δt, where m represents the amount of data sent in the first round, such as m KB, and m is a positive integer; Δt = t2 - t1, where t1 represents the start transmission time of the first round of data transmission, and t2 represents the reception time of the ACK for the first round of data transmission. In other words, v_start = m / Δt.

[0267] If the ACK is returned to the terminal device before the first round of data (m KB) is sent, the terminal device can stop sending the remaining part of the first round of data at v_test in advance. In this way, the transmission rate can be adjusted to v_start earlier and adapt to the network faster.

[0268] If the terminal device has not received an ACK after the first round of data transmission (m KB) is completed, the terminal device can stop sending subsequent data and wait for the ACK to be returned. In this way, network congestion and packet loss can be avoided.

[0269] S54. The terminal device adjusts the transmission rate to the starting rate v_start.

[0270] The rate adjustment of subsequent data transmission can be performed by a congestion control method. Specifically, the terminal device can use a congestion control algorithm to adjust the transmission rate starting from the second RTT.

[0271] Congestion control methods

[0272] The existing congestion control method can include, as shown in Figure 16: based on the delay information, packet loss rate, receiving rate, data packet ACK and other information sent by the network information sampling and processing module, the network feedback-driven rate adjustment module can determine the available bandwidth B to instruct the service rate to be transmitted according to the available bandwidth B.

[0273] However, WAN transmission faces complex weak network environments (such as random packet loss, RTT jitter, and bandwidth fluctuations). Weak network signals are prone to occur and do not accurately reflect the true network status. The existing congestion control method in Figure 16 causes the estimated available bandwidth B to fluctuate widely and be too small, leading to low transmission efficiency.

[0274] The congestion control method provided in the embodiment of the present application can improve this problem. A new weak network identification and information correction module is introduced to resist the interference of weak network noise in the wide area network, and a traffic feature-aware congestion control module is introduced to adjust the rate in time according to the traffic characteristics of the application to ensure transmission efficiency.

[0275] Specifically, as shown in FIG17 and FIG18 , the fast link establishment method provided in the embodiment of the present application may include:

[0276] RTT-granular congestion control (S61-S63)

[0277] S61. The terminal device can obtain data such as latency, packet loss rate, reception rate, and packet ACKs through the network information sampling and processing module. The weak network identification and information correction module can then analyze this data to identify the noise intensity of weak network signals, such as through adaptive dynamic learning of RTT jitter and the noise intensity of random packet loss. The greater the noise intensity, the less reliable the conclusion of a weak network signal, and corrections are required to ensure that the output of available bandwidth B more closely reflects the actual network status. For example, if only one or two packets are lost out of 100 consecutive data packets, the noise intensity of this packet loss is high and does not reflect a poor network status. For another example, if only one packet out of 100 consecutive data packets has an abnormally high RTT, this RTT jitter does not reflect a poor network status.

[0278] S62. If the noise intensity is greater than the first threshold, the terminal device may correct the abnormal data causing the noise through the weak network identification and information correction module, so that the abnormal data is close to the average level after correction.

[0279] S63. The terminal device can determine the available bandwidth B based on the corrected delay information, packet loss rate, receiving rate, data packet ACK and other data through the rate adjustment module driven by network feedback.

[0280] Traffic feature-aware congestion control (S64-S67)

[0281] S64. The terminal device may record the available bandwidth B determined in S63 through the historical bandwidth cache module.

[0282] S65. The terminal device can identify the traffic low period and traffic peak period during the execution of the first program through the traffic feature recognition module.

[0283] A low traffic period may refer to a stage when a terminal device transmits data to a cloud server at a low rate, such as when transmitting a task with a small amount of data. A high traffic period may refer to a stage when a terminal device transmits data to a cloud server at a high rate, such as when transmitting a task with a large amount of data. In other words, a low traffic period may refer to a stage when a terminal device transmits code for a first task to a server, where the amount of data in the code for the first task is greater than the first amount of data. A high traffic period may refer to a stage when a terminal device transmits code for a second task to a server, where the amount of data in the code for the second task is less than the second amount of data; and the second amount of data is less than or equal to the first amount of data.

[0284] In some embodiments, the terminal device can traverse the data volume of each task code in the first program, and determine the transmission period of the task code with a large data volume as the traffic peak period, and determine the transmission period of the task code with a small data volume as the traffic valley period. In other embodiments, the end-cloud collaborative programming framework can provide an interface, and program developers can implement this interface to write the data volume of each task to a specified variable or execution storage area. In this way, once the terminal device reads the data volume of each task from the specified variable or execution storage area, it can determine which tasks are transmitted during the traffic valley period.

[0285] S66. During the low traffic period, the terminal device can detect the network bandwidth through the bandwidth rapid upward detection module, and the detected network bandwidth is recorded as the upward detection bandwidth.

[0286] Specifically, the terminal device can fill the low-rate data with patch data (such as data "0" or other data) to form a high-rate data transmission. If timely feedback is received from the network, the device can further increase the padding until the network feedback times out. The highest rate received in a timely manner from the network can be determined as the upward detection bandwidth. This process can also be called "upward detection of network bandwidth."

[0287] S67. During traffic peaks, such as when burst traffic arrives, the terminal device can adjust the transmission rate based on the upper probe bandwidth and the maximum cached available bandwidth B (also known as the historical maximum available bandwidth) through the burst traffic sending rate calculation module. The adjusted transmission rate can be the smaller of the upper probe bandwidth and the maximum cached available bandwidth B.

[0288] FIG19 shows a terminal device 200 provided in an embodiment of the present application.

[0289] In the embodiment of the present application, the device type of the terminal device 200 can be any of a mobile phone, a tablet computer, a handheld computer, a desktop computer, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a cellular phone, a personal digital assistant (PDA), as well as smart home devices such as smart large screens and smart speakers, wearable devices such as smart bracelets, smart watches, and smart glasses, extended reality (XR) devices such as augmented reality (AR), virtual reality (VR), and mixed reality (MR), in-vehicle devices or smart city devices, etc.

[0290] As shown in FIG19 , the terminal device 200 may include: a processor 110, a memory 120, a display screen 130, a display driver integrated circuit (DDIC), an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone jack 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, and a subscriber identification module (SIM) card interface 195. The sensor module 180 may include a gyroscope sensor 180B, an acceleration sensor 180E, and a touch sensor 180K. The various components in the terminal device 200 may be connected via a bus.

[0291] The processor 110 may be one or more, and they may be integrated into an integrated circuit of a system on chip (SOC). SOC is a system-on-chip. The processor 110 may include a central processing unit (CPU), a graphics processing unit (GPU), and a display driver integrated circuit (DDIC). The CPU may be an application processor (AP). The CPU and GPU may be used to render and synthesize images to be displayed on the display screen 130. The processor 110 may also include a neural-network processing unit (NPU), a modem processor, and the like.

[0292] The processor 110 may include one or more interfaces, such as an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input and output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface.

[0293] The processor 110 may be provided with a cache memory for storing instructions or data that have just been used or are cyclically used by the processor 110. If the processor 110 needs to use the instruction or data again, it can be directly called from the cache memory, which can reduce the waiting time of the processor 110 and improve the efficiency of program operation.

[0294] The memory 120 may include a program storage area and a user data storage area. The program storage area may store an operating system and one or more application programs (such as game applications), and the data storage area may store data (such as photos and contacts) created by the user during the use of the terminal device 200. The memory 120 may be a high-speed random access memory or a non-volatile memory, such as a disk, flash memory, or universal flash storage (UFS). The memory 120 may also be an external memory card, such as a Micro SD card.

[0295] The memory 120 may also store code instructions for the display partition refresh method provided in the embodiment of the present application. When the processor 110 reads the code instructions from the memory 120 and runs the code instructions, the terminal device 200 may execute the method.

[0296] The memory 120 may also be integrated with the processor 110 into an integrated circuit of a SOC.

[0297] The terminal device 200 can realize the display function through the SOC, DDIC, and display screen 130.

[0298] The display screen 130 may have a partitioned frequency conversion function and may be the display screen 130 described in the aforementioned embodiment. The display screen 130 may include a display panel, a timing controller (TCON), etc., wherein the display panel may include multiple pixel units, and may be an organic light-emitting diode (OLED) display panel, each pixel unit including an OLED. The TCON is mainly used to connect the GPU or SOC with the display panel. After receiving image data or control signals from the GPU or SOC, the TCON sets the timing of the control signal and the data signal according to the relevant data or signals, and transmits the control signal and the data signal to the display driver circuit according to the timing, thereby achieving the purpose of driving the display panel to display graphics.

[0299] The display driver integrated circuit (DDIC) serves as the control core of display screen 130, driving it and receiving data, such as image data and instructions, from the SOC (processor 110). The DDIC transmits drive signals and data to the display panel of display screen 130 via electrical signals, thereby controlling screen brightness and color, allowing image information such as letters and images to appear on the screen and completing screen refresh.

[0300] The image data to be displayed sent by the SOC to the DDIC may be sent to a frame buffer for storage to complete display transmission (or image transmission). Then, the DDIC retrieves the image data from the frame buffer and drives the display screen 130 for display.

[0301] The wireless communication function of the terminal device 200 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0302] Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in terminal device 200 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In other embodiments, the antennas can be used in conjunction with a tuning switch.

[0303] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied to the terminal device 200. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0304] The modem processor may include a modulator and a demodulator. The modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to the speaker 170A, the receiver 170B, etc.) or displays an image or video through the display screen 130. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0305] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the terminal device 200. The wireless communication module 160 can be one or more devices that integrate at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, frequency modulate it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0306] In some embodiments, the antenna 1 of the terminal device 200 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the terminal device 200 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0307] The terminal device 200 can realize the shooting function through the ISP, camera 193, video codec, GPU, display screen 130 and application processor.

[0308] The ISP processes data fed back by camera 193. For example, when taking a photo, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, which is then passed to the ISP for processing and converted into a visible image. The ISP can also perform algorithmic optimization on image noise, brightness, and skin tone. It can also optimize parameters such as exposure and color temperature of the captured scene. In some embodiments, the ISP can be located within camera 193.

[0309] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then passes the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the terminal device 200 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0310] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the terminal device 200 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0311] Video codecs are used to compress or decompress digital video. Terminal device 200 may support one or more video codecs. This allows terminal device 200 to play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, and MPEG4.

[0312] The NPU is a neural network (NN) computing processor. Drawing on the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it rapidly processes input information and can continuously self-learn. The NPU enables intelligent cognitive applications in the terminal device 200, such as image recognition, face recognition, speech recognition, and text comprehension.

[0313] The terminal device 200 can implement audio functions such as music playback and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0314] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be provided in the processor 110, or some functional modules of the audio module 170 can be provided in the processor 110.

[0315] The speaker 170A, also called a "speaker", is used to convert audio electrical signals into sound signals. The terminal device 200 can listen to music or listen to hands-free calls through the speaker 170A.

[0316] The receiver 170B, also called a "handset", is used to convert audio electrical signals into sound signals. When the terminal device 200 receives a call or voice message, the user can hear the voice by placing the receiver 170B close to the ear.

[0317] Microphone 170C, also known as "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to the microphone 170C to input the sound signal into the microphone 170C. The terminal device 200 can be provided with at least one microphone 170C. In other embodiments, the terminal device 200 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the terminal device 200 can also be provided with three, four or more microphones 170C to realize sound signal collection, noise reduction, and can also identify the source of sound, realize directional recording function, etc.

[0318] The headphone jack 170D is used to connect a wired headphone and can be the USB interface 130 or a 3.5mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0319] Keys 190 include a power button, volume button, and the like. Keys 190 can be mechanical or touch-sensitive. Terminal device 200 can receive key inputs and generate key signal inputs related to user settings and function control of terminal device 200. Motor 191 can generate vibration prompts. SIM card interface 195 is used to connect a SIM card. A SIM card can be connected to and disconnected from terminal device 200 by inserting or removing it from SIM card interface 195.

[0320] The structure shown in FIG19 does not constitute a specific limitation on terminal device 200. Terminal device 200 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0321] Figure 20 shows a server 300 provided in an embodiment of the present application. Server 300 may be the cloud server 30 mentioned in the previous embodiment. As shown in Figure 20, server 300 may include: a processor 210, a memory 220, input and output devices 230, a communication module 240, etc. These components may be coupled via a bus.

[0322] The server 300 may have powerful computing resources, and the processor 210 thereon may include one or more processors with powerful computing power, such as a central processing unit (CPU), a neural network processing unit (NPU), a graphics processing unit (GPU), etc.

[0323] The processor 210 may include one or more interfaces, such as an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input and output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface.

[0324] The processor 210 may be provided with a cache memory for storing instructions or data that have just been used or are cyclically used by the processor 210. If the processor 210 needs to use the instruction or data again, it can be directly called from the cache memory, which can reduce the waiting time of the processor 210 and improve the efficiency of program operation.

[0325] The processor 210 may also be connected to an external memory. The memory may be a high-speed random access memory or a non-volatile memory, such as a disk, flash memory, universal flash memory (UFS), etc. The memory may also be an external memory card, such as a Micro SD card.

[0326] Processor 210 is the computing core of server 300 and has powerful computing capabilities. It is coupled to memory 220 and can be used to read and execute computer-readable instructions in memory 220, running the operating system and various programs. Specifically, CPU 210 can be used to call programs stored in memory 220, such as the implementation program of the end-cloud collaborative computing power scheduling method provided in the embodiments of the present application, and execute the instructions contained in the program.

[0327] Memory 220 may include high-speed random access memory, non-volatile memory, such as a disk, flash memory, or other non-volatile solid-state storage device. Memory 220 may be used to store various software programs and multiple sets of instructions. Memory 220 may store an operating system, such as an operating system such as Linux. Memory 220 may also store one or more programs, such as programs involved in patch creation, such as a compiler or linker. Memory 220 may also store an implementation program for the end-cloud collaborative computing power scheduling method provided in the embodiments of the present application.

[0328] The input and output devices 230 may include a display screen, a keyboard, a mouse and other devices, which can be used to receive user input and output program execution results to the user.

[0329] The communication module 240 may include a wired communication module and a wireless communication module. The wired communication module may support wired communication protocols such as universal serial bus (USB), serial port, Ethernet, and other protocols, and communicate with other devices through physical communication cables. The wireless communication module may include 2G / 3G / 4G / 5G wireless communication modules, Wi-Fi communication modules, etc. The wireless communication module receives electromagnetic waves via an antenna, modulates and filters the electromagnetic wave signals, and sends the processed signals to the CPU 210. The wireless communication module may also receive signals to be transmitted from the CPU 210, modulate and amplify them, and convert them into electromagnetic waves for radiation via the antenna.

[0330] The structure shown in FIG20 does not constitute a limitation on the server 300. The server 300 may include more or fewer components than shown, or may combine or separate some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0331] The steps of the method or algorithm described in conjunction with the disclosure of the embodiments of the present application can be implemented in hardware or by executing software instructions by a processor. The software instructions can be composed of corresponding software modules, which can be stored in RAM, flash memory, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and storage medium can be located in an ASIC. In addition, the ASIC can be located in a transceiver or a relay device. Of course, the processor and storage medium can also be present in a wireless access network device or a user device as discrete components.

[0332] Those skilled in the art will appreciate that in one or more of the above examples, the functions described in the embodiments of the present application can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any media that facilitates the transmission of computer programs from one place to another. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0333] The above specific implementation methods further explain in detail the purpose, technical solutions and beneficial effects of the embodiments of the present application. It should be understood that the above are only specific implementation methods of the embodiments of the present application and are not intended to limit the scope of protection of the embodiments of the present application. Any modifications, equivalent replacements, improvements, etc. made on the basis of the technical solutions of the embodiments of the present application should be included in the scope of protection of the embodiments of the present application.

Claims

1. A method for scheduling computing power between end and cloud, characterized in that: The method is applied to a terminal device in a first communication system, the first communication system comprising: the terminal device and a server, a first program is installed on the terminal device, a software package of the first program comprises a program code, the program code comprises a cloud-side task code and a terminal-side task code; the cloud-side task code comprises a first code segment, the first code segment is used to declare that the task type is a cloud-side task; the terminal-side task code comprises a second code segment, the second code segment is used to declare that the task type is a terminal-side task; The method comprises: The terminal device determines whether to schedule the to-be-executed task in the first program to be executed on the server according to a scheduling strategy; the scheduling strategy is formulated based on one or more of the following factors: task type, power consumption of the terminal device, temperature of the terminal device, network status between the terminal device and the server, and cloud-side computing load; If the task is scheduled to be executed on the server, the terminal device sends the code of the task to be executed to the server, requesting the server to use the cloud-side computing resources to execute the task to be executed; otherwise, the terminal device uses the terminal-side computing resources to execute the task to be executed; the terminal-side computing resources include the processor resources on the terminal device, and the cloud-side computing resources include the processor resources on the server; The terminal device receives the execution result of the task to be executed returned by the server.

2. The method according to claim 1, characterized in that The program code includes a code for an end-cloud task, and the code for the end-cloud task includes a third code segment, and the third code segment is used to declare that the task type is an end-cloud task.

3. The method according to claim 1 or 2, characterized in that The software package of the first program also includes: a first file, wherein the first file is used to indicate the dependency relationship between various tasks in the first program.

4. The method according to claim 3, characterized in that Also includes: The terminal device sends the first file to the server, so that the server can determine the dependency relationship of the tasks to be executed.

5. The method according to claim 3 or 4, characterized in that The first file is specifically used to indicate the dependency relationship between the tasks in the first program by recording the execution order and context data of the tasks; the context data includes: input parameters and output parameters of the tasks.

6. The method according to any one of claims 1 to 5, characterized in that The scheduling strategy includes one or more of the following: task type strategy, end-side power consumption strategy, end-side temperature strategy, network status strategy, and cloud-side computing load strategy; wherein, The type strategy includes: scheduling the cloud-side task to be executed on the server, and scheduling the terminal-side task to be executed on the terminal device; The terminal side power consumption strategy includes: if the remaining power of the terminal device is higher than the first power, scheduling the to-be-executed task to be executed on the server; otherwise, scheduling the to-be-executed task to be executed on the terminal device; The terminal-side temperature strategy includes: if the temperature of the terminal device is lower than a first temperature, scheduling the task to be executed on the server for execution; otherwise, scheduling the task to be executed on the terminal device for execution; The network status strategy includes: if the communication quality between the terminal device and the server satisfies one or more of the following: the communication bandwidth is greater than the first bandwidth, and the communication delay is less than the first delay, then scheduling the task to be executed on the server, otherwise scheduling the task to be executed on the terminal device; The cloud-side computing power load strategy includes: if the occupancy of the cloud-side computing power resources is less than the first ratio, scheduling the task to be executed on the server; otherwise, scheduling the task to be executed on the terminal device.

7. The method according to any one of claims 1 to 6, characterized in that The software package of the first program also includes: a second file, and the second file is used to record the scheduling strategy.

8. The method according to any one of claims 1 to 7, characterized in that The terminal device sends the code of the task to be executed to the server, specifically including: The terminal device sends a first request message to the server, wherein the message header of the first request message carries a byte array serialized in The first file, the message body of the first request message carries the code of the task to be executed.

9. The method according to claim 8, characterized in that The first request message also carries input parameters of the task to be executed.

10. The method according to claim 9, characterized in that The input parameters of the task to be executed include: the execution result of the task on which the task to be executed depends.

11. The method according to any one of claims 1 to 10, characterized in that The terminal device sends the code of the task to be executed to the server, specifically including: The terminal device transmits data at a test rate v_test within a first round trip time RRT of sending the code of the task to be executed to the server; The terminal device receives a confirmation ACK for the first round of data transmission returned by the server; The terminal device adjusts the transmission rate of the code for sending the task to be executed to the starting rate v_start based on the reception time of the ACK, where v_start=m / Δt, where m represents the amount of data sent in the first round; Δt=t2-t1, where t1 represents the starting transmission time of the first round of data sent, and t2 represents the reception time of the ACK for the first round of data sent; The test rate v_test is greater than the start rate v_start.

12. The method according to any one of claims 1 to 11, characterized in that The method comprises: The terminal device obtains one or more data of delay information, packet loss rate, and receiving rate, and analyzes the data to identify the noise intensity of the weak network signal; including: the noise intensity of RTT jitter and random packet loss; If the noise intensity is greater than a first threshold, the terminal device corrects the abnormal data causing the noise; The terminal device determines the available bandwidth based on one or more of the corrected delay information, packet loss rate, and receiving rate.

13. The method according to claim 12, characterized in that The method further comprises: The terminal device identifies a traffic trough period during the running of the first program, and upwardly detects the network bandwidth during the traffic trough period and obtains the upward detection bandwidth; the traffic trough period refers to a stage in which the terminal device transmits the code of the first task to the server, and the data volume of the code of the first task is greater than the first data volume; During traffic peak periods, the terminal device adjusts the transmission rate according to the upward detection bandwidth and the historical maximum available bandwidth, and the adjusted transmission rate is the smaller of the upward detection bandwidth and the historical maximum available bandwidth; the traffic peak period refers to the stage when the terminal device transmits the code of the second task to the server, and the data volume of the code of the second task is less than the second data volume; the second data volume is less than or equal to the first data volume.

14. A method for scheduling computing power between end and cloud, characterized in that: The method is applied to a server in a first communication system, the first communication system comprising: the terminal device and the server, the terminal device having a first program installed thereon, the software package of the first program comprising a program code, the program code comprising a cloud-side task code and a terminal-side task code; the cloud-side task code comprising a first code segment, the first code segment being a machine code converted from a source code used to declare that the task type is a cloud-side task; the terminal-side task code comprising a second code segment, the second code segment being a machine code converted from a source code used to declare that the task type is a terminal-side task; The method comprises: The server receives the code of the task to be executed in the first program sent by the terminal device; the task to be executed is scheduled by the terminal device to be executed on the server; The server executes the task to be executed by utilizing cloud-side computing resources; the cloud-side computing resources include processor resources on the server; The server sends the execution result of the task to be executed to the terminal device.

15. The method according to claim 14, characterized in that The program code includes a code for an end-cloud task, and the code for the end-cloud task includes a third code segment, and the third code segment is used to declare that the task type is an end-cloud task.

16. The method according to any one of claims 14 to 15, characterized in that The software package of the first program also includes: a first file, wherein the first file is used to indicate the dependency relationship between various tasks in the first program.

17. The method according to claim 16, characterized in that Also includes: The server receives a first file sent by the terminal device, and determines the dependency relationship of the tasks to be executed according to the first file.

18. The method according to claim 16 or 17, characterized in that The first file is specifically used to indicate the dependency relationship between the tasks in the first program by recording the execution order and context data of the tasks; the context data includes: input parameters and output parameters of the tasks.

19. The method according to any one of claims 14 to 18, characterized in that The scheduling strategy includes one or more of the following: task type strategy, end-side power consumption strategy, end-side temperature strategy, network status strategy, and cloud-side computing load strategy; wherein, The type strategy includes: scheduling the cloud-side task to be executed on the server, and scheduling the terminal-side task to be executed on the terminal device; The terminal side power consumption strategy includes: if the remaining power of the terminal device is higher than the first power, scheduling the to-be-executed task to be executed on the server; otherwise, scheduling the to-be-executed task to be executed on the terminal device; The terminal-side temperature strategy includes: if the temperature of the terminal device is lower than a first temperature, scheduling the task to be executed on the server for execution; otherwise, scheduling the task to be executed on the terminal device for execution; The network status strategy includes: if the communication quality between the terminal device and the server satisfies one or more of the following: the communication bandwidth is greater than the first bandwidth, and the communication delay is less than the first delay, then scheduling the task to be executed on the server, otherwise scheduling the task to be executed on the terminal device; The cloud-side computing power load strategy includes: if the occupancy of the cloud-side computing power resources is less than the first ratio, scheduling the task to be executed on the server; otherwise, scheduling the task to be executed on the terminal device.

20. The method according to any one of claims 14 to 19, characterized in that The software package of the first program also includes: a second file, and the second file is used to record the scheduling strategy.

21. The method according to any one of claims 14 to 20, characterized in that The server receives the code of the task to be executed in the first program sent by the terminal device, specifically including: The server receives a first request message sent by the terminal device, wherein a message header of the first request message carries the first file serialized into a byte array, and a message body of the first request message carries a code of the task to be executed.

22. The method according to claim 21, characterized in that The first request message also carries input parameters of the task to be executed.

23. The method of claim 22, wherein: The input parameters of the task to be executed include: the execution result of the task on which the task to be executed depends.

24. A terminal device, characterized in that: It comprises a processor and a memory; wherein the memory is coupled to the processor, the memory is used to store computer program code, the computer program code comprises computer instructions, and when the processor executes the computer instructions, the terminal device executes the method as described in any one of claims 1-13.

25. A server, characterized in that: It comprises a processor and a memory; wherein the memory is coupled to the processor, the memory is used to store computer program code, the computer program code comprises computer instructions, and when the processor executes the computer instructions, the server executes the method as described in any one of claims 14-23.

26. A communication system, characterized in that: It comprises a terminal device and a server; wherein, the terminal device is the terminal device described in claim 24, and the server is the server described in claim 25.

27. A computer-readable storage medium comprising instructions, characterized in that: When the instruction is executed on the terminal device, the terminal device executes the method according to any one of claims 1 to 13.

28. A computer-readable storage medium comprising instructions, characterized in that: When the instruction is executed on the server, the server is caused to execute the method according to any one of claims 14 to 23.

Images