A task control method and system in an edge computing scenario

Abstract

The application discloses a task control method and system in an edge computing scenario, comprising: obtaining resource information of an edge node where a task is located, the resource information comprising whether a disk is normal, a network line connection state and a mounting point corresponding to each disk; obtaining container information occupied by the task when running on the edge node, and constructing a task running resource tree based on the container information and the resource information; obtaining resource request information expected by the task running from a cloud end through a communication method based on a publish / subscribe model, and constructing a task request resource tree based on the resource request information; comparing the task running resource tree and the task request resource tree to obtain tuning information; and tuning based on the tuning information through a depth-first traversal method to ensure task synchronization. The application can shorten the tuning time to 1 minute.

Description

Technical Field

[0001] This invention belongs to the field of edge computing technology, specifically relating to a task control method and system in edge computing scenarios. Background Technology

[0002] In traditional centralized cloud container deployment systems, container operation requires prerequisites such as disks, networks, and images. Furthermore, once a container is deployed on an edge node, if the container or its dependencies fail, the deployment system will not redeploy based on the latest node state. However, edge computing nodes face inherent constraints such as limited hardware and network resources, inability to pool resources, and insufficient scalability. Additionally, edge nodes are heterogeneous and unstable. Therefore, during task execution, resources such as hard drives and network connections that the task depends on may fail at any time, necessitating awareness of changes in edge node resources and dynamic task adjustments. Traditional centralized cloud task deployment controllers, such as Kubernetes, perform task tuning based on the assumption that node resources are reliable. If a hard drive assigned to a task fails, the Kubernetes task controller will not automatically remove the failed disk or restart the task. Summary of the Invention

[0003] To address the above problems, this invention proposes a task control system method and system for edge computing scenarios, which can dynamically adjust the task deployment topology on edge nodes when edge node resources change. The technical solution adopted by this invention to solve the above technical problems is as follows:

[0004] A task control method for edge computing scenarios includes the following steps:

[0005] S1, obtain the resource information of the edge node where the task is located. The resource information includes whether the disk is normal, the network line connectivity status, and the mount point corresponding to each disk.

[0006] S2, obtain the container information occupied by the task running on the edge node, and construct the task running resource tree based on the container information and the resource information in step S1;

[0007] S3 obtains the resource request information required for task execution from the cloud through a publish-subscribe model-based communication method, and constructs a task request resource tree based on the resource request information;

[0008] S4, compare the task execution resource tree and the task request resource tree to obtain tuning information;

[0009] S5 utilizes a depth-first traversal method to perform tuning based on tuning information to ensure task synchronization.

[0010] Step S2 includes the following steps:

[0011] S2.1, Obtain all containers used during task execution;

[0012] S2.2, Obtain the resource usage information of all containers in step S2.2, the resource usage information including network lines, disks occupied by containers, mount points and parameters used by containers;

[0013] S2.3, with the task as the root node and the disk, mount point and container as leaf nodes, constructs the task execution resource tree;

[0014] S2.4 uses the longest prefix matching algorithm to associate the mount point used by the container in the task execution resource tree with the mount point of the resource information.

[0015] In step S2.3, the construction of the task execution resource tree involves using an extended Berkeley packet filter to obtain the addition, deletion, or modification of resources associated with the task, including containers, disks, or network lines, in real time, and synchronizing the changed resource status to the task execution resource tree.

[0016] Step S3 includes the following steps:

[0017] S3.1, reads the resource request information issued during task deployment from the task's configuration file or the cloud;

[0018] S3.2, Update the resource request information based on the resource information obtained in step S1;

[0019] S3.3 Based on the updated resource request information, the task is used as the root node, and the disk, mount point and container are used as leaf nodes to construct the task request resource tree.

[0020] Step S4 includes the following steps:

[0021] S4.1, compare the task request resource tree with the task execution resource tree to obtain the newly added leaf nodes;

[0022] S4.2, compare the task request resource tree with the task execution resource tree to obtain the reduced leaf nodes;

[0023] S4.3, compare the task request resource tree with the task execution resource tree to obtain the changed leaf nodes;

[0024] S4.4 Construct the set of newly added nodes and the set of deleted nodes based on the leaf nodes obtained in steps S4.1-S4.3.

[0025] Step S5 includes the following steps:

[0026] S5.1, Traverse the set of newly added nodes in the tuning information, and start the container, mount point, disk or network line corresponding to each leaf node in the set of newly added nodes to update the task running resource tree;

[0027] S5.2, traverse the set of deleted nodes in the tuning information, and close the container, mount point, disk or network line corresponding to each leaf node in the set of deleted nodes to update the task running resource tree.

[0028] A task control system for edge computing scenarios includes:

[0029] Resource information acquisition module: used to acquire resource information of the edge node where the task is located in real time. The resource information includes whether the disk is normal, the network line connectivity status, and the mount point corresponding to each disk.

[0030] Task execution resource tree building module: used to obtain the container information occupied by the task when it runs on the edge node, and build the task execution resource tree based on the container information and the resource information of the edge node;

[0031] Task Request Resource Tree Construction Module: Used to obtain resource request information expected by task execution from the cloud and construct a task request resource tree based on the resource request information;

[0032] Task Tuning Controller Module: Compares the task execution resource tree and the task request resource tree to obtain tuning information containing the set of newly added nodes and the set of deleted nodes, and uses a depth-first traversal method to perform task tuning based on the tuning information.

[0033] The beneficial effects of this invention are:

[0034] This application can perceive the resource information of edge nodes in real time, dynamically perceive the status of tasks and their associated resources, as well as the expected resources during task deployment. By establishing and comparing the task request resource tree and the task execution resource tree, it connects the resource status during task execution with the expected resource status of the task. Combined with the real-time perceived resource information, it achieves rapid task tuning and, in the event of a failure, tunes the task to the expected state with minimal cost. It can efficiently solve the problem of tasks failing to automatically adjust under resource loss or disk damage. Using a Linux Hook-based approach, it can detect resource damage and loss in near real-time and trigger task synchronization. Compared to the 5-10 minutes of traditional methods, the tuning time of this application can be shortened to 1 minute. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 This is a schematic diagram of the process of the present invention.

[0037] Figure 2 This diagram illustrates the construction and tuning of the task execution resource tree and the task request resource tree. Detailed Implementation

[0038] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0039] Example 1: A task control method in an edge computing scenario, such as Figure 1 As shown, it includes the following steps:

[0040] S1, obtain the resource information of the edge node where the task is located. The resource information includes whether the disk is normal, the network line connectivity status, and the mount point corresponding to each disk.

[0041] In edge cloud scenarios, issues such as sudden disk loss or network interruption may occur during task execution. This application provides real-time monitoring of edge node resource information to help the task deployment system detect these issues promptly and avoid impacting task processing quality. Implementation can be based on polling and / or event-driven methods. Polling involves repeatedly executing S1 to check node resource information, while event-driven methods use eBPF in S2.3 to obtain events such as disk loss or network card loss, triggering updates to the task's resource tree. S2.3 offers higher real-time performance, while S1 is more reliable; however, not all resources have events, so a combination of S1 and S2.3 is optimal.

[0042] S2, Obtain the container information occupied by the task running on the edge node, and construct the task execution resource tree based on the container information and the resource information in step S1, including the following steps:

[0043] S2.1, Obtain all containers used during task execution;

[0044] S2.2, Obtain the resource usage information of all containers in step S2.2, the resource usage information including network lines, disks occupied by containers, mount points and parameters used by containers;

[0045] The parameters include the container's environment variables and startup parameters, which can affect the on / off states, options, and configuration parameters of programs inside the container. In a Docker container, network connections represent the correspondence between a container and one or more DockerNetworks. Specifically, the container list and parameters can be obtained by calling the ` / var / run / docker.sock` interface.

[0046] S2.3, such as Figure 2 As shown, the task is used as the root node, and the disk, mount point and container are all used as leaf nodes to build the task execution resource tree;

[0047] Specifically, by dynamically inserting hook points into the Linux kernel using the extended Berkeley Packet Filter (eBPF, a technology that allows user-written programs to run within the Linux kernel without modifying kernel code or loading kernel modules), it's possible to obtain near real-time data on the creation, deletion, modification, and querying of resources associated with the task, such as containers, disks, and network lines, and write the changed status to the task's current status tree. This tree structure effectively describes the resources that the task depends on.

[0048] S2.4 uses the longest prefix matching algorithm to associate the mount points used by containers in the task execution resource tree with the mount points of edge nodes, and the network lines of containers with the network lines of edge nodes.

[0049] Specifically, the longest prefix matching algorithm is used to associate the container's mount point with the edge node's mount point to ensure that the association between the container and the disk can be obtained correctly. Then, the container's network line is associated with the node's network line. Associating lines and mount points is essentially a resource allocation. Allocating mount points to containers allows them to obtain more persistent storage capacity and read / write performance; allocating lines to containers allows them to obtain greater bandwidth to support higher network throughput; allocating multiple lines allows the container network to have greater fault tolerance, so that if one line fails, another line can be used to provide network services.

[0050] Traditional task controllers retain some data in the system. For example, the / var / lib / kubelet / pods directory in Kubernetes stores metadata such as mount points and configurations of all containers. However, when the metadata is corrupted, the controller will malfunction. In contrast, this application obtains the association between containers and disks without relying on any additional data. It relies entirely on real-time hardware information in the system to make the association, ensuring the accuracy and real-time nature of the association data.

[0051] S3 obtains the resource request information required for task execution from the cloud through a publish-subscribe model-based communication method, and constructs a task request resource tree based on the resource request information, including the following steps:

[0052] S3.1, reads the resource request information issued during task deployment from the task's configuration file or the cloud;

[0053] The resource request information refers to the system's deployment expectation for the current task, including the containers, network lines, disks, and mount points used for execution. The edge nodes communicate with the cloud based on the publish-subscribe model, and the cloud can accurately and in real time synchronize the user's latest expected configuration for the task to the edge nodes through a publishing method.

[0054] For example: `docker run --name=pai-task-1, --volume / host / data-1: / data-1, --network=wan0, --env=PROD_ENV=yes --cpus2 --memory4gregistry.cn-shanghai.aliyuncs.com / ppio / pai-task:2201121107`, where `--network=wan0` declares that this container uses the wan0 line, and `--volume= / host / data-1` declares that this container uses the / host / data-1 mount point; the values ​​of `--network` and `--volume` in the running container can be obtained through the Docker Inspect API; the association between the container and the line can be directly obtained through `--network`.

[0055] S3.2, Update the resource request information based on the resource information obtained in step S1;

[0056] Specifically, if the line, disk, or mount point in the resource request information is absent or damaged on the edge node, the information is removed, and the changed resource request information triggers task control tuning. This ensures that in the event of resource failure on the edge node, such as disk failure, the task deployment is not detected and adjusted.

[0057] S3.3, such as Figure 2 As shown, based on the updated resource request information, the task is used as the root node, and the disk, mount point and container are all used as leaf nodes to construct the task request resource tree;

[0058] Specifically, firstly, the longest prefix matching algorithm is used to bind the disk and mount point in the resource request information. Then, the longest prefix matching algorithm is used to bind the container and mount point. Finally, the network line in the resource request information is bound to the container, thus forming the Desired Status Tree. As can be seen from the two resource trees, the association between the container and the disk needs to be derived through two association relationships: the association between the disk and the mount point, and the association between the mount point and the container.

[0059] The initial resource allocation during container startup often determines the running efficiency of tasks within the container. Reasonable resource allocation can ensure the stable and efficient operation of the container while making full use of computing resources to complete the required tasks.

[0060] S4, compare the task execution resource tree and the task request resource tree to obtain tuning information, including the following steps:

[0061] S4.1, compare the task request resource tree with the task execution resource tree to obtain the newly added leaf nodes;

[0062] S4.2, compare the task request resource tree with the task execution resource tree to obtain the reduced leaf nodes;

[0063] S4.3, compare the task request resource tree with the task execution resource tree to obtain the changed leaf nodes;

[0064] S4.4 Construct the set of newly added nodes and the set of deleted nodes based on the leaf nodes obtained in steps S4.1-S4.3.

[0065] At the implementation level, each leaf node implements two methods: `Down` and `Up`. Deletion destroys the corresponding resource, while addition deploys / creates it. For example, the `Up` method for a container leaf node starts the corresponding container, while deletion destroys it. Similarly, the `Up` method for a disk leaf node partitions and formats the disk, while deletion cleans up the partitions. Based on this approach, controller development doesn't need to consider the actual business logic of the leaf nodes. To improve comparison efficiency, when a leaf node in two trees is inconsistent, all its subordinate leaf nodes are marked as inconsistent. For example, if `DiskA->PartA->ContainerA` and `DiskA'->PartA'->ContainerA'` find inconsistencies when comparing `DiskA`, the comparison between `PartA` and `ContainerA` will not continue, greatly improving comparison efficiency.

[0066] S5, the controller uses a depth-first traversal method to perform tuning based on tuning information to ensure task synchronization, including the following steps:

[0067] S5.1, Traverse the set of newly added nodes, and start the container, mount point, disk or line corresponding to each leaf node in the set of newly added nodes to realize the update of the task running resource tree;

[0068] S5.2, traverse the set of deleted nodes, and close the container, mount point, disk or line corresponding to each leaf node in the set of deleted nodes to update the task running resource tree.

[0069] Reconcile is a mechanism for adjusting the current state to the desired state. Task synchronization is a state that refers to synchronizing the Current Status Tree to the Desired Status Tree based on the reconcile mechanism. When a new expected configuration is received from the cloud, changes to the task request resource tree will immediately trigger synchronization. When it is learned that the resources used by the task have been removed or corrupted, changes to the task's runtime resource tree will also immediately trigger synchronization.

[0070] This application abstracts the heterogeneous deployment of edge nodes as a task, treating each task on a node as a tree structure. The resources associated with the task are the leaf nodes of this tree. Tuning is achieved by minimizing the difference between the Desired Task Tree and the Current Task Tree. When any resource changes, the shortest path from the Current Task Tree to the Desired Task Tree can be calculated, thus achieving the lowest-cost change. Based on the current running state and the specified expected state of the task on the edge node, the task deployment topology on that node is dynamically adjusted, taking into account the current state of the edge node during the adjustment process. This minimizes the impact on the task deployment state by deploying and changing tasks at the lowest cost.

[0071] Example 2: A task control system for edge computing scenarios, comprising:

[0072] Resource information acquisition module: used to acquire resource information of the edge node where the task is located in real time. The resource information includes whether the disk is normal, the network line connectivity status, and the mount point corresponding to each disk.

[0073] Task execution resource tree building module: used to obtain the container information occupied by the task when it runs on the edge node, and build the task execution resource tree based on the container information and the resource information of the edge node;

[0074] The container information includes the container and resource usage information including network lines, the disk occupied by the container, the mount points used by the container, and parameters.

[0075] Task Request Resource Tree Construction Module: Used to obtain resource request information expected by task execution from the cloud and construct a task request resource tree based on the resource request information;

[0076] The resource request information refers to the system's expected deployment for the current task, including the containers, network lines, disks, and mount points required for execution.

[0077] Task Tuning Controller Module: Compares the task execution resource tree and the task request resource tree to obtain tuning information containing the set of newly added nodes and the set of deleted nodes, and uses a depth-first traversal method to perform task tuning based on the tuning information.

[0078] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A task control method in an edge computing scenario, characterized in that, Includes the following steps: S1, obtain the resource information of the edge node where the task is located. The resource information includes whether the disk is normal, the network line connectivity status, and the mount point corresponding to each disk. S2, obtain the container information occupied by the task running on the edge node, and construct the task running resource tree based on the container information and the resource information in step S1; S3 obtains the resource request information required for task execution from the cloud through a publish-subscribe model-based communication method, and constructs a task request resource tree based on the resource request information; S4, compare the task execution resource tree and the task request resource tree to obtain tuning information; S5 utilizes a depth-first traversal method to perform tuning based on tuning information to ensure task synchronization; Step S4 includes the following steps: S4.1, compare the task request resource tree with the task execution resource tree to obtain the newly added leaf nodes; S4.2, compare the task request resource tree with the task execution resource tree to obtain the reduced leaf nodes; S4.3, compare the task request resource tree with the task execution resource tree to obtain the changed leaf nodes; S4.4, construct the set of newly added nodes and the set of deleted nodes based on the leaf nodes obtained in steps S4.1-S4.3; Step S5 includes the following steps: S5.1, Traverse the set of newly added nodes in the tuning information, and start the container, mount point, disk or network line corresponding to each leaf node in the set of newly added nodes to update the task running resource tree; S5.2, traverse the set of deleted nodes in the tuning information, and close the container, mount point, disk or network line corresponding to each leaf node in the set of deleted nodes to update the task running resource tree.

2. The task control method in the edge computing scenario according to claim 1, characterized in that, Step S2 includes the following steps: S2.1, Obtain all containers used during task execution; S2.2, Obtain the resource usage information of all containers in step S2.

1. The resource usage information includes network lines, disks occupied by containers, mount points and parameters used by containers; The parameters include the container's environment variables and startup parameters; S2.3, with the task as the root node and the disk, mount point and container as leaf nodes, constructs the task execution resource tree; S2.4 uses the longest prefix matching algorithm to associate the mount point used by the container in the task execution resource tree with the mount point of the resource information.

3. The task control method in the edge computing scenario according to claim 2, characterized in that, In step S2.3, the construction of the task execution resource tree involves using an extended Berkeley packet filter to obtain the addition, deletion, or modification of resources associated with the task, including containers, disks, or network lines, in real time, and synchronizing the changed resource status to the task execution resource tree.

4. The task control method in the edge computing scenario according to claim 1, characterized in that, Step S3 includes the following steps: S3.1, reads the resource request information issued during task deployment from the task's configuration file or the cloud; S3.2, Update the resource request information based on the resource information obtained in step S1; S3.3 Based on the updated resource request information, the task is used as the root node, and the disk, mount point and container are used as leaf nodes to construct the task request resource tree.

5. A task control system for edge computing scenarios, characterized in that, include: Resource information acquisition module: used to acquire resource information of the edge node where the task is located in real time. The resource information includes whether the disk is normal, the network line connectivity status, and the mount point corresponding to each disk. Task execution resource tree building module: used to obtain the container information occupied by the task when it runs on the edge node, and build the task execution resource tree based on the container information and the resource information of the edge node; Task Request Resource Tree Construction Module: Used to obtain resource request information expected by task execution from the cloud and construct a task request resource tree based on the resource request information; Task Tuning Controller Module: Compares the task running resource tree and the task request resource tree to obtain tuning information containing the set of newly added nodes and the set of deleted nodes, and uses the depth-first traversal method to perform task tuning based on the tuning information; The step of comparing the task execution resource tree and the task request resource tree to obtain tuning information including the set of newly added nodes and the set of deleted nodes includes the following steps: S4.1, compare the task request resource tree with the task execution resource tree to obtain the newly added leaf nodes; S4.2, compare the task request resource tree with the task execution resource tree to obtain the reduced leaf nodes; S4.3, compare the task request resource tree with the task execution resource tree to obtain the changed leaf nodes; S4.4, construct the set of newly added nodes and the set of deleted nodes based on the leaf nodes obtained in steps S4.1-S4.3; The method of using depth-first traversal to perform task tuning based on tuning information includes the following steps: S5.1, Traverse the set of newly added nodes in the tuning information, and start the container, mount point, disk or network line corresponding to each leaf node in the set of newly added nodes to update the task running resource tree; S5.2, traverse the set of deleted nodes in the tuning information, and close the container, mount point, disk or network line corresponding to each leaf node in the set of deleted nodes to update the task running resource tree.

Images