Container-level network resource efficient management method for deterministic distributed computing scene

Abstract

The invention provides a container-level network resource efficient management method for a deterministic distributed computing scene, and mainly solves the problems that network resources cannot be effectively managed on a container level, the task scheduling real-time performance is low, the success rate is low, resources are wasted and the like in the prior art. For a deterministic distributed computing scene, container-level efficient network resource management is carried out, firstly, a network resource controller needs to be expanded, and Service service needs to be configured, so that a container can communicate with an external cloud server; then, by configuring a network strategy, controlling the container to use network resources; furthermore, the container tasks are aggregated according to the real-time and deterministic requirements of the tasks, so that the transmission efficiency of the container tasks is improved; and finally, storing and transmitting the aggregation task. According to the method, a container virtualization technology is taken as an implementation means, multi-time-slot aggregation is carried out on container tasks based on a dynamic programming algorithm, and algorithm parameters are continuously optimized in combination with an artificial intelligence technology, so that the real-time performance and the success rate of the tasks are guaranteed, and the resource utilization rate is also remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of resource management, and in particular to an efficient container-level network resource management method for deterministic distributed computing scenarios. Background Art

[0002] In the context of the rapid development of information technology, the Industrial Internet of Things has put forward higher requirements on the quality of service (QoS), real-time, reliability and security of tasks. Although cloud computing and edge computing provide feasible solutions for such scenarios, how to effectively distribute and manage the load of a large number of edge nodes remains a challenge.

[0003] One possible solution is to use virtualized container technology. Currently, several mainstream container orchestration platforms have implemented efficient and stable resource scheduling mechanisms that can provide a reliable operating environment for a variety of tasks. Although these container orchestration systems perform well in computing resource management and scheduling, they still have deficiencies in container-level network resource management. This deficiency may lead to fierce competition for network resources between containers, making it difficult to meet the network requirements of different tasks, which in turn affects the efficiency of task scheduling and may lead to a reduction in overall system performance. Summary of the invention

[0004] In view of the problems that the existing technology cannot effectively manage network resources at the container level, the task scheduling has low real-time performance, low success rate, and waste of resources. The present invention proposes a container-level network resource efficient management method for deterministic distributed computing scenarios. For the convenience of description, the present invention takes Kubernetes as an example for explanation. The method extends the network plug-in for the Kubernetes cluster edge system, proposes a multi-slot container task aggregation algorithm based on dynamic programming, and continuously optimizes the algorithm parameters in combination with artificial intelligence technology, performs fine-grained network resource allocation for container tasks, improves the task transmission rate and the success rate of task scheduling, and effectively reduces resource waste and reduces the overall system cost.

[0005] The technical solution adopted by the present invention to achieve the above-mentioned purpose is:

[0006] The efficient management method of container-level network resources for deterministic distributed computing scenarios includes the following steps:

[0007] 1) Expand the network resource controller;

[0008] 2) Configure network policies according to container task requirements;

[0009] 3) Aggregate container tasks in multiple time slots;

[0010] 4) Store and transmit the aggregated container tasks.

[0011] The step 1) comprises the following steps:

[0012] 1.1) Expand Calico network plug-in;

[0013] 1.2) Configure network bandwidth;

[0014] 1.3) Parse the Pod configuration file;

[0015] 1.4) Set TC rules;

[0016] 1.5) Configure the Service.

[0017] The step 1.5) is specifically as follows:

[0018] Use a NodePort type Service to map the Service port to a port on the Node edge node. When a request reaches the entry address of the Service, kube-proxy distributes the request to the corresponding container to achieve data transmission between the container and the external cloud server.

[0019] The step 2) comprises the following steps:

[0020] 2.1) Define the network policy for container tasks;

[0021] 2.2) Use the kubectl command to apply the defined network policy to the cluster;

[0022] 2.3) Deploy a temporary container task to test the network strategy.

[0023] The step 2.1) is specifically as follows:

[0024] The network policy is defined in the form of a YAML file, including: apiVersion, kind, metadata and spec, where apiVersion specifies the api version, kind specifies the type of policy, metadata specifies the name and namespace of the network policy, spec specifies the containers affected by the policy and the allowed or denied traffic, in which podSelector specifies which pods the policy applies to, and policyTypes specifies the type of policy, which includes inbound traffic control Ingress, outbound traffic control Egress or both, in Ingress and Egress rules, by specifying specific IP addresses, port numbers and protocols, the allowed and denied traffic sources or destinations are defined to control the communication between containers and external cloud servers.

[0025] The step 3) comprises the following steps:

[0026] 3.1) Define the properties and global parameters of each container task;

[0027] 3.2) Sort the container tasks according to the deadline of each container task;

[0028] 3.3) State initialization;

[0029] 3.4) Construct state transfer equation;

[0030] 3.5) Determine the effectiveness of container task aggregation;

[0031] 3.6) Perform multi-slot container task aggregation.

[0032] The step 3.4) is specifically as follows:

[0033] For each task k, find the best way to combine it with several previous tasks and construct the state transition equation:

[0034] dp[k]=min(dp[k],dp[j-1]+1)

[0035] Among them, dp[k] represents the minimum number of data packets required for the first k tasks, and dp[j-1] represents the optimal solution for the first j-1 tasks, that is, the minimum number of data packets.

[0036] The step 3.5) comprises the following steps:

[0037] 3.5.1) Validity determination based on task data packet size:

[0038] The packet size constraint expression C1 is:

[0039]

[0040] in, represents the data volume of the i-th task of user u, M is the maximum limit of a single data packet, that is, the sum of the data volumes of the aggregated container tasks must be less than or equal to the maximum limit of the data packet;

[0041] 3.5.2) Validity determination based on task real-time performance:

[0042] a) The deadline difference constraint expression C2 of the aggregation task is:

[0043]

[0044] in, represents the deadline of the i-th task of user u in the current time slot, and Δt is the allowed deadline difference, that is, the deadline difference of the aggregate container tasks must be less than or equal to the allowed deadline difference;

[0045] b) The latest transmission time constraint expression C3 of the aggregation task data packet is:

[0046]

[0047] Among them, t send Indicates the latest transmission time of a data packet. That is, when multiple container tasks are aggregated into the same data packet, the latest transmission time of the data packet must be less than the earliest deadline in the aggregated tasks.

[0048] c) The maximum waiting time constraint expression C4 of the aggregation task is:

[0049]

[0050] Among them, d min_p represents the minimum deadline of the data packet in the previous time slot, and t represents the length of each time slot, that is, the deadline of the new task cannot exceed the minimum deadline of the aggregated data packet in the previous time slot minus the time slot length.

[0051] The step 3.6) comprises the following steps:

[0052] 3.6.1) When time slot T=1, traverse each task k sequentially, initialize the total data size of the current data packet, the maximum deadline and the minimum deadline of the data packet;

[0053] 3.6.2) For each task k, traverse backward from task k, accumulate the total data size of the current data packet, update the maximum and minimum deadlines, and determine whether the validity of the container task aggregation meets the requirements, that is, whether C1, C2 and C3 all meet the validity determination requirements. If they do, execute step 3.6.3), otherwise, continue to the next j loop;

[0054] 3.6.3) Determine whether dp[j-1]+1 is less than dp[k]. If so, update dp[k] and packets[k] and continue with the next j loops. Otherwise, continue directly with the next j loops.

[0055] 3.6.4) When loop j ends, traverse the next task k sequentially until all tasks are traversed sequentially, and output the minimum number of packets in dp[n] and the aggregation task status of each packet in packets[n];

[0056] 3.6.5) When time slot T>1, perform cross-time slot aggregation, traverse the tasks of the current time slot, add the tasks of the current time slot to the data packet of the previous time slot, and determine whether the validity of the container task aggregation meets the requirements, that is, whether C1, C2, C3 and C4 all meet the requirements of validity judgment. If the requirements are met, aggregate the tasks to the existing data packets, update the minimum deadline of the current data packet, and output the task aggregation status. If the requirements are not met, execute steps 3.6.1) to 3.6.4) to aggregate the tasks that cannot be aggregated across time slots in the current time slot, and output the task aggregation status.

[0057] The step 4) comprises the following steps:

[0058] 4.1) Store the aggregated tasks;

[0059] 4.2) Transmit the stored results:

[0060] 4.2.1) Write and build container servers and client scripts inside and outside the cluster;

[0061] 4.2.2) Deploy container images in the cluster;

[0062] 4.2.3) Configure and deploy container tasks;

[0063] 4.2.4) Expose container service ports;

[0064] 4.2.5) Send and receive data within and outside the cluster.

[0065] The present invention has the following beneficial effects and advantages:

[0066] 1. The present invention conducts in-depth research on the problems of low real-time performance, low success rate, and waste of resources in current task scheduling. The proposed method improves the task transmission rate and the success rate of task scheduling, and also further improves the intelligence level of deterministic distributed computing scenarios, which has certain practical application significance.

[0067] 2. Compared with the prior art, the container-level network resource efficient management method for deterministic distributed computing scenarios proposed in the present invention takes into account the network resource allocation at the container level, can perform fine-grained allocation of a large number of tasks submitted by users in multiple time slots, and can efficiently manage network resources in deterministic distributed computing scenarios, thus having better applicability.

[0068] 3. The present invention proposes a multi-slot container task aggregation algorithm based on dynamic programming, which can aggregate container tasks according to the real-time requirements of container tasks and the maximum capacity of data packets, solve the optimal data packet aggregation solution, avoid the bottleneck problem caused by serial transmission of multiple containers, improve the task transmission rate, and ensure the real-time and reliability of task scheduling. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 is a flow chart of the method of the present invention;

[0070] Figure 2 This is a schematic diagram of a cloud server outside the cluster accessing a container of the present invention;

[0071] Figure 3 This is a multi-slot container task aggregation flow chart based on dynamic programming of the present invention;

[0072] Figure 4 This is a schematic diagram of the structure of the aggregation task data packet of the present invention. DETAILED DESCRIPTION

[0073] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0074] In order to make the technical solution of the present invention more specific and clear, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be noted that the specific embodiments described below are only used to explain the present invention and are not used to limit the present invention.

[0075] The present invention is implemented as follows: a method for efficient management of container-level network resources for deterministic distributed computing scenarios. In deterministic distributed computing scenarios, taking Kubernetes as an example, network resource management at the container level is performed. First, it is necessary to expand the network resource controller to manage the network resources of the Pod layer, expand the Calico network plug-in, control the corresponding network bandwidth, and configure the Service service so that the containers in the cluster can communicate with the outside of the cluster; then, configure the network policy, that is, configure the network policy for the container task, and control the use of network resources by the container task by controlling the port of the container task; further, aggregate the container tasks to improve the transmission efficiency, that is, according to the real-time requirements of the container task and the maximum capacity of the data packet, perform multi-time slot aggregation on the container task based on the dynamic programming algorithm to ensure the real-time performance of the task and solve the optimal data packet aggregation scheme, output the task aggregation information table and minimize the number of data packets; finally, store and transmit the aggregated tasks. That is, according to the task aggregation information table generated by the present invention, select the data of multiple aggregated container tasks for shared storage, write and build the container server and client scripts inside and outside the cluster, deploy the container image in the cluster, configure and deploy the container application and service port, and realize the data transmission inside and outside the cluster. The method flow chart of the present invention is as follows Figure 1 As shown, specifically:

[0076] Extend the network resource controller: Extend the Calico network plug-in, configure the corresponding network bandwidth, and configure the Service service so that the container can communicate with the outside of the cluster;

[0077] Configure network policies: Define network policies, specify policy types, apply network policies, and control the use of network resources by container tasks according to container task requirements;

[0078] Aggregate container tasks: Based on the real-time requirements of container tasks and the maximum capacity of data packets, multi-slot aggregation of container tasks is performed based on the dynamic programming algorithm to solve the optimal data packet aggregation solution, output task aggregation information and minimize the number of data packets;

[0079] Storage and transmission of aggregated tasks: Choose to share the data of multiple aggregated container tasks, write and build container servers and client scripts inside and outside the cluster, deploy container images within the cluster, configure and deploy container tasks and service ports, and realize data transmission within and outside the cluster.

[0080] Specifically, the implementation of the present invention includes the following four steps:

[0081] Step 1: Expand the network resource controller;

[0082] Step 2: Configure network policy;

[0083] Step 3: Aggregate container tasks;

[0084] Step 4: Storage and transmission of aggregated tasks.

[0085] The extended network resource controller in step 1 includes:

[0086] Step 1.1: Extend the Calico network plugin;

[0087] Step 1.2: Configure the corresponding network bandwidth;

[0088] Step 1.3: Parse the Pod configuration file;

[0089] Step 1.4: Set TC rules;

[0090] Step 1.5: Configure the Service.

[0091] The step 1.1 is specifically as follows:

[0092] In deterministic distributed computing scenarios, based on the Kubernetes cluster edge system, choose the Calico network plug-in from among many network plug-ins. Calico uses a layer 3 network-based architecture to directly route IP packets between nodes through routing tables, reducing the overhead of encapsulating and decapsulating packets, so that each Pod has a unique, routable IP address, and improving the efficiency of communication between Pods. When installing the Calico plug-in, first confirm that the Kubernetes cluster has been deployed and is running, then use the kubectl command to install the Calico CNI plug-in, ensure that the Calico CNI plug-in is installed on the Kubernetes control plane, and after the installation is complete, further verify whether the Calico plug-in is correctly deployed and running.

[0093] The step 1.2 is specifically as follows:

[0094] In the Kubernetes cluster that has extended the Calico network plug-in, configure network resources. Mainly by configuring the network bandwidth parameters "ingress-bandwidth" and "egress-bandwidth" in the Pod's YAML file to limit the Pod's ingress and egress bandwidth, thereby limiting network resources.

[0095] The step 1.3 is specifically as follows:

[0096] The Kubernetes cluster parses the Pod's configuration file and extracts the annotation part. With this information, the network plug-in Calico detects this annotation, parses the network bandwidth limit value of the Pod's ingress and egress, and determines the configured network bandwidth parameters.

[0097] The step 1.4 is specifically as follows:

[0098] Through the Calico network plug-in, the TC tool of the Linux kernel function is used to perform QoS technical restrictions on network traffic. According to the parsed bandwidth limit value, the network plug-in generates the corresponding TC (Traffic Control) rules. The TC tool schedules traffic by creating classes and queues to ensure that the network traffic of the Pod does not exceed the defined bandwidth limit. The TC rule operates on the qdisc (queueing discipline) layer of the network interface and uses HTB (Hierarchical Token Bucket) to limit traffic. HTB provides hierarchical traffic control, controls the maximum bandwidth of each Pod by assigning different classes and queues, and manages the transmission rate of traffic through the token bucket algorithm. Apply the TC rule to the network interface associated with the Pod (the network interface of the node where the Pod is located). Once the TC rule takes effect, the ingress and egress traffic of the Pod will be scheduled and controlled according to the values ​​configured in the annotation.

[0099] The step 1.5 is specifically as follows:

[0100] In order to enable cloud servers outside the cluster to access the containers in the Pod, Service resources are introduced. Service provides a fixed entry address for the Pod, so that external cloud servers can access the containers in the Pod through Service. Select the NodePort type Service, map the Service port to a port on the Node edge node, and the outside can access the service through the Node IP address and the mapped port number NodePort. When the request reaches the entry address of the Service, kube-proxy will distribute the request to the corresponding container to realize data transmission between the container and the external cloud server. The access process is as follows: Figure 2 shown.

[0101] The network policy configuration in step 2 includes:

[0102] Step 2.1: Define the network policy for the container task;

[0103] Step 2.2: Apply the defined network policy;

[0104] Step 2.3: Test the network policy.

[0105] The step 2.1 is specifically as follows:

[0106] In deterministic distributed computing scenarios, choose to control network bandwidth on IP addresses and ports, and control the use of network resources by containers by configuring network policies. First, you need to define network policies, which are defined in the form of YAML files and usually include several key parts: apiVersion, kind, metadata, and spec. Set apiVersion to networking.k8s.io / v1 and kind to NetworkPolicy. In metadata, specify the name and namespace of the network policy. Among them, spec is the core of the network policy, defining which containers are affected by the policy and which traffic is allowed or denied. Through podSelector, specify which Pods the policy applies to, usually matched by label selectors. policyTypes specifies the type of policy, which can be Ingress (inbound traffic control), Egress (outbound traffic control), or both. In Ingress and Egress rules, define the allowed and denied traffic sources or destinations by specifying specific IP addresses, port numbers, and protocols to fine-grainedly control the communication between containers and external cloud servers.

[0107] The step 2.2 is specifically as follows:

[0108] Based on the network policy defined in step 2.1, use the kubectl command to apply it to the cluster.

[0109] The step 2.3 is specifically as follows:

[0110] After applying the network policy, deploy some temporary container tasks and use tools such as curl to send network requests to test the actual effect of the policy.

[0111] The aggregation container tasks in step 3 include:

[0112] Step 3.1: Container task definition

[0113] Step 3.2: Container task sequencing

[0114] Step 3.3: State initialization

[0115] Step 3.4: State transfer equation construction

[0116] Step 3.5: Determine the effectiveness of container task aggregation

[0117] Step 3.6: Multi-slot container task aggregation

[0118] The complete flowchart of the multi-slot container task aggregation algorithm based on dynamic programming is as follows: Figure 3As shown, specifically, the step 3.1 is as follows:

[0119] For container tasks, you need to define the properties of each task, including data size, deadline, and original task number (used to output the original order). Define global parameters, including the maximum size limit of each data packet max_packet_size, deadline difference max_delay_diff, number of users num_users, number of tasks per user tasks_per_user, number of time slots T, and time slot length t.

[0120] The step 3.2 is specifically as follows:

[0121] For tasks submitted by users, we first sort the tasks in ascending order according to their deadlines. Sorting ensures that similar tasks have similar deadlines, so that tasks with similar deadlines are aggregated in the same data packet. In subsequent state transfers, we can quickly evaluate whether the deadline difference is within the allowable range. Sorting can reduce the complexity of dynamic planning state transfers and improve the efficiency of real-time judgment.

[0122] The step 3.3 is specifically as follows:

[0123] Define an array dp[k] to represent the minimum number of packets required for the first k tasks. In other words, dp[k] represents the minimum number of packets required for the optimal combination from the first task to the kth task, and the initial state is 0. Define a packets array to record the task number contained in each packet, and the initial state is empty.

[0124] The step 3.4 is specifically as follows:

[0125] For each task k, find the best way to combine it with the previous tasks. To calculate dp[k], we can find the optimal solution from all possible previous situations:

[0126] dp[k]=min(dp[k],dp[j-1]+1)

[0127] Where dp[j-1] represents the optimal solution (i.e., the minimum number of data packets) for the first j-1 tasks. Then decide whether tasks j to k can be merged into one data packet. If they can be merged (meeting the data packet size limit and task real-time requirements), then they can be considered as a new data packet. Therefore, the solution of dp[k] is dp[j-1]+1.

[0128] The step 3.5 comprises the following steps:

[0129] Step 3.5.1: Determine the validity of the task data packet size. In task scheduling and aggregation problems, determining the data size of the aggregated task can avoid potential packet loss problems due to some tasks exceeding the data packet limit.

[0130] The packet size constraint expression C1 is,

[0131]

[0132] in, represents the data volume of the i-th task of user u, and M is the maximum limit of a single data packet, that is, the sum of the data volumes of the aggregated container tasks must be less than or equal to the maximum limit of the data packet.

[0133] Step 3.5.2: Determine the effectiveness based on the real-time nature of the task. In task scheduling and aggregation problems, real-time nature means that tasks must be completed within a certain time constraint. This is a strict requirement of the system on task time, which is mainly reflected in the deadline of each task. For the aggregation of multiple tasks, real-time nature means the following three aspects:

[0134] Task deadline difference constraint: When aggregating multiple tasks into the same data packet, the deadline differences between these tasks cannot be too large, ensuring that these tasks can be processed simultaneously and completed within the same time window.

[0135] The deadline difference constraint expression C2 of the aggregation task is:

[0136]

[0137] in, represents the deadline of the i-th task of user u, and Δt is the allowed deadline difference, that is, the deadline difference of the aggregated container tasks must be less than or equal to the allowed deadline difference. For the allowed deadline difference Δt, choose to use machine learning technology to train an AI model. The model dynamically predicts the optimal allowed deadline difference Δt of the current system based on parameters such as task deadlines and task quantity, further improving scheduling efficiency and task real-time performance.

[0138] The latest transmission time constraint of a data packet: The earliest deadline of all tasks in a data packet is the latest transmission time of the data packet, otherwise the real-time requirements of the tasks will not be met.

[0139] The latest transmission time constraint expression C3 of the aggregation task data packet is:

[0140]

[0141] Among them, t send Indicates the latest transmission time of the data packet. It represents the deadline of the i-th task of user u. That is, when multiple container tasks are aggregated into the same data packet, the transmission time of the data packet must take into account the earliest deadline in the aggregated tasks to ensure that the latest transmission time of the data packet is less than the earliest deadline in the aggregated tasks, ensuring that all tasks can be transmitted in time and ensuring the real-time performance of the tasks.

[0142] Maximum waiting time constraint for aggregated tasks: When aggregating tasks across time slots, in order to reduce the number of data packets, try to add the tasks of the current time slot to the aggregated data packets of the previous time slot.

[0143] The maximum waiting time constraint expression C4 of the aggregation task is:

[0144]

[0145] in, Indicates the deadline for users to submit tasks in the current time slot, d min_p represents the minimum deadline of the data packet in the previous time slot, and t represents the length of each time slot. That is, the deadline of the new task cannot exceed the minimum deadline of the aggregated data packet in the previous time slot minus the time slot length.

[0146] The step 3.6 is specifically as follows:

[0147] When the time slot T = 1, use the dynamic programming algorithm to aggregate the tasks, traverse each task k in sequence, initialize the total data size of the current data packet, the maximum deadline and the minimum deadline of the data packet. For each task k, traverse backward from task k (i.e. j from k to 1), accumulate the total data size of the current data packet, update the maximum and minimum deadlines, and judge whether the data packet size, the deadline difference of the tasks and the latest data packet transmission time requirements of the aggregated tasks meet the requirements according to the effectiveness judgment of the container task aggregation in step 3.5. If the requirements are met, judge whether dp[j-1]+1 is less than dp[k]. If it is less, then update dp[k] and packets[k] and continue to the next j loop. If it is not less, then continue to the next j loop; if it does not meet the requirements, then continue to the next j loop. When the j loop ends, traverse the next task k in sequence until all tasks are traversed in sequence, and output the minimum number of data packets of dp[n] and the aggregation task status of each data packet of packets[n]. When slot T>1, perform cross-slot aggregation, traverse the tasks of the current slot, and try to add the tasks of the current slot to the data packet of the previous slot to reduce the number of data packets. Determine whether the data packet size, the deadline difference of the tasks, the latest data packet transmission time requirement of the aggregated task, and the maximum waiting time of the aggregated task meet the requirements. If the requirements are met, aggregate the tasks into the existing data packets, update the minimum deadline of the current data packet, and output the task aggregation situation. If the requirements are not met, use the dynamic programming algorithm to aggregate the tasks that cannot be aggregated across time slots in the current slot, and output the task aggregation situation.

[0148] The storage and transmission of the aggregation task in step 4 includes:

[0149] Step 4.1: Storage of Aggregate Tasks

[0150] Step 4.2: Transfer of Aggregate Tasks

[0151] The step 4.1 is specifically as follows:

[0152] In order to avoid the overhead caused by serial transmission of each container task through its own port number and improve the stability and efficiency of data transmission, choose to store the data of multiple container tasks in the shared storage volume emptyDir, and splice these data into a complete data packet, and transmit them uniformly through the port number of a certain container, reducing the number of serial transmissions between containers, improving the task transmission rate, and ensuring the real-time performance of the task.

[0153] The step 4.2 comprises the following steps:

[0154] Step 4.2.1: Write and build container server and client scripts inside and outside the cluster

[0155] First, write the TCP server script tcp_server.py and the TCP client script tcp_send.py in the cluster. These two scripts run in a container of the aggregation task. tcp_server.py is used to receive data packets sent by external clients and distribute the data to each container in the shared storage volume. tcp_send.py is used to send data that needs to be sent by multiple containers in the shared storage volume, and send data to the outside of the cluster through IP addresses and port numbers. Secondly, write the TCP client send.py and the TCP server script receive.py outside the cluster. These two scripts run in the external cloud server. send.py is used to send data required by multiple containers in the shared storage volume, and send data to the container through the Node's IP address and the mapped port number NodePort. receive.py is used to receive data that needs to be sent by multiple containers in the shared storage volume and identify which container the data comes from.

[0156] Step 4.2.2: Deploy container images in the cluster

[0157] Create a Dockerfile, package the two Python scripts of the TCP server and client in the cluster into a container image, build the container image, and push the image to the cloud image factory library to facilitate the use of subsequent container tasks.

[0158] Step 4.2.3: Configure and deploy container tasks

[0159] Write the configuration file for the container task, configure the network bandwidth according to step 1.2, and use the container image in step 4.2.2 to provide an environment for sending and receiving container tasks.

[0160] Step 4.2.4: Expose container service ports

[0161] Write the Service configuration file, configure the Service for the container task according to step 1.5, and open the container port so that the cloud server outside the cluster can access the container.

[0162] Step 4.2.5: Sending and receiving data within and outside the cluster

[0163] The data transmission is carried out through the TCP protocol. Specifically, it is necessary to first read the data formatted as byte sequences from multiple containers, and add identifiers to the data of each container during the data packet splicing process to ensure that the receiving end can correctly parse and distinguish the data, and splice the data in sequence to form a data packet. The schematic diagram of the aggregation task data packet structure is shown in the figure. Figure 4 shown.

[0164] In order to ensure that the spliced ​​data packet can be transmitted through a TCP segment, the size of the data packet must be controlled so that it does not exceed the maximum segment size (MSS). MSS is the maximum amount of data that can be accommodated in each TCP segment negotiated by both parties of the TCP connection. It is usually slightly smaller than the maximum transmission unit (MTU) to reserve space for the IP and TCP headers. When the size of the data packet is effectively controlled, the data packet will be encapsulated into a TCP segment and the necessary TCP header information will be attached, such as key fields such as the source port, destination port, sequence number, and acknowledgment number. Then, the TCP segment is encapsulated into an IP data packet and IP header information is added, such as the source IP address and destination IP address. Finally, these encapsulated IP data packets are transmitted through the network. When transmitting data packets, it is necessary to ensure that the latest time to send the data packet must be less than the earliest deadline of the aggregated task to ensure that all tasks can be transmitted in time and the real-time nature of the tasks is guaranteed.

[0165] Similarly, if the container needs to obtain data from an external cloud server, the data to be transmitted will also undergo similar processing. The data to be sent to each container is spliced ​​into a complete data packet, ensuring that each part represents the data required by different containers. Before sending the data, the system checks whether the length of the spliced ​​data packet meets the MSS limit. If the length of the data packet is within the allowed range, the data packet will be sent to the specified container through the TCP connection.

Claims

1. An efficient container-level network resource management method for deterministic distributed computing scenarios, characterized in that: The following steps are involved: 1) Expand the network resource controller; 2) Configure network policies according to container task requirements; 3) Aggregate container tasks in multiple time slots; 4) Store and transmit the aggregated container tasks.

2. The container-level network resource efficient management method for deterministic distributed computing scenarios according to claim 1 is characterized in that: The step 1) comprises the following steps: 1.1) Expand Calico network plug-in; 1.2) Configure network bandwidth; 1.3) Parse the Pod configuration file; 1.4) Set TC rules; 1.5) Configure the Service.

3. The container-level network resource efficient management method for deterministic distributed computing scenarios according to claim 2 is characterized in that: The step 1.5) is specifically as follows: Use a NodePort type Service to map the Service port to a port on the Node edge node. When a request reaches the entry address of the Service, kube-proxy distributes the request to the corresponding container to achieve data transmission between the container and the external cloud server.

4. The container-level network resource efficient management method for deterministic distributed computing scenarios according to claim 1 is characterized in that: The step 2) comprises the following steps: 2.1) Define the network policy for container tasks; 2.2) Use the kubectl command to apply the defined network policy to the cluster; 2.3) Deploy a temporary container task to test the network strategy.

5. The container-level network resource efficient management method for deterministic distributed computing scenarios according to claim 4 is characterized in that: The step 2.1) is specifically as follows: The network policy is defined in the form of a YAML file, including: apiVersion, kind, metadata and spec, where apiVersion specifies the api version, kind specifies the type of policy, metadata specifies the name and namespace of the network policy, spec specifies the containers affected by the policy and the allowed or denied traffic, in which podSelector specifies which pods the policy applies to, and policyTypes specifies the type of policy, which includes inbound traffic control Ingress, outbound traffic control Egress or both, in Ingress and Egress rules, by specifying specific IP addresses, port numbers and protocols, the allowed and denied traffic sources or destinations are defined to control the communication between containers and external cloud servers.

6. The container-level network resource efficient management method for deterministic distributed computing scenarios according to claim 1 is characterized in that: The step 3) comprises the following steps: 3.1) Define the properties and global parameters of each container task; 3.2) Sort the container tasks according to the deadline of each container task; 3.3) State initialization; 3.4) Construct state transfer equation; 3.5) Determine the effectiveness of container task aggregation; 3.6) Perform multi-slot container task aggregation.

7. The container-level network resource efficient management method for deterministic distributed computing scenarios according to claim 6 is characterized in that: The step 3.4) is specifically as follows: For each task k, find the best way to combine it with several previous tasks and construct the state transition equation: dp[k]=min(dp[k],dp[j-1]+1) Among them, dp[k] represents the minimum number of data packets required for the first k tasks, and dp[j-1] represents the optimal solution for the first j-1 tasks, that is, the minimum number of data packets.

8. The container-level network resource efficient management method for deterministic distributed computing scenarios according to claim 6 is characterized in that: Step 3.5) The following steps are involved: 3.5.1) Validity determination based on task data packet size: The packet size constraint expression C1 is: in, represents the data volume of the i-th task of user u, M is the maximum limit of a single data packet, that is, the sum of the data volumes of the aggregated container tasks must be less than or equal to the maximum limit of the data packet; 3.5.2) Validity determination based on task real-time performance: a) The deadline difference constraint expression C2 of the aggregation task is: in, represents the deadline of the i-th task of user u in the current time slot, and Δt is the allowed deadline difference, that is, the deadline difference of the aggregate container tasks must be less than or equal to the allowed deadline difference; b) The latest transmission time constraint expression C3 of the aggregation task data packet is: Among them, t send Indicates the latest transmission time of a data packet. That is, when multiple container tasks are aggregated into the same data packet, the latest transmission time of the data packet must be less than the earliest deadline in the aggregated tasks. c) The maximum waiting time constraint expression C4 of the aggregation task is: Among them, d min_p represents the minimum deadline of the data packet in the previous time slot, and t represents the length of each time slot, that is, the deadline of the new task cannot exceed the minimum deadline of the aggregated data packet in the previous time slot minus the time slot length.

9. The container-level network resource efficient management method for deterministic distributed computing scenarios according to claim 6 is characterized in that: The step 3.6) comprises the following steps: 3.6.1) When time slot T=1, traverse each task k sequentially, initialize the total data size of the current data packet, the maximum deadline and the minimum deadline of the data packet; 3.6.2) For each task k, traverse backward from task k, accumulate the total data size of the current data packet, update the maximum and minimum deadlines, and determine whether the validity of the container task aggregation meets the requirements, that is, whether C1, C2 and C3 all meet the validity determination requirements. If they do, execute step 3.6.3), otherwise, continue to the next j loop; 3.6.3) Determine whether dp[j-1]+1 is less than dp[k]. If so, update dp[k] and packets[k] and continue with the next j loops. Otherwise, continue directly with the next j loops. 3.6.4) When loop j ends, traverse the next task k sequentially until all tasks are traversed sequentially, and output the minimum number of packets in dp[n] and the aggregation task status of each packet in packets[n]; 3.6.5) When time slot T>1, perform cross-time slot aggregation, traverse the tasks of the current time slot, add the tasks of the current time slot to the data packet of the previous time slot, and determine whether the validity of the container task aggregation meets the requirements, that is, whether C1, C2, C3 and C4 all meet the requirements of validity judgment. If the requirements are met, aggregate the tasks to the existing data packets, update the minimum deadline of the current data packet, and output the task aggregation status. If the requirements are not met, execute steps 3.6.1) to 3.6.4) to aggregate the tasks that cannot be aggregated across time slots in the current time slot, and output the task aggregation status.

10. The container-level network resource efficient management method for deterministic distributed computing scenarios according to claim 1, characterized in that: The step 4) comprises the following steps: 4.1) Store the aggregated tasks; 4.2) Transmit the stored results: 4.2.1) Write and build container servers and client scripts inside and outside the cluster; 4.2.2) Deploy container images in the cluster; 4.2.3) Configure and deploy container tasks; 4.2.4) Expose container service ports; 4.2.5) Send and receive data within and outside the cluster.

Images