Container-level network resource efficient management method for deterministic distributed computing scenarios

Abstract

The application provides a container-level network resource efficient management method for a deterministic distributed computing scene, and mainly solves the problems that the prior art cannot effectively manage network resources at the container level, the real-time performance of task scheduling is low, the success rate is low, and resources are wasted and the like. For the deterministic distributed computing scene, efficient network resource management at the container level is needed, first, the network resource controller needs to be extended, and a Service service is configured so that the container can communicate with the external cloud server; then, the use of network resources by the container is controlled by configuring a network strategy; further, the container tasks are aggregated according to the real-time performance and determinacy requirements of the tasks, and the transmission efficiency of the container tasks is improved; finally, the aggregated tasks are stored and transmitted. The method takes the container virtualization technology as the implementation means, aggregates the container tasks in multiple time slots based on the dynamic programming algorithm, and continuously optimizes the algorithm parameters in combination with the artificial intelligence technology, which not only guarantees the real-time performance and success rate of the tasks, but also significantly improves the resource utilization.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of resource management, and in particular to a container-level network resource efficient management method for deterministic distributed computing scenarios. BACKGROUND

[0002] Under the background of rapid development of information technology, industrial Internet of Things puts forward higher requirements on the quality of service (QoS), real-time performance, reliability and security of tasks. Although cloud computing and edge computing provide a feasible solution for such scenarios, how to effectively allocate and manage the load of a large number of edge nodes is still a challenge.

[0003] One possible solution is to use virtualization container technology. Currently, multiple mainstream container orchestration platforms have implemented efficient and stable resource scheduling mechanisms, which can provide reliable running environments for diversified tasks. Although these container orchestration systems perform well in computing resource management and scheduling, they still have deficiencies in network resource management at the container level. This deficiency may lead to fierce competition for network resources between containers, making it difficult to meet the network needs of different tasks, thereby affecting the efficiency of task scheduling and possibly leading to a decrease in overall system performance. SUMMARY

[0004] To address the problems of the prior art in terms of ineffective management of network resources at the container level, low real-time performance of task scheduling, low success rate, and resource waste, the present application proposes a container-level network resource efficient management method for deterministic distributed computing scenarios. For ease of description, the present application is described with Kubernetes as an example. This method extends the network plugin for the Kubernetes cluster edge system, proposes a multi-time slot container task aggregation algorithm based on dynamic programming, and continuously optimizes algorithm parameters using artificial intelligence technology to allocate network resources to container tasks in a fine-grained manner, thereby improving task transmission rate and task scheduling success rate, effectively reducing resource waste, and reducing overall system cost.

[0005] The technical solution adopted by the present application to achieve the above-mentioned purposes is as follows:

[0006] The container-level network resource efficient management method for deterministic distributed computing scenarios comprises the following steps:

[0007] 1) Extend the network resource controller;

[0008] 2) Configure network policies according to container task requirements;

[0009] 3) Aggregate container tasks in multiple time slots;

[0010] 4) Store and transmit the aggregated container tasks.

[0011] The step 1) comprises the following steps:

[0012] 1.1) extending the Calico network plug-in;

[0013] 1.2) configuring network bandwidth;

[0014] 1.3) parsing the configuration file of the Pod;

[0015] 1.4) setting TC rules;

[0016] 1.5) configuring the Service service.

[0017] The step 1.5) specifically comprises:

[0018] Using the Service of the NodePort type, the port of the Service is mapped to a port of the Node edge node, when the request reaches the entry address of the Service, the kube-proxy distributes the request to the corresponding container, realizing the data transmission between the container and the external cloud server.

[0019] The step 2) comprises the following steps:

[0020] 2.1) defining the network policy of the container task;

[0021] 2.2) applying the defined network policy to the cluster using the kubectl command;

[0022] 2.3) deploying a temporary container task to test the network policy.

[0023] The step 2.1) specifically comprises:

[0024] The network policy is defined in the form of a YAML file, including apiVersion, kind, metadata and spec, wherein the api version is specified by apiVersion, the type of the policy is specified by kind, the name and namespace of the network policy are specified by metadata, and the containers affected by the policy and the allowed or denied traffic are specified by spec, in the spec, the pods to which the policy applies are specified by podSelector, and the type of the policy is specified by policyTypes, the policy type includes Ingress, Egress or both, in the Ingress and Egress rules, the allowed and denied traffic sources or targets are defined by specifying specific IP addresses, port numbers and protocols, thereby controlling the communication between the container and the external cloud server.

[0025] The step 3) comprises the following steps:

[0026] 3.1) defining the attributes of each container task and global parameters;

[0027] 3.2) sorting the container tasks according to the deadline of each container task;

[0028] 3.3) state initialization;

[0029] 3.4) constructing state transition equation;

[0030] 3.5) judging the effectiveness of container task aggregation;

[0031] 3.6) performing multi-time slot container task aggregation.

[0032] The step 3.4) is specifically:

[0033] For each task k, find the best way to combine it with the previous several tasks, and construct the state transition equation:

[0034] dp[k] = min(dp[k], dp[j-1] + 1)

[0035] Wherein, dp[k] represents the minimum number of data packets required for the first k tasks, and dp[j-1] represents the optimal solution of the first j-1 tasks, i.e. the minimum number of data packets.

[0036] The step 3.5) comprises the following steps:

[0037] 3.5.1) judging the effectiveness based on the task data packet size:

[0038] The data packet size constraint expression C1 is:

[0039]

[0040] Wherein, represents the data volume of the i-th task of the user u, and M is the maximum limit of a single data packet, i.e. the sum of the data volumes of the aggregated container tasks must be less than or equal to the maximum limit of the data packet;

[0041] 3.5.2) judging the effectiveness based on the task real-time:

[0042] a) The deadline difference constraint expression C2 of the aggregated task is:

[0043]

[0044] Wherein, denotes the deadline of the i-th task of user u in the current time slot, and Δt is the allowed deadline difference, i.e. the deadline difference of the aggregated container task must be less than or equal to the allowed deadline difference;

[0045] b) the data packet latest transmission time constraint expression C3 of the aggregated task is:

[0046]

[0047] wherein t send denotes the latest transmission time of the data packet, i.e. when multiple container tasks are aggregated into the same data packet, the latest transmission time of the data packet must be less than the earliest deadline of the aggregated tasks;

[0048] c) the maximum waiting time constraint expression C4 of the aggregated task is:

[0049]

[0050] wherein d min_p denotes the minimum deadline of the data packet of the previous time slot, and t denotes the length of each time slot, i.e. the deadline of the new task cannot exceed the minimum deadline of the aggregated data packet of the previous time slot minus the length of the time slot.

[0051] The step 3.6) comprises the following steps:

[0052] 3.6.1) when the time slot T = 1, sequentially traverse each task k, initialize the total data size of the current data packet, the maximum deadline and the minimum deadline of the data packet;

[0053] 3.6.2) for each task k, traverse in reverse order from the task k, accumulate the total data size of the current data packet, update the maximum and minimum deadlines, and determine whether the effectiveness of the container task aggregation meets the requirements, i.e. whether C1, C2 and C3 all meet the effectiveness determination requirements, if the requirements are met, execute step 3.6.3), otherwise, continue the next j loop;

[0054] 3.6.3) determine whether dp[j-1]+1 is less than dp[k], if yes, update dp[k] and packets[k], and continue the next j loop, otherwise, directly continue the next j loop;

[0055] 3.6.4) when the j loop ends, sequentially traverse the next task k until all tasks are sequentially traversed, and output the minimum data packet number dp[n] and the aggregated task condition of each data packet packets[n];

[0056] 3.6.5) When the time slot T>1, cross time slot aggregation is performed, the tasks of the current time slot are added to the data packet of the previous time slot, and the validity of the container task aggregation is determined whether it meets the requirements, that is, whether C1, C2, C3 and C4 meet the requirements of the validity determination, if the requirements are met, the tasks are aggregated into the existing data packet, the minimum deadline of the current data packet is updated, and the task aggregation situation is output, if the requirements are not met, steps 3.6.1)~3.6.4) are executed to aggregate the tasks of the current time slot that fail to cross time slot aggregation, and the task aggregation situation is output.

[0057] The step 4) comprises the following steps:

[0058] 4.1) The aggregated tasks are stored;

[0059] 4.2) The storage result is transmitted:

[0060] 4.2.1) Container server and client script inside and outside the cluster are written and constructed;

[0061] 4.2.2) The container image of the cluster is deployed;

[0062] 4.2.3) The container task is configured and deployed;

[0063] 4.2.4) The container service port is exposed;

[0064] 4.2.5) The data inside and outside the cluster is sent and received.

[0065] The present application has the following beneficial effects and advantages:

[0066] 1. The present application is aimed at the problems of low real-time performance, low success rate and resource waste of current task scheduling, and the method improves the task transmission rate and the success rate of task scheduling, and further improves the intelligent level of the deterministic distributed computing scene, and has certain practical application significance.

[0067] 2. Compared with the prior art, the container-level network resource efficient management method for the deterministic distributed computing scene can consider the network resource allocation of the container level, can perform fine-grained allocation on a large number of tasks submitted by users in multiple time slots, can efficiently manage the network resources in the deterministic distributed computing scene, and has better applicability.

[0068] 3.The application provides a multi-time slot container task aggregation algorithm based on dynamic programming, which can aggregate container tasks according to the real-time requirements of the container tasks and the maximum capacity of the data packets, solve the optimal data packet aggregation scheme, avoid the bottleneck problem caused by the serial transmission of multiple containers, improve the task transmission rate, and ensure the real-time performance and reliability of task scheduling. BRIEF DESCRIPTION OF DRAWINGS

[0069] Figure 1 a flowchart of the method of the application;

[0070] Figure 2 a schematic diagram of the access of a container by a cluster external cloud server;

[0071] Figure 3 a flowchart of the multi-time slot container task aggregation based on dynamic programming of the application;

[0072] Figure 4 a schematic diagram of the aggregation task data packet structure of the application. DETAILED DESCRIPTION

[0073] The application will be further described in detail below in combination with the drawings and examples.

[0074] In order to make the technical solutions involved in the application more specific and clear, the application will be further described in detail below in combination with the drawings and examples. It should be noted that the specific examples described below are only used to explain the application and do not limit the application.

[0075] The application is implemented in a kind of container level network resource efficient management method for deterministic distributed computing scene, in deterministic distributed computing scene, with Kubernetes as an example, carry out container level network resource management, first need to extend network resource controller, manage the network resource of Pod layer, extend Calico network plug-in, control corresponding network bandwidth, configure Service service, so that container in cluster can communicate with outside cluster;Then, configure network policy, i.e. configure network policy to container task, control the use of network resource by the port of container task;Further, aggregate container task, improve transmission efficiency, i.e. according to the real-time requirement of container task and the maximum capacity of data packet, based on dynamic programming algorithm, multiple time slot aggregation is carried out to container task, guarantee the real-time solving optimal data packet aggregation scheme of task, output task aggregation information table and the minimum data packet number;Finally, store and transmit aggregated task, i.e. according to the task aggregation information table generated by the application, select to share the data of multiple aggregated container tasks, write and build container server and client script inside and outside cluster, deploy container image in cluster, configure and deploy container application and service port, realize data transmission inside and outside cluster.The method flow chart of the application is as shown in Figure 1 Specifically,

[0076] Extend network resource controller: extend Calico network plug-in, configure corresponding network bandwidth, configure Service service, so that container can communicate with outside cluster;

[0077] Configure network policy: define network policy according to the requirement of container task, specify policy type, apply network policy, control the use of network resource by container task;

[0078] Aggregate container task: according to the real-time requirement of container task and the maximum capacity of data packet, based on dynamic programming algorithm, multiple time slot aggregation is carried out to container task, solve optimal data packet aggregation scheme, output task aggregation information and the minimum data packet number;

[0079] The storage and transmission of aggregated task: select to share the data of multiple aggregated container tasks, write and build container server and client script inside and outside cluster, deploy container image in cluster, configure and deploy container task and service port, realize data transmission inside and outside cluster.

[0080] Specifically, the implementation of the application includes the following four steps:

[0081] Step 1: extend network resource controller;

[0082] Step 2: configure network policy;

[0083] Step 3: Aggregation container task;

[0084] Step 4: Storage and transmission of aggregation task.

[0085] The extended network resource controller in step 1 comprises:

[0086] Step 1.1: Extend Calico network plugin;

[0087] Step 1.2: Configure corresponding network bandwidth;

[0088] Step 1.3: Parse the configuration file of the Pod;

[0089] Step 1.4: Set TC rules;

[0090] Step 1.5: Configure Service service.

[0091] The step 1.1 specifically comprises:

[0092] In the deterministic distributed computing scenario, the Kubernetes cluster edge system is used as the basis, and the Calico network plugin is selected from a plurality of network plugins. Calico adopts a third layer network-based architecture, directly routes IP packets between nodes through a routing table, reduces the overhead of packet encapsulation and decapsulation, enables each Pod to have a unique, routable IP address, and improves the efficiency of Pod-to-Pod communication. When installing the Calico plugin, it is first necessary to confirm that the Kubernetes cluster has been deployed and is running, then use the kubectl command to install the Calico CNI plugin, ensure that the Calico CNI plugin is installed on the Kubernetes control plane, and further verify whether the Calico plugin is correctly deployed and running after installation is complete.

[0093] The step 1.2 specifically comprises:

[0094] On the Kubernetes cluster extended with the Calico network plugin, the network resources are configured, and the network bandwidth parameters "ingress-bandwidth" and "egress-bandwidth" are configured in the YAML file of the Pod to limit the ingress and egress bandwidth of the Pod, thereby achieving the limitation of network resources.

[0095] The step 1.3 specifically comprises:

[0096] The Kubernetes cluster parses the configuration file of the Pod and extracts the annotation part, and through this information, the network plugin Calico detects this annotation, parses the ingress and egress network bandwidth limit values of the Pod, and determines the configured network bandwidth parameters.

[0097] The step 1.4 is specifically;

[0098] Through the Calico network plugin, the TC tool using the Linux kernel function limits the QoS technology of network traffic. According to the parsed bandwidth limit value, the network plugin generates the corresponding TC (Traffic Control) rule. The TC tool schedules traffic by creating classes and queues to ensure that the network traffic of the Pod does not exceed the defined bandwidth upper limit. The TC rule operates on the qdisc (queueing discipline) layer of the network interface, using HTB (Hierarchical Token Bucket) to limit traffic. HTB provides hierarchical traffic control by allocating different classes and queues to control the maximum bandwidth of each Pod and manage the transmission rate of traffic through the token bucket algorithm. Apply the TC rule to the network interface associated with the Pod (the network interface of the node where the Pod is located). Once the TC rule takes effect, the ingress and egress traffic of the Pod will be scheduled and controlled according to the values configured in the annotation.

[0099] The step 1.5 is specifically:

[0100] In order to enable the external cloud server of the cluster to access the container in the Pod, the Service resource is introduced, which provides a fixed entry address for the Pod, so that the external cloud server can access the container in the Pod through the Service. Select the NodePort type of Service to map the port of the Service to a port on the Node edge node. External can access the service through the IP address of the Node and the mapped port number NodePort. When the request arrives at the entry address of the Service, kube-proxy will distribute the request to the corresponding container, realizing the data transmission between the container and the external cloud server. The access process is as shown in Figure 2 .

[0101] The network policy in step 2 includes:

[0102] Step 2.1: Define the network policy of the container task;

[0103] Step 2.2: Apply the defined network policy;

[0104] Step 2.3: Test the network policy.

[0105] The step 2.1 is specifically:

[0106] In deterministic distributed computing scenarios, control over network bandwidth is exercised by selecting IP addresses and ports, and controlling the use of network resources by configuring network policies. First, a network policy needs to be defined, which is done in the form of a YAML file, typically including several key sections: apiVersion, kind, metadata, and spec. Set apiVersion to networking.k8s.io / v1 and kind to NetworkPolicy. In metadata, specify the name and namespace of the network policy. Spec is the core of the network policy, defining which containers are affected by the policy and which traffic is allowed or denied. By using podSelector, specify which Pods the policy applies to, usually by matching label selectors. policyTypes specifies the type of policy, which can be Ingress (inbound traffic control), Egress (outbound traffic control), or both. In Ingress and Egress rules, specify specific IP addresses, port numbers, and protocols to define allowed and denied traffic sources or targets, to control communication between containers and external cloud servers in detail.

[0107] Step 2.2 is specifically:

[0108] According to the defined network policy of step 2.1, use the kubectl command to apply it to the cluster.

[0109] Step 2.3 is specifically:

[0110] After applying the network policy, deploy some temporary container tasks, and use tools such as curl to send network requests to test the actual effect of the policy.

[0111] The aggregation container task in step 3 includes:

[0112] Step 3.1: Container task definition

[0113] Step 3.2: Container task ordering

[0114] Step 3.3: State initialization

[0115] Step 3.4: State transition equation construction

[0116] Step 3.5: Effectiveness determination of container task aggregation

[0117] Step 3.6: Multi-time slot container task aggregation

[0118] The flowchart of the complete multi-time slot container task aggregation algorithm based on dynamic programming is as follows: Figure 3As shown, specifically, the step 3.1 is specifically:

[0119] For container tasks, the attributes of each task need to be defined, including data size, deadline, and original number of tasks (for outputting the original order). Global parameters are defined, including the maximum size limit of each data packet max_packet_size, deadline difference max_delay_diff, number of users num_users, number of tasks per user tasks_per_user, number of time slots T and time slot length t, etc.

[0120] The step 3.2 is specifically:

[0121] For user-submitted tasks, first sort the tasks in ascending order of their deadlines. The sorting ensures that tasks with similar deadlines have similar deadlines, so that tasks with similar deadlines can be aggregated in the same data packet. In subsequent state transitions, the deadline difference can be quickly evaluated to determine whether it is within the allowed range. By sorting, the complexity of dynamic programming state transition can be reduced, and the efficiency of real-time judgment can be improved.

[0122] The step 3.3 is specifically:

[0123] An array dp[k] is defined to represent the minimum number of data packets required for the first k tasks. That is, dp[k] represents the minimum number of data packets required for the optimal combination of the first k tasks, and the initial state is 0. A packets array is defined to record the task numbers included in each data packet, and the initial state is empty.

[0124] The step 3.4 is specifically:

[0125] For each task k, find the best way to combine it with the previous several tasks. To calculate dp[k], the optimal solution can be found from all possible previous cases:

[0126] dp[k] = min(dp[k], dp[j-1] + 1)

[0127] Where dp[j-1] represents the optimal solution (i.e., the minimum number of data packets) for the first j-1 tasks. Then decide whether tasks j to k can be combined into a data packet. If it can be combined (satisfying the data packet size limit and the real-time requirement of the task), it can be considered as a new data packet. Therefore, the solution of dp[k] is dp[j-1] + 1.

[0128] The step 3.5 includes the following steps:

[0129] Step 3.5.1: Validity determination based on task data packet size, in the task scheduling and aggregation problem, the data size determination of the aggregated task can avoid some potential packet loss problems of tasks due to exceeding the data packet limit.

[0130] The data packet size constraint expression C1 is,

[0131]

[0132] wherein, represents the data volume of the i-th task of user u, and M is the maximum limit of a single data packet, i.e., the sum of the data volumes of the aggregated container tasks must be less than or equal to the maximum limit of the data packet.

[0133] Step 3.5.2: Validity determination based on task real-time performance, in the task scheduling and aggregation problem, real-time performance refers to the requirement that a task needs to be completed within a certain time constraint. This is a strict requirement of the system on the time of the task, mainly embodied in the deadline of each task. For the aggregation of multiple tasks, real-time performance means the following three aspects:

[0134] Deadline difference constraint of tasks: when multiple tasks are aggregated into the same data packet, the deadline difference of these tasks cannot be too large, ensuring that these tasks can be processed at the same time and completed within the same time window.

[0135] The deadline difference constraint expression C2 of the aggregated task is,

[0136]

[0137] wherein, represents the deadline of the i-th task of user u, and Δt is the allowed deadline difference, i.e., the deadline difference of the aggregated container task must be less than or equal to the allowed deadline difference. For the allowed deadline difference Δt, a machine learning technique is used to train an AI model. The model dynamically predicts the optimal allowed deadline difference Δt of the current system according to the deadline of the task, the number of tasks, and other parameters, further improving the scheduling efficiency and task real-time performance.

[0138] Data packet latest transmission time constraint: the earliest deadline of all tasks in a data packet is the latest transmission time of the data packet, otherwise the real-time performance requirement of the task cannot be met.

[0139] The data packet latest transmission time constraint expression C3 of the aggregated task is,

[0140]

[0141] wherein, t send represents the latest transmission time of the data packet, The deadline of the i-th task of the user u, that is, when multiple container tasks are aggregated into the same data packet, the transmission time of the data packet must consider the earliest deadline in the aggregated task, ensure that the latest transmission time of the data packet is less than the earliest deadline in the aggregated task, ensure that all tasks can be transmitted in time, and ensure the real-time performance of the task.

[0142] The maximum waiting time constraint of the aggregated task: when the task aggregation is performed across time slots, in order to reduce the number of data packets, the task of the current time slot is tried to be added to the aggregated data packet of the previous time slot.

[0143] The maximum waiting time constraint expression C4 of the aggregated task is,

[0144]

[0145] Wherein, The deadline of the task submitted by the user in the current time slot, d min_p The minimum deadline of the data packet of the previous time slot, t represents the length of each time slot. That is, the deadline of the new task cannot exceed the minimum deadline of the aggregated data packet of the previous time slot minus the time slot length.

[0146] The step 3.6 is specifically:

[0147] When the time slot T = 1, the tasks are aggregated using the dynamic programming algorithm, sequentially traversing each task k, initializing the total size of the data of the current data packet, the maximum deadline and the minimum deadline of the data packet. For each task k, traverse in reverse order from task k to the front (i.e. j from k to 1), accumulate the total size of the data of the current data packet, update the maximum and minimum deadline, and determine whether the data packet size, the deadline difference of the task, and the latest transmission time requirement of the aggregated task data packet meet the requirements according to the validity judgment of step 3.5 container task aggregation. If they meet the requirements, determine whether dp[j-1]+1 is less than dp[k], if it is less than, then update dp[k] and packets[k], continue the next j loop, if it is not less than, then continue the next j loop; if they do not meet the requirements, then continue the next j loop. When the j loop ends, sequentially traverse the next task k until all tasks are sequentially traversed, and output the minimum data packet number dp[n] and the aggregated task condition of each data packet packets[n]. When the time slot T > 1, cross-slot aggregation is performed, and the tasks in the current time slot are traversed to attempt to join the tasks in the current time slot into the data packet in the previous time slot to reduce the number of data packets. Determine whether the data packet size, the deadline difference of the task, the latest transmission time requirement of the aggregated task data packet, and the maximum waiting time of the aggregated task meet the requirements. If they meet the requirements, aggregate the tasks into the existing data packet, update the minimum deadline of the current data packet, and output the task aggregation condition. If they do not meet the requirements, use the dynamic programming algorithm to aggregate the tasks in the current time slot that cannot be aggregated across time slots, and output the task aggregation condition.

[0148] The storage and transmission of the aggregated tasks in step 4 include:

[0149] Step 4.1: Storage of aggregated tasks

[0150] Step 4.2: Transmission of aggregated tasks

[0151] The step 4.1 specifically includes:

[0152] In order to avoid the overhead caused by serial transmission of each container task through the respective port number, and to improve the stability and efficiency of data transmission. It is selected to store the data of multiple container tasks in a shared storage volume emptyDir, and to splice these data into a complete data packet, and to perform unified transmission through a certain port number of a container, to reduce the number of serial transmissions between containers, to improve the task transmission rate, and to ensure the real-time performance of the tasks.

[0153] The step 4.2 includes the following steps:

[0154] Step 4.2.1: Writing and constructing container server and client scripts inside and outside the cluster

[0155] First, write the cluster TCP server script tcp_server.py, TCP client script tcp_send.py. These two scripts run in a container of the aggregation task, tcp_server.py is used to receive the data packet sent by the external client, and distribute the data to each container in the shared storage volume. tcp_send.py is used to send the data that multiple containers in the shared storage volume need to send, and send the data to the outside of the cluster through the IP address and port number. Second, write the external TCP client send.py, TCP server script receive.py. These two scripts run in the external cloud server, send.py is used to send the data that multiple containers in the shared storage volume need to send, and send the data to the container through the IP address and mapped port number NodePort of the Node. receive.py is used to receive the data that multiple containers in the shared storage volume need to send, and identify the data from which container.

[0156] Step 4.2.2: Deploy cluster container image

[0157] Create a Dockerfile to package the two Python scripts of the TCP server and client in the cluster into a container image, build the container image, and push the image to the cloud image library for subsequent container task use.

[0158] Step 4.2.3: Configure and deploy container task

[0159] Write the configuration file of the container task, configure the network bandwidth according to step 1.2, and use the container image in step 4.2.2 to provide an environment for sending and receiving of the container task.

[0160] Step 4.2.4: Expose container service port

[0161] Write the Service configuration file, configure the Service service for the container task according to step 1.5, open the container port, and make the cloud server outside the cluster can access the container.

[0162] Step 4.2.5: Cluster internal and external data sending and receiving

[0163] The transmission of data is carried out through the TCP protocol. Specifically, first, the data formatted as a byte sequence needs to be read from multiple containers. An identifier is added to the data of each container during the data packet splicing process to ensure that the receiving end can correctly parse and distinguish the data and splice these data in order to form a data packet. The aggregation task data packet structure diagram is shown in Figure 4 .

[0164] To ensure that this concatenated packet can be transmitted through a TCP segment, the size of the packet must be controlled so that it does not exceed the maximum segment length (MSS). The MSS is the maximum amount of data that each TCP segment can accommodate, which is usually slightly smaller than the maximum transmission unit (MTU) to reserve space for the IP and TCP headers. When the size of the packet is effectively controlled, the packet is encapsulated into a TCP segment and the necessary TCP header information is added, such as the source port, destination port, sequence number, and acknowledgment number, among other key fields. Then, the TCP segment is encapsulated into an IP packet, adding IP header information such as the source IP address and destination IP address. Finally, these encapsulated IP packets are transmitted over the network. When transmitting packets, the latest time to send the packet must be less than the earliest deadline of the aggregated task to ensure that all tasks are transmitted in a timely manner, ensuring the real-time nature of the tasks.

[0165] Similarly, if the container needs to obtain data from the external cloud server, the data to be transmitted will also undergo similar processing. The data to be sent to each container is concatenated into a complete packet, ensuring that each part represents the data required by a different container. Before sending the data, the system checks whether the length of the concatenated packet meets the MSS limit. If the length of the packet is within the allowed range, the packet will be sent to the specified container through the TCP connection.

Claims

1. A method for efficient management of container-level network resources in deterministic distributed computing scenarios, characterized in that, Includes the following steps: 1) Expand the network resource controller; 2) Configure network policies according to the requirements of container tasks; 3) Aggregate container tasks across multiple time slots; 4) Store and transmit the aggregated container tasks; Step 1) includes the following steps: 1.1) Extend the Calico network plugin; 1.2) Configure network bandwidth; 1.3) Parse the Pod's configuration file; 1.4) Set TC rules; 1.5) Configure the Service; Step 3) includes the following steps: 3.1) Define the properties and global parameters for each container task; 3.2) Sort the container tasks according to their deadlines; 3.3) State initialization; 3.4) Construct the state transition equations; 3.5) Determine the validity of container task aggregation; 3.6) Perform multi-slot container task aggregation; Step 3.5) includes the following steps: 3.5.1) Validity determination based on task data packet size: The packet size constraint expression C1 is: ； in, M represents the data size of the i-th task of user u, and M is the maximum limit of a single data packet, meaning that the sum of the data sizes of the aggregate container tasks must be less than or equal to the maximum limit of the data packets. 3.5.2) Validity determination based on task real-time performance: a) The deadline difference constraint expression C2 for the aggregation task is: ； in, This represents the deadline for the i-th task of user u in the current time slot. The allowable deadline difference means that the deadline difference of the aggregate container task must be less than or equal to the allowable deadline difference; b) The latest data packet transmission time constraint expression C3 for the aggregation task is: ； in, This indicates the latest transmission time of the data packet. When multiple container tasks are aggregated into the same data packet, the latest transmission time of the data packet must be less than the earliest deadline in the aggregated task. c) The maximum latency constraint expression C4 for the aggregation task is: ； in, This indicates the minimum deadline for the data packet in the previous time slot, and t represents the length of each time slot. That is, the deadline for the new task cannot exceed the minimum deadline of the aggregated data packet in the previous time slot minus the time slot length. Step 3.6) includes the following steps: 3.6.1) When time slot T=1, sequentially traverse each task k, and initialize the total data size of the current data packet, the maximum deadline time of the data packet, and the minimum deadline time of the data packet; 3.6.2) For each task k, iterate backwards from task k, accumulate the total size of the current data packet, update the maximum and minimum deadlines, and determine whether the validity of the container task aggregation meets the requirements, i.e., whether C1, C2 and C3 all meet the validity requirements. If the requirements are met, proceed to step 3.6.3); otherwise, continue to the next j loop. 3.6.3) Judgment Is it less than If it is less than, then update. and If the condition is met, continue to the next j loop; otherwise, continue directly to the next j loop. 3.6.4) When loop j ends, sequentially traverse the next task k until all tasks have been sequentially traversed, and output the result. Minimum number of packets and Information on each data packet aggregation task; 3.6.5) When time slot T>1, perform cross-time slot aggregation, traverse the tasks in the current time slot, add the tasks in the current time slot to the data packets of the previous time slot, and determine whether the validity of container task aggregation meets the requirements, that is, whether C1, C2, C3 and C4 all meet the validity determination requirements. If the requirements are met, aggregate the tasks to the existing data packets, update the minimum deadline of the current data packets, and output the task aggregation status. If the requirements are not met, execute steps 3.6.1) to 3.6.4) to aggregate the tasks that could not be aggregated across time slots in the current time slot, and output the task aggregation status.

2. The method for efficient management of container-level network resources in deterministic distributed computing scenarios according to claim 1, characterized in that, Step 1.5) specifically refers to: Using a NodePort type Service, the Service's port is mapped to a port on a Node edge node. When a request arrives at the Service's entry address, kube-proxy distributes the request to the corresponding container, enabling data transmission between the container and the external cloud server.

3. The method for efficient management of container-level network resources in deterministic distributed computing scenarios according to claim 1, characterized in that, Step 2) includes the following steps: 2.1) Define the network policy for container tasks; 2.2) Use the kubectl command to apply the defined network policy to the cluster; 2.3) Deploy temporary container tasks to test network policies.

4. The method for efficient management of container-level network resources in deterministic distributed computing scenarios according to claim 3, characterized in that, Step 2.1) specifically refers to: Network policies are defined in the form of YAML files, including: apiVersion, kind, metadata, and spec. apiVersion specifies the API version, kind specifies the policy type, metadata specifies the network policy name and namespace, and spec specifies the containers affected by the policy and the traffic that is allowed or denied. Within the spec, podSelector specifies which pods the policy applies to, and policyTypes specifies the policy type, which includes inbound traffic control (Ingress), outbound traffic control (Egress), or both. In Ingress and Egress rules, specific IP addresses, port numbers, and protocols are specified to define the allowed and denied traffic sources or destinations, controlling communication between containers and external cloud servers.

5. The method for efficient management of container-level network resources in deterministic distributed computing scenarios according to claim 1, characterized in that, Step 3.4) specifically refers to: For each task k, find the optimal way to combine it with the previous tasks and construct the state transition equation: ； in, This represents the minimum number of data packets required for the first k tasks. This represents the optimal solution for the first j-1 tasks, i.e., the minimum number of data packets.

6. The method for efficient management of container-level network resources for deterministic distributed computing scenarios according to claim 1, characterized in that, Step 4) includes the following steps: 4.1) Store the aggregated tasks; 4.2) Transmit the stored results: 4.2.1) Write and build container server and client scripts for both inside and outside the cluster; 4.2.2) Deploy the cluster container image; 4.2.3) Configure and deploy container tasks; 4.2.4) Expose container service ports; 4.2.5) Sending and receiving data inside and outside the cluster.

Images