Rootfs quota management system and method, computer equipment and storage medium

Abstract

The embodiment of the invention provides a Rootfs quota management system and method, computer equipment and a storage medium, and relates to the technical field of computers. The Rootfs quota management system comprises a main control component, a working component and an instruction execution component corresponding to the working component, wherein the working component can be connected with a container of a working node during operation; the main control component is used for acquiring a task quota request of a target task, the target task corresponds to a target container, and a corresponding root file system Rootfs is mounted on the target container; querying quota management data based on the task quota request, determining a quota instruction, and sending the quota instruction to the target working component; and the working component is used for receiving the limit instruction sent by the main control component under the condition that the working component is successfully connected with the container during operation, and calling the instruction execution component to execute the limit on the Rootfs writable layer of the target container if the target container corresponding to the container identifier carried by the limit instruction exists and is in an operation state, so that the flexibility and compatibility of the limit can be improved.

Description

TECHNICAL FIELD

[0001] The embodiments in the specification relate to the technical field of computer technology, and particularly relate to a Rootfs quota management system and method, computer equipment and storage medium. BACKGROUND

[0002] A container orchestration platform is mainly used for automatically managing the deployment, expansion and operation and maintenance of containerized applications, and ensuring that the application runs stably in the production environment. Kubernetes (K8s for short) is a mainstream container orchestration platform, which can solve the problems of resource scheduling, fault recovery, elastic scaling and the like in container management.

[0003] In the related art, when the K8s platform performs quota on the container Rootfs, it often needs to highly depend on a specific container runtime (Container Runtime) implementation. The container runtime is responsible for managing the entire life cycle of the container, is the basis for running the container, is responsible for pulling the container image, starting and stopping the container, and providing the environment required for running the container. The specific container runtime can include Docker, Containerd Runtime, and the like. In the related art, such a quota mechanism depending on the specific container runtime easily causes the related system under the K8s platform to be bound to a specific runtime ecosystem, and the flexibility of the storage resource quota or limit is low, and the compatibility is poor.

[0004] Therefore, it is urgent to provide a Rootfs quota management system to improve the flexibility and compatibility of the storage resource quota. SUMMARY

[0005] Therefore, the embodiments in the specification are committed to providing a Rootfs quota management system and method, computer equipment and storage medium to improve the flexibility and compatibility of the storage resource quota.

[0006] The embodiments of the present specification provide a Rootfs quota management system, comprising a master component, at least one working component and an instruction execution component corresponding to each of the working components, the working components can respectively interact with the master component and the corresponding instruction execution component, the master component is deployed on a master node of a container orchestration platform, and any of the working components and the corresponding instruction execution component are deployed on any working node of the container orchestration platform; the working component can be connected with a container runtime in the working node; the master component is configured to obtain a task quota request of a target task, wherein the target task corresponds to at least one target container, the at least one target container is scheduled on one or more working nodes, the working component in the working node scheduling the target container is a target working component, and the target container is mounted with a corresponding root file system Rootfs; quota management data is queried based on the task quota request to determine a quota instruction, wherein the quota management data describes or records quota information related to the task; and the quota instruction is sent to the target working component; the working component is configured to, in the case that the connection with the container runtime is successful, receive the quota instruction sent by the master component, wherein the quota instruction carries a container identifier; if the target container corresponding to the container identifier carried by the quota instruction exists and is in a running state, the writable layer path of the root file system Rootfs mounted by the target container is obtained; and the instruction execution component is called to execute the quota in the Rootfs writable layer of the target container.

[0007] In some embodiments, the Rootfs quota management system further comprises a task controller component and an APIServer component deployed in the master node, and the APIServer component can respectively interact with the master component and the task controller component; the task controller is configured to receive the target task submitted by a user, create the target container corresponding to the target task by using the APIServer component, and schedule the target container to run on any of the working nodes; and in the case that the target container is scheduled to run on any of the working nodes, the task quota request is initiated to the master component.

[0008] In some embodiments, the quota management data includes a quota policy configuration table and / or a task quota status table; the quota policy configuration table is used to describe the correspondence between the task type and the default quota, so as to enable differentiated quota based on the task type of the target task; wherein the task type is divided based on the required storage space of the task; the task quota status table is obtained by querying the running state, the current quota, the current usage, and the node distribution of the container corresponding to the task in the current running of the container orchestration platform by the APIServer component, so as to enable dynamic adjustment of the quota of the target task in running.

[0009] The embodiments of the present specification provide a Rootfs quota management method, applied to a master component of a Rootfs quota management system, the quota management system further comprising at least one working component and an instruction execution component corresponding to each working component, the working component being capable of interacting with the master component and the corresponding instruction execution component respectively, the master component being deployed on a master node of a container orchestration platform, and any working component and the corresponding instruction execution component thereof being deployed on any working node of the container orchestration platform; the method comprises: obtaining a task quota request of a target task; wherein the target task corresponds to at least one target container, the at least one target container being scheduled on one or more working nodes, the working component in the working node scheduling the target container being a target working component, and the target container being mounted with a corresponding root file system Rootfs; querying quota management data based on the task quota request to determine a quota instruction; wherein the quota management data describes or records quota information related to the task; and sending the quota instruction to the target working component, so that the target working component invokes the instruction execution component to execute the quota in the Rootfs writable layer of the target container.

[0010] In some embodiments, the task quota request is an initialization quota request or a dynamic adjustment quota request, the initialization quota request carrying a task type of the target task; the quota management data includes a quota policy configuration table, the quota policy configuration table being used to describe the correspondence between the task type and the default quota; and the querying of the quota management data based on the task quota request to determine the quota instruction comprises: in the case that the task quota request is the initialization quota request, querying the quota policy configuration table based on the task type of the target task to obtain a target quota value; and generating a quota instruction; wherein the quota instruction carries the target quota value.

[0011] In some embodiments, the task quota request is an initialization quota request or a dynamic adjustment quota request, the dynamic adjustment quota request carries a task identifier of the target task and an application quota value; the quota management data includes a task quota state table, the task quota state table is used to describe a current quota and a current usage of a container corresponding to a task currently running; the querying the quota management data based on the task quota request to determine a quota instruction includes: in the case that the task quota request is the dynamic adjustment quota request, querying the task quota state table based on the task identifier of the target task to obtain the current quota and the current usage of a target container corresponding to the target task; if the application quota value of the target task, the current quota and the current usage of the target container satisfy a set quota adjustment condition, taking the application quota value as a target quota value; generating the quota instruction; wherein the quota instruction carries the target quota value.

[0012] In some embodiments, the task quota request carries a task identifier of the target task and a corresponding container identifier, the task identifier is used to uniquely identify the corresponding task, and the container identifier is used to uniquely identify a container of the corresponding task; the quota management data includes a task quota state table, the task quota state table is used to describe a node distribution of a container corresponding to a task currently running or an initialization task; the querying the quota management data based on the task quota request to determine a quota instruction includes: querying the task quota state table based on the task identifier of the target task and the corresponding container identifier to determine a task state of the target task and the corresponding target work component; in the case that the task state of the target task is in a running state or an initialization state, generating the quota instruction; wherein the quota instruction carries the container identifier corresponding to the target task.

[0013] The embodiments of the present specification provide a Rootfs quota management method, applied to a work component of a Rootfs quota management system, the quota management system further includes a master component and an instruction execution component corresponding to the work component, the work component can respectively interact with the master component and the corresponding instruction execution component, the master component is deployed on a master node of a container orchestration platform, the work component and the corresponding instruction execution component are deployed on a worker node of the container orchestration platform, and the work component can be connected with a container runtime in the worker node; the method comprises: in the case that the connection with the container runtime is successful, receiving a quota instruction sent by the master component; wherein the quota instruction carries a container identifier; if the target container corresponding to the container identifier carried by the quota instruction exists and is in a running state, obtaining a writable layer path of a root file system Rootfs mounted by the target container; and calling the instruction execution component to execute a quota on the Rootfs writable layer of the target container.

[0014] The embodiment of the present specification provides a computer device, comprising a memory and a processor, the memory stores a computer program, and the processor implements the Rootfs quota management method of any of the above embodiments when executing the computer program.

[0015] The embodiment of the present specification provides a computer readable storage medium, which stores a computer program, and the computer program implements the Rootfs quota management method of any of the above embodiments when executed.

[0016] In the plurality of embodiments provided by the present specification, first, the Rootfs quota management system comprises a master component, at least one working component, and an instruction execution component corresponding to each working component, the master component is deployed on the master node of the container orchestration platform, any working component and the corresponding instruction execution component are deployed on any working node of the container orchestration platform, the working component can interact with the master component and the corresponding instruction execution component respectively, and the working component can be connected with the container runtime in the working node. Next, the master component obtains a task quota request of a target task corresponding to at least one target container, the at least one target container is scheduled in one or more working nodes, the working component in the working node where the target container is scheduled is called a target working component, and any target container is mounted with a corresponding root file system Rootfs. Then, the master component queries the quota management data based on the task quota request to determine the quota instruction, and sends the quota instruction to the target working component. Then, the working component receives the quota instruction carrying the container identifier sent by the master component in the case of successful connection with the container runtime. If the target container corresponding to the container identifier carried by the quota instruction exists and is in a running state, the working component obtains the writable layer path of the root file system Rootfs mounted by the target container, and calls the instruction execution component to execute the quota on the Rootfs writable layer of the target container. In this way, the Rootfs quota can be implemented without relying on a specific type of container runtime, and the flexibility and compatibility of the storage resource quota can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 A schematic diagram of the Rootfs quota management system provided by the embodiment of the present specification; Figure 2 An interaction schematic diagram of the Rootfs quota management method provided by the embodiment of the present specification; Figure 3 Another interaction schematic diagram of the Rootfs quota management method provided by the embodiment of the present specification; Figure 4A flowchart of a Rootfs quota management method provided by an embodiment of the present specification is shown in the figure; Figure 5 A schematic diagram of a computer device provided by an embodiment of the present specification is shown in the figure. DETAILED DESCRIPTION

[0018] In order to enable those skilled in the art to better understand the present specification, the technical solutions in the present specification will be described clearly and completely in conjunction with the accompanying drawings in the embodiments of the present specification. Obviously, the described embodiments are only part of the embodiments of the present specification, not all the embodiments. Based on the embodiments in the present specification, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present specification.

[0019] In related technologies, taking the container orchestration platform K8s as an example, when the quota for the container is set, it needs to rely on a specific container runtime such as Docker, Containerd, etc. to implement, which leads to the fact that when the container runtime needs to be replaced, the quota mechanism may fail, the flexibility and compatibility of the quota under the K8s platform are low, and the utilization rate of storage resources is low. Therefore, the flexibility and compatibility of the container quota under the container orchestration platform need to be improved.

[0020] In still other related technologies, a cluster Cluster can refer to a supercomputer abstraction composed of nodes Node, which can be physical servers or virtual machines, and the nodes Node can be installed with a K8s platform. K8s is a master-slave device architecture, that is, the nodes in the cluster can include a master node, which can be represented by Master, and other nodes are worker nodes, which can be represented by Node or Worker. The master node is responsible for the scheduling, management and operation and maintenance of the cluster, and the worker node is the computing workload node in the cluster. In other words, K8s includes a master node and one or more worker nodes corresponding to the master node.

[0021] The embodiments of the present specification provide a Rootfs quota management system, which can include a master component, at least one worker component, and an instruction execution component corresponding to each worker component. Taking the container orchestration platform Kubernetes as an example for description.

[0022] Please refer to Figure 1 , Figure 1A schematic diagram of a Rootfs quota management system provided for an embodiment of the present specification. Rootfs Quota-Master represents a master component, Rootfs Quota-Worker represents a worker component, and CMD-Agent represents a command execution component. The master component can be deployed on a master node of a container orchestration platform K8s, and any worker component and its corresponding command execution component can be deployed on any worker node of the container orchestration platform K8s. The worker component can interact with the master component and the corresponding command execution component respectively, and the worker component can connect with a container runtime in the worker node. Exemplarily, the container runtime can be Docker, Containerd or other container runtimes, which are not limited herein.

[0023] In the embodiment, the master component can obtain a task quota request of a target task, determine a quota command based on the task quota request, and send the quota command to a target worker component in at least one worker component. The worker component can receive the quota command sent by the master component in the case of successful connection with the container runtime, and if it is determined that the corresponding target container exists and is in a running state, the worker component can call the command execution component to perform quota on the target container Rootfs.

[0024] The target task can be an artificial intelligence (AI) task or a task related to AI. For example, the AI task can be a phased task at different stages of the life cycle, such as model development, model training, model inference, etc.

[0025] The target task corresponding to the task quota request obtained by the master component has at least one target container, or in other words, the running target task can correspond to one or more Pods, and the Pod and the container can have a one-to-one correspondence. The at least one target container is scheduled on one or more worker nodes, and the worker component in the worker node where the target container is scheduled is the target worker component. The target container is mounted with a corresponding root file system Rootfs.

[0026] Specifically, the master component can store or maintain quota management data, wherein the quota management data describes or records quota information related to the task. The master component can query the quota management data based on the task quota request, and determine a quota instruction according to a query result of the quota management data, and then send the quota instruction to the target worker component. The worker component can receive the quota instruction carrying the container identifier sent by the master component in the case of successful connection with the container runtime, and if the target container corresponding to the container identifier carried by the quota instruction exists and is in a running state, obtain the writeable layer path of the Rootfs mounted by the target container, and call the instruction execution component to execute the quota in the Rootfs writeable layer of the target container.

[0027] In the above embodiment, by deploying the master component in the master node and deploying the worker component and the corresponding instruction execution component in the worker node, when performing the quota, first, the master component obtains the task quota request of the target task corresponding to the target container, wherein the target task of the target container is scheduled in one or more worker nodes, the worker component in the worker node scheduling the target container is called the target worker component, and any target container is mounted with a corresponding root file system Rootfs, then the master component queries the quota management data based on the task quota request to determine the quota instruction, and sends the quota instruction to the target worker component, and then the worker component receives the quota instruction carrying the container identifier sent by the master component in the case of successful connection with the container runtime, if the target container corresponding to the container identifier carried by the quota instruction exists and is in a running state, the worker component obtains the writeable layer path of the Rootfs mounted by the target container, and calls the instruction execution component to execute the quota in the Rootfs writeable layer of the target container, in this way, the Rootfs quota can be performed without relying on a specific type of container runtime, so that a plurality of or any runtime environment such as CRI-O, Docker, Containerd, etc. can be compatible, reducing the problem of invalidation of the quota mechanism caused by replacement of the runtime, improving the flexibility and compatibility of the storage resource quota.

[0028] In some embodiments, in the K8s platform, the worker component can be run in the form of DaemonSet, responsible for cooperating with the instruction execution component to perform specific quota operations. The worker component can communicate with the master component through the HTTP protocol to receive the quota instruction and feedback the quota instruction execution result. The worker component can communicate with the corresponding container runtime through the gRPC protocol to obtain the state information of the container corresponding to the target task.

[0029] In some embodiments, when the work component invokes the instruction execution component to execute the quota instruction, the instruction execution component is executed in the form of XFS that meets the Rootfs requirement in the work node, i.e., the host machine, and can implement operations such as querying the quota, initializing the quota, or adjusting the quota. The instruction execution component can return the execution result to the corresponding work component. Illustratively, the instruction execution component encapsulates the xfs instruction operation of the work node or the host machine, and can provide a unified interface for the corresponding work component to invoke through the socks protocol, ensuring the safety and consistency of instruction execution. At the same time, the execution result of the quota instruction can be parsed and processed and returned to the work component.

[0030] In some embodiments, please refer to Figure 1 , the Rootfs quota management system can further include a task controller component and an APIServer component deployed in the master node, and the APIServer component can interact with the master component and the task controller component, respectively.

[0031] Specifically, the task controller can receive a target task submitted by a user, create a target container corresponding to the target task by using the APIServer component, and schedule the target container to run on any work node, or in other words, create and schedule a Pod corresponding to the target task. After scheduling the target container to run on any work node, the task controller can initiate a task quota request for the target task to the master component, so as to perform quota on the Rootfs of the target container of the target task through the master component, the work component, and the instruction execution component. As an example, taking an AI task as the target task, the task controller component can be an AI task controller component. In this way, the flexibility of the target task can be achieved, or in other words, the configuration flexibility of the target container corresponding to the target task can be achieved. By combining the quota management data, the configuration flexibility and dynamic adjustment of the quota strategy for the target task can be further improved, meeting various business requirements, such as the quota of the Rootfs space of the target task running in Containerd, improving the adaptability of the system.

[0032] In some embodiments, the quota management data can include a quota strategy configuration table and / or a task quota state table.

[0033] In the present embodiment, in order to reduce the situation that resource allocation is unreasonable and cannot provide differentiated quota strategies for different types of tasks, the quota strategy configuration table can be used to describe the correspondence between the task type and the default quota, or in other words, the correspondence between the task type and the maximum quota, so that when the quota instruction is determined based on the quota management data and the Rootfs of the target container is quotaed, differentiated quota can be implemented based on the task type of the target task.

[0034] Exemplarily, the task types can be classified based on the required storage space of the tasks. Taking the target task as an AI task as an example, the required storage space is different at different stages of the life cycle of the AI task, such as the model development stage, the model training stage, and the model inference stage. Therefore, the task types in the quota policy configuration table can be determined according to the required storage space at different stages of the life cycle of the AI task, that is, the task types of the quota policy configuration table can at least include model development, model training, and model inference, and the default quota / maximum quota corresponding to each of the task types is different.

[0035] For example, the quota policy configuration table supports a flexible and configurable quota policy, and different quota policies can be set according to the AI task type to adapt to the needs of different AI tasks. Model development can provide a relatively large quota, such as 20 GB, to meet the needs of installing dependencies and storing temporary files during development. Model training can set a dynamic quota range according to the training data volume and model size, for example, 10 GB-50 GB, which can be configured through the parameters when submitting the AI task. Model inference can be set to a smaller quota, such as 5 GB, since it mainly reads data and outputs inference results. In this way, the quota can be allocated accordingly for different task types to optimize resource allocation and significantly improve storage resource utilization.

[0036] In this embodiment, the task quota status table can be obtained by querying the running state, current quota, current usage, and node distribution of the container corresponding to the current running task of the container orchestration platform K8s through the API server component, so that when the quota instruction is determined based on the quota management data and the target container Rootfs quota is adjusted, the task quota status table can be used to dynamically adjust the quota of the running target task that has been limited. In this way, when the resources required by the program running in the container corresponding to the target task change, the user can adjust the quota according to the specific circumstances of different business scenarios and task requirements, avoiding the situation that the quota cannot be dynamically adjusted again after the container of the task is started during the entire life cycle of the container, and significantly improving the task efficiency and operation stability.

[0037] Exemplarily, the master component can be deployed in the form of Deployment on the k8s cluster, and the master component can communicate with external components through the HTTP protocol / RESTful API, interact with the cluster through the container orchestration platform such as the K8s API client library, and query the related information of the tasks and nodes.

[0038] The embodiment of the present specification provides a Rootfs quota management method, which is described below taking a container orchestration platform as K8s and a target task as an AI task. Please refer to Figure 2 , Figure 2 The embodiment provides an interactive diagram of a Rootfs quota management method. The embodiment provides operation steps of the method as shown in the interactive diagram, but more or fewer operation steps can be included based on conventional or non-creative labor. The order of steps listed in the embodiment is only one of the many execution orders, and does not represent the only execution order. In actual system or server product execution, the method order shown in the embodiment can be executed in sequence or in parallel (for example, in a parallel processor or multi-threaded processing environment). The Rootfs quota management method can be applied to the master component of the Rootfs quota management system, as shown in Figure 2 The Rootfs quota management method can include the following steps.

[0039] Step S210: Obtain a task quota request of a target task; wherein the target task corresponds to at least one target container, the at least one target container is scheduled on one or more worker nodes, a worker component in a worker node where the target container is scheduled is a target worker component, and the target container is mounted with a corresponding root file system Rootfs.

[0040] Specifically, the task controller can initiate a task quota request to the master component through the HTTP protocol to request a quota configuration.

[0041] Step S220: Query quota management data based on the task quota request to determine a quota instruction; wherein the quota management data describes or records quota information related to the task.

[0042] Step S230: Send the quota instruction to the target worker component to make the target worker component call an instruction execution component to execute the quota in the Rootfs writable layer of the target container.

[0043] Specifically, the master component can send the quota instruction to the target worker component through the HTTP protocol to make the worker component receiving the quota instruction call the corresponding instruction execution component to execute the quota instruction. Illustratively, the instruction execution component can execute in the writable layer of the target container Rootfs in the form of XFS, or in other words, the instruction execution component can execute the XFS quota instruction and return the execution result to the worker component. After receiving the execution result returned by the instruction execution component, the worker component can feed back the execution result to the master component. After receiving the execution result fed back by the worker component, the master component can return the quota result to the task controller, and the task controller can determine whether to continue the subsequent task.

[0044] Exemplarily, the task controller is configured with a corresponding platform interface through which a user can interact with the task controller, and the task controller can display the quota result through the platform interface. The quota result includes quota failure or quota success. If the quota fails, the task controller can notify the user and terminate the job.

[0045] In some embodiments, the task quota request can carry at least one or more of a task identifier, a container identifier, and a task type. The task identifier is used to uniquely identify the corresponding task, and the container identifier is used to uniquely identify the container of the corresponding task. The task identifier can be referred to as a task ID, and the container identifier can be referred to as a container ID. The task quota request can be an initialization quota request or a dynamic adjustment quota request.

[0046] The quota management data can include a quota policy configuration table and a task quota state table, wherein the quota policy configuration table is used to describe the correspondence between the task type and the default quota, and the task quota state table is used to describe the current quota, the current usage, and the node distribution of the corresponding container of the currently running task or the initialized task.

[0047] In some embodiments, the initialization quota request can carry at least the task type of the target task. In step S220, based on the task quota request, the quota management data is queried to determine the quota instruction, which can include steps S310-S320.

[0048] Step S310: In the case of an initialization quota request, the target quota value is obtained by querying the quota policy configuration table based on the task type of the target task.

[0049] In some cases, please continue to refer to Figure 2 , the user can submit a target task to the task controller, the target task can be an AI task, and the target task contains information such as a task type and a task name. After receiving the target task, the task controller can use the APIServer component in K8s to create a target container / Pod corresponding to the target task, and schedule it to run on any worker node. Then, the task controller can initiate an initialization quota request to the master component through the HTTP protocol.

[0050] Specifically, the master component can query the quota policy configuration table according to the task type of the target task carried by the initialization quota request, obtain the default quota / maximum quota corresponding to the task type of the target task, and take it as the target quota value. For example, the request parameters of the initialization quota request can include: "task-id": "a16514557146296320444804", "task-type": model-training. Among them, task-id represents task identification, task-type represents task type, and model-training represents that the task type is a model training type.

[0051] Step S320: generating a quota instruction; wherein the quota instruction carries the target quota value.

[0052] For example, after generating the quota, the master component can send a quota instruction to the corresponding target worker node to instruct the target worker node to cooperate with the instruction execution component to limit the target container in the form of SFX. The parameters carried by the quota instruction can include: "container-id": "f9ae6ce6d1a2c393f969f539bddc719ef93d689736be002a3c612a844add9449", "size": "40G". Among them, container-id represents container identification, and size represents the target quota value.

[0053] In some embodiments, the dynamic adjustment quota request can at least carry the task identification of the target task and the application quota value. In step S220, based on the task quota request to query the quota management data, the quota instruction can include steps S410-S430.

[0054] Step S410: in the case of a dynamic adjustment quota request, querying the task quota state table based on the task identification of the target task to obtain the current quota and the current usage of the target container corresponding to the target task.

[0055] In some cases, referring to Figure 3 , the user can initiate a dynamic adjustment quota request to the task controller through the platform interface or API to apply for adjusting the quota of the target task, for example, to apply for expanding or reducing the current quota of the container corresponding to the target task. The dynamic adjustment quota request carries the target size of the application quota value. After receiving the dynamic adjustment quota request of the user, the task controller can forward the dynamic adjustment quota request to the master component.

[0056] Specifically, after receiving the dynamic adjustment request, the master component can obtain the current limit, current usage and remaining disk space of the target container corresponding to the target task by querying the task limit state table, or querying the APIServer component, or querying the worker node.

[0057] Step S420: If the application limit value of the target task, the current limit and the current usage of the target container satisfy the set limit adjustment condition, the application limit value is taken as the target limit value.

[0058] Specifically, if the application limit value is less than the current limit of the target task, and the application limit value is greater than the current usage, it is determined that the limit adjustment condition is satisfied, and the limit can be subsequently executed by the worker component in cooperation with the instruction execution component; otherwise, a failure is returned, and the user is informed of the reason, for example, the application limit value is too small to meet the actual usage, etc. The current limit, i.e., the current Rootfs limit of the target task; the current usage, i.e., the actual Rootfs size of the target task.

[0059] If the application limit value is greater than the current limit of the target task, and the application limit value is less than the remaining disk space, it is determined that the limit adjustment condition is satisfied, and the limit can be subsequently executed by the worker component in cooperation with the instruction execution component; otherwise, a failure is returned, and the user is prompted of the reason, for example, the disk space is insufficient to meet the application, etc.

[0060] Step S430: generating a limit instruction; wherein the limit instruction carries the target limit value.

[0061] Exemplarily, in the case of a task limit request being a dynamic adjustment limit request, the obtained limit instruction is an adjustment instruction.

[0062] In some embodiments, the task limit request carries a task identifier of the target task and a corresponding container identifier. In step S220, based on the task limit request querying the limit management data to determine the limit instruction can include steps S510-520.

[0063] Step S510: querying the task limit state table based on the task identifier of the target task and the corresponding container identifier to determine the task state of the target task and the corresponding target worker component.

[0064] In some cases, please continue to refer to Figure 2 or Figure 3 , after receiving the task limit request such as the initialization limit request or the dynamic adjustment limit request, the master component can obtain the task state of the target task, and obtain the node information of the target worker component corresponding to the target task by querying the task limit state table, or querying the APIServer component.

[0065] Step S520: generating a quota instruction in the case that the task state of the target task is in a running state or an initialization state; wherein the quota instruction carries a container identifier corresponding to the target task.

[0066] If the target task is not in a running state, the master component can return an error message to the user, prompting that the target task is not running and cannot be limited.

[0067] The Rootfs quota management method provided by the embodiments of the present specification can be applied to a working component in a Rootfs quota management system. Please refer to Figure 4 The Rootfs quota management method can include the following steps.

[0068] Step S610: receiving a quota instruction sent by the master component in the case that the connection with the container runtime is successful; wherein the quota instruction carries a container identifier.

[0069] Specifically, please refer to Figure 4 In the process of Rootfs quota management, the working node and the corresponding working component need to be started first. After the working component is started, the container runtime is connected through gRPC. If the connection fails, exponential backoff reconnection can be performed until the connection is successful.

[0070] Among them, exponential backoff reconnection can mean reconnection at 0th power, 1st power, 2nd power, 3rd power…nth power of the specified time length from the current connection failure time, until the maximum reconnection time length is set, for example, if the current time fails to connect, the specified time length is set to 5 seconds, and the maximum reconnection time length is set to 5 minutes, then the reconnection can be performed at 1 second, 5 seconds, 25 seconds…after the current time, until 5 minutes or the connection is successful.

[0071] Step S620: if the target container corresponding to the container identifier carried by the quota instruction exists and is in a running state, obtaining the writable layer path of the root file system Rootfs mounted by the target container.

[0072] Specifically, after the working component is successfully connected with the container runtime, the quota instruction can be waited for. After receiving the quota instruction, if the target container does not exist or is not running, a failure is returned; if it exists and is running, the writable layer path Upperdir of the target container is obtained.

[0073] Step S630: calling an instruction execution component to execute the quota in the Rootfs writable layer of the target container.

[0074] Specifically, after the working component calls the instruction execution component to execute the quota, the execution result can be returned to the master component.

[0075] In the above embodiment, through the cooperative matching among the master component, the working component and the instruction execution component, the initialization or dynamic adjustment of the Rootfs quota for a single container can be realized, such as the capacity expansion requirement in the middle of an AI training task. In addition, the xfs is directly called through an independent service at the host level, thereby improving the runtime independence.

[0076] In the above embodiment, through the cooperative matching among the master component, the working component and the instruction execution component, the integration of the quota logic in the native scheduling process of the container orchestration platform K8s can be improved while avoiding deep modification or extension of the container runtime, thereby increasing the complexity and maintenance cost thereof.

[0077] The embodiments of the present specification also provide a computer readable storage medium having a computer program stored thereon, the computer program being executed by a computer to implement the Rootfs quota management method in any of the above embodiments.

[0078] The embodiments of the present specification also provide a computer program product containing instructions, the instructions being executed by a computer to cause the computer to implement the Rootfs quota management method in any of the above embodiments.

[0079] The embodiments of the present specification also provide a computer device including a memory and a processor, the memory having a computer program stored therein, the computer program being executed by the processor to implement the Rootfs quota management method in any of the above embodiments.

[0080] In some embodiments, referring to Figure 5 , the computer device can be a terminal, and an internal structure diagram thereof can be as shown in Figure 5 . The computer device includes a processor, a memory and a communication interface connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The communication interface of the computer device is configured to perform wired or wireless communication with an external terminal. The wireless communication can be achieved through WIFI, a carrier network, NFC (Near Field Communication) or other technologies. The computer program is executed by the processor to implement the Rootfs quota management method.

[0081] It can be understood that the specific examples herein are only to help those skilled in the art better understand the embodiments of the present specification, and not to limit the scope of the present application.

[0082] It can be understood that, in various embodiments in the present specification, the size of the sequence number of each process does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments in the present specification.

[0083] It can be understood that the various embodiments described in the present specification can be implemented alone or in combination, and the embodiments in the present specification do not limit this.

[0084] Unless otherwise specified, all technical and scientific terms used in the embodiments of the present specification have the same meanings as understood by those skilled in the art of the technology of the present specification. The terms used in the present specification are only for the purpose of describing the specific embodiments and are not intended to limit the scope of the present specification. The term "and / or" used in the present specification includes any and all combinations of one or more of the related listed items. The singular forms "a", "an" and "the" used in the embodiments of the present specification and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise.

[0085] It can be understood that the processor of the embodiments of the present specification can be an integrated circuit chip with signal processing capability. In the implementation process, each step of the above method embodiments can be completed by the integrated logic circuit of hardware in the processor or the instruction in the form of software. The processor described above can be a general processor, a digital signal processor (Digital Signal Processor, DSP), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component. The disclosed methods, steps and logic block diagrams in the embodiments of the present specification can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in combination with the embodiments of the present specification can be directly embodied as a hardware code processor for execution, or a combination of hardware and software modules in the code processor for execution. The software module can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register or other mature storage medium in the art. The storage medium is located in the memory, and the processor reads the information in the memory, and combines the hardware to complete the steps of the above method.

[0086] It is appreciated that the memory in the embodiments of the present specification can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM). It should be noted that the memory of the system and method described herein is intended to include, but not limited to, these and any other suitable types of memory.

[0087] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in connection with the embodiments disclosed herein can be realized in electronic hardware, or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present specification.

[0088] Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working processes of the above-described system, device and unit can refer to the corresponding processes in the foregoing method embodiments, which will not be described here.

[0089] The above is only a specific implementation of the present specification, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present specification, which should be covered within the protection scope of the present specification. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A Rootfs quota management system, characterized in that: The system comprises a main control component, at least one working component, and an instruction execution component corresponding to each working component. The working component can interact with the main control component and the corresponding instruction execution component respectively. The main control component is deployed on the master node of the container orchestration platform, and any working component and its corresponding instruction execution component are deployed on any working node of the container orchestration platform. The working component can connect to the container runtime in the working node. The main control component is configured to obtain a task quota request for a target task, wherein the target task corresponds to at least one target container, the at least one target container is scheduled on one or more work nodes, the work component in the work node where the target container is scheduled is the target work component, and the target container is mounted with a corresponding root file system Rootfs; query quota management data based on the task quota request to determine a quota instruction, wherein the quota management data describes or records quota information related to the task; and send the quota instruction to the target work component; The working component is configured to receive a quota instruction sent by the main control component when the connection with the container runtime is successful, wherein the quota instruction carries a container identifier; if the target container corresponding to the container identifier carried by the quota instruction exists and is in a running state, obtain the writable layer path of the root file system Rootfs mounted by the target container; and call the instruction execution component to execute the quota in the writable layer of the Rootfs of the target container.

2. The system according to claim 1, wherein: The Rootfs quota management system further includes a task controller component and an APIServer component deployed in the master node, and the APIServer component can interact with the master control component and the task controller component respectively; The task controller is used to receive the target task submitted by the user, use the APIServer component to create a target container corresponding to the target task, and schedule the target container to run on any of the working nodes; when the target container is scheduled to run on any of the working nodes, initiate the task limit request to the main control component.

3. The system according to claim 2, characterized in that The quota management data includes a quota policy configuration table and / or a task quota status table; The quota policy configuration table is used to describe the correspondence between task types and default quotas, so that differentiated quotas can be set based on the task types of the target tasks; wherein the task types are divided based on the storage space required by the tasks; The task limit status table is obtained by using the APIServer component to query one or more of the running status, current limit, current usage, and node distribution of the container corresponding to the currently running task of the container orchestration platform, so as to enable dynamic adjustment of the limit of the target task in operation.

4. A Rootfs quota management method, characterized in that: A master control component applied to a Rootfs quota management system, the quota management system further comprising at least one working component and an instruction execution component corresponding to each working component, the working component being capable of interacting with the master control component and the corresponding instruction execution component, respectively, the master control component being deployed on a master node of a container orchestration platform, and any working component and its corresponding instruction execution component being deployed on any working node of the container orchestration platform; the method comprising: Obtaining a task limit request for a target task; wherein the target task corresponds to at least one target container, the at least one target container is scheduled on one or more work nodes, the work component in the work node scheduled with the target container is a target work component, and the target container is mounted with a corresponding root file system Rootfs; querying quota management data based on the task quota request to determine a quota instruction; wherein the quota management data describes or records quota information related to the task; The quota instruction is sent to the target work component, so that the target work component calls the instruction execution component to execute the quota in the Rootfs writable layer of the target container.

5. The method according to claim 4, characterized in that The task quota request is an initialization quota request or a dynamic quota adjustment request, wherein the initialization quota request carries the task type of the target task; the quota management data includes a quota policy configuration table, wherein the quota policy configuration table is used to describe the correspondence between the task type and the default quota; The querying of quota management data based on the task quota request to determine the quota instruction includes: In a case where the task quota request is the initialization quota request, querying the quota policy configuration table based on the task type of the target task to obtain a target quota value; Generate a quota instruction; wherein the quota instruction carries the target quota value.

6. The method according to claim 4, characterized in that The task quota request is an initialization quota request or a dynamic quota adjustment request, wherein the dynamic quota adjustment request carries the task identifier and the requested quota value of the target task; the quota management data includes a task quota status table, which is used to describe the current quota and current usage of the container corresponding to the currently running task; The querying of quota management data based on the task quota request to determine the quota instruction includes: In a case where the task quota request is the dynamic quota adjustment request, querying the task quota status table based on the task identifier of the target task to obtain the current quota and current usage of the target container corresponding to the target task; If the application limit value of the target task, the current limit of the target container, and the current usage meet the set limit adjustment conditions, the application limit value is used as the target limit value; Generate a quota instruction; wherein the quota instruction carries the target quota value.

7. The method according to claim 4, characterized in that The task quota request carries the task identifier of the target task and the corresponding container identifier, where the task identifier is used to uniquely identify the corresponding task, and the container identifier is used to uniquely identify the container of the corresponding task; the quota management data includes a task quota status table, which is used to describe the node distribution of the container corresponding to the currently running task or the initialization task; The querying of quota management data based on the task quota request to determine the quota instruction includes: Querying the task quota status table based on the task identifier of the target task and the corresponding container identifier to determine the task status of the target task and the corresponding target work component; When the task state of the target task is in the running state or the initialization state, the limit instruction is generated; wherein the limit instruction carries the container identifier corresponding to the target task.

8. A Rootfs quota management method, characterized in that: A working component applied to a Rootfs quota management system, the quota management system further comprising a main control component and an instruction execution component corresponding to the working component, the working component being capable of interacting with the main control component and the corresponding instruction execution component respectively, the main control component being deployed on a master node of a container orchestration platform, the working component and its corresponding instruction execution component being deployed on a working node of the container orchestration platform, the working component being capable of connecting to a container runtime in the working node; the method comprising: When the connection with the container runtime is successful, receiving a quota instruction sent by the main control component; wherein the quota instruction carries a container identifier; If the target container corresponding to the container identifier carried by the limit instruction exists and is in a running state, obtain the writable layer path of the root file system Rootfs mounted by the target container; The instruction execution component is called to execute quotas on the RootFS writable layer of the target container.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the Rootfs quota management method according to any one of claims 4 to 8 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed, the Rootfs quota management method according to any one of claims 4 to 8 is implemented.

Images