Method and system for saving large ai configuration files based on configmap

Abstract

The application provides a method and system for saving large AI configuration files based on ConfigMap. The method includes the following steps: dividing the content of a large AI configuration file into multiple small AI configuration files, creating a sub-ConfigMap resource for each small AI configuration file, setting the names of the multiple sub-ConfigMap resources in the main ConfigMap resource configuration, and combining the data key contents of the sub-ConfigMap resources into a configuration file by a node agent service and saving the configuration file to the workload mounting directory.

Description

Technical Field

[0001] This invention belongs to the field of computer software, and in particular relates to a method and system for saving large AI configuration files based on ConfigMap. Background Technology

[0002] ConfigMap is a configuration resource type in Kubernetes used to store configuration data. Organized in key-value pairs, it can store and manage the configuration information required by AI applications. However, since the default size limit for ConfigMap resources is 1MB, it cannot store AI configuration files exceeding this limit, making it unsuitable for situations with large AI configuration files. Therefore, the current method of not being able to store large AI configuration files using ConfigMap has shortcomings and needs improvement. Summary of the Invention

[0003] The technical problem to be solved by this invention is how to save very large AI configuration files through ConfigMap resources. A method and system for saving large AI configuration files based on ConfigMap are proposed.

[0004] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: A method for storing large AI configuration files based on ConfigMap includes the following steps: Step 1: Divide the large AI configuration file into multiple smaller AI configuration files whose file size can be accepted by the ConfigMap configuration resource type. The large AI configuration file refers to a file whose size is too large and cannot be accepted by ConfigMap due to ConfigMap resource limitations. The small AI configuration file refers to a file whose size is not limited by ConfigMap resource limitations and can be accepted by ConfigMap. Step 2: Create a ConfigMap configuration resource type for each of the multiple small AI configuration files and define it as a sub-ConfigMap configuration resource type. Specify that the data key content in the configuration information of each sub-ConfigMap configuration resource type is the content of each small AI configuration file. Step 3: Create another ConfigMap configuration resource type and define it as the main ConfigMap configuration resource type. Set the data key content in the configuration information of the main ConfigMap configuration resource type to a string formed by merging the names of all child ConfigMap configuration resource types. Step 4: Configure the main ConfigMap resource type in the workload resources of the container orchestration software; Step 5: When the container running the workload is running, the node agent service combines the data key content of each sub-ConfigMap resource according to the content of the data key in the main ConfigMap resource configuration information to form a large AI configuration file and save it to the mount directory of the workload. Step 6: The AI ​​application within the container where the workload runs uses the combined configuration file loader.

[0005] Furthermore, when dividing a large AI configuration file into smaller AI configuration files, the names of the smaller AI configuration files are named sequentially according to the order of division. When creating sub-ConfigMap resources based on each smaller AI configuration file, the names of the sub-ConfigMap resources are also named sequentially according to the order of division of each smaller AI configuration file.

[0006] Furthermore, when combining the data key content of each sub-ConfigMap resource, the node proxy service extracts the data key content from the configuration information of each sub-ConfigMap resource according to the name order of the sub-ConfigMap resources, and combines them in order to form a large AI configuration file.

[0007] Furthermore, during the combination, the string data in the data key content of the main ConfigMap resource is parsed to obtain the names of each sub-ConfigMap resource. The data key content of each sub-ConfigMap resource is obtained through the names of each sub-ConfigMap resource. The data key content of each sub-ConfigMap resource is combined into a configuration file according to the order of the names of each sub-ConfigMap resource and saved to the directory of the workload mount directory on the host machine.

[0008] Furthermore, in step 3, the content of the `data` key in the main ConfigMap resource configuration information is set to a string formed by merging the names of all child ConfigMap resources. The specific setting method is as follows: Create string data for each sub-ConfigMap resource name, then merge the string data into a single string, and finally set this string in the data key of the main ConfigMap resource configuration information.

[0009] This invention also provides a system for storing large AI configuration files based on ConfigMap, and a method for storing large AI configuration files based on ConfigMap, including the following steps.

[0010] By adopting the above technical solution, the present invention has the following beneficial effects: This invention provides a method and system for storing large AI configuration files based on ConfigMap. The method involves splitting the content of a large AI configuration file into multiple smaller AI configuration files, creating sub-ConfigMap resources for each smaller AI configuration file, setting the names of the multiple sub-ConfigMap resources in the main ConfigMap resource configuration, and having a node proxy service combine the data key content of each sub-ConfigMap resource into a single configuration file and save it to the workload mount directory. This enables the storage of very large AI configuration files using ConfigMap resources. Attached Figure Description

[0011] Figure 1 This is a system flowchart of the present invention. Detailed Implementation

[0012] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0013] Figure 1 This invention illustrates a specific embodiment of a method for saving large AI configuration files based on a ConfigMap, comprising the following steps: Step 1: Divide the large AI configuration file into multiple smaller AI configuration files whose file size is acceptable to the ConfigMap resource type. The large AI configuration file refers to a file whose size is too large to be accepted by ConfigMap due to resource limitations. The small AI configuration file refers to a file whose size is not limited by ConfigMap resources and is acceptable to ConfigMap. In this embodiment, by dividing the large configuration file into multiple smaller configuration files whose size is acceptable to ConfigMap, the purpose of storing the large AI configuration file through ConfigMap is achieved when the AI ​​configuration file becomes very large and exceeds the limit.

[0014] Step 2: Create a ConfigMap configuration resource type for each of the multiple small AI configuration files and define it as a sub-ConfigMap configuration resource type. Specify that the data key content in the configuration information of each sub-ConfigMap configuration resource type is the content of each small AI configuration file.

[0015] In this embodiment, the ConfigMap configuration resource type is the Kubernetes container orchestration software configuration resource type. The methods for creating a ConfigMap in the Kubernetes container orchestration software include: kubectl apply, kubectl create configmap, calling the Kubernetes API, etc.

[0016] When dividing a large AI configuration file into smaller AI configuration files, the names of the smaller AI configuration files are sequentially named according to the order of division. Similarly, when creating sub-ConfigMap resources based on each smaller AI configuration file, the names of the sub-ConfigMap resources are also sequentially named according to the order of division of the smaller AI configuration files. This sequential naming makes it easy to find the corresponding smaller AI configuration file when assembling it into a complete configuration file.

[0017] Step 3: Create another ConfigMap configuration resource type and define it as the main ConfigMap configuration resource type. Set the data key content in the configuration information of the main ConfigMap configuration resource type to the string formed by merging the names of all the child ConfigMap configuration resource types in Step 2. In this embodiment, the content of the data key in the main ConfigMap resource configuration information is set to the names of all child ConfigMap resources in step 2. The specific setting method is as follows: Create string data for each sub-ConfigMap resource name; then merge these string data into a single string. Finally, set this string in the data key of the main ConfigMap resource configuration information.

[0018] For example, create the string data "config1, config2, config3" for each of the sub-ConfigMap resource names config1, config2, and config3. Use the vim command to edit the main ConfigMap resource, specify the data key content of the main ConfigMap resource as the string data "config1, config2, config3", specify the name key of the main ConfigMap resource as test, and use the kubectl apply command to create the main ConfigMap resource.

[0019] Step 4: Set the main ConfigMap configuration resource type in the workload resources of the container orchestration software.

[0020] In this embodiment, the workload of the Kubernetes container orchestration software includes: Deployment, StatefulSet, DaemonSet, Job, CronJob, etc.

[0021] In this embodiment, setting the ConfigMap configuration resource type in the workload resource configuration of the Kubernetes container orchestration software means setting the configuration resource type required by the workload resource in the workload resource configuration information of the container orchestration software, and identifying that the configuration resource type is used by the workload of the container orchestration software.

[0022] In this embodiment, a ConfigMap configuration resource type named "test" is mounted to the Deployment workload resource in the container orchestration software via a data volume, with the mount directory set, for example: / etc / config / test. The Deployment workload resource is created using the kubectl apply command provided by the Kubernetes container orchestration software. After successful creation, the container orchestration software schedules and starts the Pod resource managed by the Deployment workload resource. Once the Pod resource starts successfully, the configuration file content in the data key of the ConfigMap configuration resource type is mounted to the specified path within the Pod resource, and the ConfigMap resource is used by the Deployment workload resource.

[0023] Step 5: When the container running the workload is running, the node agent service combines the data key content of each sub-ConfigMap resource according to the data key content in the main ConfigMap resource configuration information to form an AI configuration file and save it to the mount directory of the workload.

[0024] In this embodiment, the node proxy service is the proxy service software that runs on each host of the container orchestration software. In the Kubernetes container orchestration software, the node proxy service software used is kubelet, which can obtain the workload configuration. The node proxy service software periodically accesses the main ConfigMap configuration information and extracts the content of the data key in the main ConfigMap configuration information. Since the content of the data key in the main ConfigMap configuration information is a string "config1, config2, config3" formed by the data key content of each sub-ConfigMap resource.

[0025] During the combination process, the string data in the data key content of the main ConfigMap resource is parsed to obtain the names of each sub-ConfigMap resource, config1, config2, and config3. The data key content of each sub-ConfigMap resource is then obtained using these names. Based on the order of their names, the data key contents of these sub-ConfigMap resources are combined into a single configuration file and saved to the directory on the host machine where the workload is mounted. This embodiment decomposes the configuration files, setting each small AI configuration file separately in the data key content of a sub-ConfigMap resource. Then, through a node proxy service, these files are merged to successfully mount a large AI configuration file to the workload directory, enabling the storage of very large AI configuration files using ConfigMap resources.

[0026] Step 6: The AI ​​application within the container where the workload runs uses the combined configuration file loader.

[0027] This invention also provides a system for storing large AI configuration files based on ConfigMap, and a method for storing large AI configuration files based on ConfigMap, including the following steps.

[0028] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for storing large AI configuration files based on ConfigMap, characterized in that, Includes the following steps: Step 1: Divide the large AI configuration file into multiple smaller AI configuration files whose file size can be accepted by the ConfigMap configuration resource type. The large AI configuration file refers to a file whose size is too large and cannot be accepted by ConfigMap due to ConfigMap resource limitations. The small AI configuration file refers to a file whose size is not limited by ConfigMap resource limitations and can be accepted by ConfigMap. Step 2: Create a ConfigMap configuration resource type for each of the multiple small AI configuration files and define it as a sub-ConfigMap configuration resource type. Specify that the data key content in the configuration information of each sub-ConfigMap configuration resource type is the content of each small AI configuration file. Step 3: Create another ConfigMap configuration resource type and define it as the main ConfigMap configuration resource type. Set the data key content in the configuration information of the main ConfigMap configuration resource type to a string formed by merging the names of all child ConfigMap configuration resource types. Step 4: Configure the main ConfigMap resource type in the workload resources of the container orchestration software; Step 5: When the container running the workload is running, the node agent service combines the data key content of each sub-ConfigMap resource according to the content of the data key in the main ConfigMap resource configuration information to form an AI configuration file and save it to the mount directory of the workload. When combining the data key content of each sub-ConfigMap resource, the node proxy service extracts the data key content from the configuration information of each sub-ConfigMap resource according to the name order of the sub-ConfigMap resources, and combines them in order to form a large AI configuration file. During the combination, the string data in the data key content of the main ConfigMap resource is parsed to obtain the names of each sub-ConfigMap resource. The data key content of each sub-ConfigMap resource is obtained through the names of each sub-ConfigMap resource. The data key content of each sub-ConfigMap resource is combined into a configuration file according to the order of the names of each sub-ConfigMap resource and saved to the directory of the workload mounted directory on the host machine. Step 6: The AI ​​application within the container where the workload runs uses the combined configuration file loader.

2. The method according to claim 1, characterized in that, When splitting a large AI configuration file into smaller AI configuration files, the names of the smaller AI configuration files are named sequentially according to the splitting order. When creating sub-ConfigMap resources based on the smaller AI configuration files, the names of the sub-ConfigMap resources are also named sequentially according to the splitting order of the smaller AI configuration files.

3. The method according to claim 2, characterized in that, In step 3, the content of the `data` key in the main ConfigMap resource configuration information is set to a string formed by merging the names of all child ConfigMap configuration resource types. The specific setting method is as follows: Create string data for each sub-ConfigMap resource name, then merge the string data into a single string, and finally set this string in the data key of the main ConfigMap resource configuration information.

4. A system for storing large AI configuration files based on ConfigMap, characterized in that, The steps of the method for saving large AI configuration files based on ConfigMap as described in any one of claims 1 to 3 are used.

Images