ConfigMap-based method and system for storing large AI configuration file

Abstract

The invention provides a ConfigMap-based method and a ConfigMap-based system for storing a large AI configuration file, which comprises the following steps of: dividing the content of the large AI configuration file into a plurality of small AI configuration files, respectively creating sub ConfigMap resources by using each small AI configuration file, setting names of the plurality of sub ConfigMap resources in a main ConfigMap resource configuration, and storing the sub ConfigMap resources in the main ConfigMap resource configuration. And combining the data key content of each sub ConfigMap resource into a configuration file by the node proxy service, and storing the configuration file in the workload mounting directory, thereby realizing the purpose of storing a very large AI configuration file by using the ConfigMap resource.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of computer software, and particularly relates to a method and system for saving large AI configuration files based on ConfigMap. BACKGROUND

[0002] ConfigMap is a configuration resource type in Kubernetes, used to store configuration data. It is organized in the form of key-value pairs and can store and manage the configuration information required by AI applications. Since the size of ConfigMap resources is limited to 1MB by default, AI configuration files that exceed the limit cannot be saved through ConfigMap resources, and ConfigMap cannot be used to save large AI configuration files. Therefore, the current method of saving large AI configuration files through ConfigMap has defects and needs to be improved. SUMMARY

[0003] The technical problem to be solved by the application is how to save very large AI configuration files through ConfigMap resources. The application provides a method and system for saving large AI configuration files based on ConfigMap.

[0004] To solve the above technical problems, the technical solution adopted by the application is as follows: A method for saving large AI configuration files based on ConfigMap, comprising the following steps: Step 1: dividing the large AI configuration file into multiple small AI configuration files whose sizes can be accepted by the ConfigMap configuration resource type, wherein the large AI configuration file refers to a file whose size is too large to be accepted by ConfigMap due to the limitation of ConfigMap resources, and the small AI configuration file refers to a file whose size is not limited by ConfigMap resources and can be accepted by ConfigMap; Step 2: creating ConfigMap configuration resources for each small AI configuration file and defining them as sub-ConfigMap configuration resources, and specifying the data key content in each sub-ConfigMap configuration resource as the content of each small AI configuration file; Step 3: creating a ConfigMap configuration resource and defining it as a master ConfigMap configuration resource, and setting the data key content in the master ConfigMap configuration resource to a string formed by combining the names of all sub-ConfigMap configuration resources; Step 4: setting the master ConfigMap configuration resource in the workload resources of the container orchestration software; Step 5: When the container of the workload runs, the node agent service combines the data key contents of each sub-ConfigMap resource according to the content of the data key in the main ConfigMap resource configuration information, forms a large AI configuration file, and saves it to the mounting directory of the workload; Step 6: The AI application in the container of the workload uses the combined configuration file to load the program.

[0005] Further, when the large AI configuration file is divided into small AI configuration files, the names of the small AI configuration files are sequentially named according to the division order, and when the sub-ConfigMap resources are created according to the small AI configuration files, the names of the sub-ConfigMap resources are sequentially named according to the division order of the small AI configuration files.

[0006] Further, when the node agent service combines the data key contents of each sub-ConfigMap resource, the contents of the data key in the sub-ConfigMap resource configuration information are extracted according to the name order of the sub-ConfigMap resources, and are combined in order to form a large AI configuration file.

[0007] Further, when combining, the string data in the data key content of the main ConfigMap resource is parsed into the names of each sub-ConfigMap resource, the data key contents of each sub-ConfigMap resource are obtained through the names of each sub-ConfigMap resource, and the data key contents of each sub-ConfigMap resource are combined into a configuration file according to the name order of each sub-ConfigMap resource and saved to the directory on the host of the mounting directory of the workload.

[0008] Further, in step 3, the data key content in the main ConfigMap resource configuration information is set to a string combined by all sub-ConfigMap resource names, and the specific setting method is as follows: Each sub-ConfigMap resource name is used to create string data, then the string data is combined into a string, and finally the string is set in the data key of the main ConfigMap resource configuration information.

[0009] The application also provides a system for saving a large AI configuration file based on a ConfigMap, and uses a method for saving a large AI configuration file based on a ConfigMap.

[0010] By adopting the above technical solution, the application has the following beneficial effects: The application provides a method and system for saving large AI configuration files based on ConfigMap. BRIEF DESCRIPTION OF DRAWINGS

[0011] Figure 1 The system flowchart of the application. DETAILED DESCRIPTION

[0012] The technical solutions of the application will be described clearly and completely in combination with the drawings. Obviously, the described embodiments are part of the embodiments of the application, not all. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor belong to the protection scope of the application.

[0013] Figure 1 The specific embodiment of the method for saving large AI configuration files based on ConfigMap is shown, which includes the following steps: Step 1: The large AI configuration file is divided into multiple small AI configuration files whose file size can be accepted by the ConfigMap configuration resource type. The large AI configuration file refers to a file whose size is too large to be accepted by ConfigMap due to the limitation of ConfigMap resource. The small AI configuration file refers to a file whose size is not limited by ConfigMap resource and can be accepted by ConfigMap. In this embodiment, by dividing the large configuration file into multiple small configuration files that can be accepted by ConfigMap, the purpose of saving large AI configuration files through ConfigMap when the AI configuration file is very large and exceeds the limit is achieved.

[0014] Step 2: Create ConfigMap configuration resource type for each of the multiple small AI configuration files and define it as sub-ConfigMap configuration resource type. Specify the data key content of each sub-ConfigMap configuration resource type configuration information as the content of each small AI configuration file.

[0015] The ConfigMap configuration resource type in this embodiment is a Kubernetes container orchestration software configuration resource type. The method for creating a ConfigMap in the Kubernetes container orchestration software includes kubectl apply, kubectl create configmap, calling the Kubernetes API, and the like.

[0016] When a large large-AI configuration file is split into small small-AI configuration files, the names of the small-AI configuration files are sequentially named according to the splitting order. When sub-ConfigMap resources are created according to the small-AI configuration files, the names of the sub-ConfigMap resources are sequentially named according to the splitting order of the small-AI configuration files. Sequential naming facilitates finding the corresponding small-AI configuration file when the complete configuration file is combined.

[0017] Step 3: Create another ConfigMap configuration resource type and define it as a master ConfigMap configuration resource type. Set the data key content in the master ConfigMap configuration resource type configuration information to a string that combines the names of all sub-ConfigMap configuration resource types in step 2. In this embodiment, the data key content in the master ConfigMap resource configuration information is set to the names of all sub-ConfigMap resources in step 2. The specific setting method is as follows: Create string data with each sub-ConfigMap resource name, then combine each string data into a string. Finally, set the string in the data key of the master ConfigMap resource configuration information.

[0018] For example, create string data "config1, config2, config3" with the names of each sub-ConfigMap resource config1, config2, and config3. Edit the master ConfigMap resource using the vim command, specify the data key content of the master ConfigMap resource as the string data "config1, config2, config3", specify the name key of the master ConfigMap resource as test, and create the master ConfigMap resource using the kubectl apply command.

[0019] Step 4: Set the master ConfigMap configuration resource type in the workload resource of the container orchestration software.

[0020] In this embodiment, the workloads of the Kubernetes container orchestration software include Deployment, StatefulSet, DaemonSet, Job, CronJob, etc.

[0021] In this embodiment, setting the ConfigMap configuration resource type in the workload resource configuration of the Kubernetes container orchestration software means setting the configuration resource type required by the workload resource in the workload resource configuration information of the container orchestration software, and identifying that the configuration resource type is used by the workload of the container orchestration software.

[0022] In this embodiment, the Deployment workload resource in the container orchestration software is mounted with the ConfigMap configuration resource type named test by means of a data volume, and the mounting directory is set, for example, / etc / config / test. The Deployment workload resource is created using the kubectl apply command provided by the Kubernetes container orchestration software. After the Deployment workload resource is successfully created, the container orchestration software schedules and starts the Pod resource managed by the Deployment workload resource. After the Pod resource is successfully started, the configuration file content in the data key content of the ConfigMap configuration resource type configuration information is mounted to the specified path in the Pod resource, and the ConfigMap resource is used by the Deployment workload resource.

[0023] Step 5: When the container of the workload runs, the node agent service combines the data key content of each sub-ConfigMap resource according to the content of the data key of the main ConfigMap resource configuration information, forms an AI configuration file, and saves it to the mounting directory of the workload.

[0024] In this embodiment, the node agent service is a proxy service software of the container orchestration software running on each host. The node agent service software used in the Kubernetes container orchestration software is kubelet, and this proxy service software can obtain the configuration of the workload. The node agent service software regularly accesses the main ConfigMap configuration information content, and extracts the content of the data key of the main ConfigMap configuration information. Since the content of the data key of the main ConfigMap configuration information is a string "config1, config2, config3" formed by the data key content of each sub-ConfigMap resource.

[0025] When combined, the string data in the data key content of the main ConfigMap resource is parsed into the sub-ConfigMap resource names config1, config2 and config3, the data key content of the sub-ConfigMap resources is obtained through the sub-ConfigMap resource names config1, config2 and config3, and the data key content of the sub-ConfigMap resources is combined into a configuration file according to the name order of the sub-ConfigMap resources and saved to a directory on the host in the workload mounting directory. In this embodiment, by decomposing, each small AI configuration file is set to the data key content of the sub-ConfigMap resource, and then by combining through the node agent service, a large AI configuration file is successfully mounted to the workload directory, thereby realizing saving of a very large AI configuration file using the ConfigMap resource.

[0026] Step 6: The AI application in the container running the workload uses the combined configuration file to load the program.

[0027] The application also provides a system for saving a large AI configuration file based on a ConfigMap, and each step of a method for saving a large AI configuration file based on a ConfigMap.

[0028] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the application, and not to limit them; although the application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the application.

Claims

1. A method for saving large AI configuration files based on ConfigMap, characterized in that, Includes the following steps: Step 1: Divide the large AI configuration file into multiple smaller AI configuration files whose file size can be accepted by the ConfigMap configuration resource type. The large AI configuration file refers to a file whose size is too large and cannot be accepted by ConfigMap due to ConfigMap resource limitations. The small AI configuration file refers to a file whose size is not limited by ConfigMap resource limitations and can be accepted by ConfigMap. Step 2: Create a ConfigMap configuration resource type for each of the multiple small AI configuration files and define it as a sub-ConfigMap configuration resource type. Specify that the data key content in the configuration information of each sub-ConfigMap configuration resource type is the content of each small AI configuration file. Step 3: Create another ConfigMap configuration resource type and define it as the main ConfigMap configuration resource type. Set the data key content in the configuration information of the main ConfigMap configuration resource type to a string formed by merging the names of all child ConfigMap configuration resource types. Step 4: Configure the main ConfigMap resource type in the workload resources of the container orchestration software; Step 5: When the container running the workload is running, the node agent service combines the data key content of each sub-ConfigMap resource according to the content of the data key in the main ConfigMap resource configuration information to form an AI configuration file and save it to the mount directory of the workload. Step 6: The AI ​​application within the container where the workload runs uses the combined configuration file loader.

2. The method of claim 1, wherein, When splitting a large AI configuration file into smaller AI configuration files, the names of the smaller AI configuration files are named sequentially according to the splitting order. When creating sub-ConfigMap resources based on the smaller AI configuration files, the names of the sub-ConfigMap resources are also named sequentially according to the splitting order of the smaller AI configuration files.

3. The method of claim 2, wherein, When combining the data key content of each sub-ConfigMap resource, the node proxy service extracts the data key content from the configuration information of each sub-ConfigMap resource according to the name order of the sub-ConfigMap resources, and combines them in order to form a large AI configuration file.

4. The method of claim 3, wherein, During the combination, the string data in the data key content of the main ConfigMap resource is parsed to obtain the names of each sub-ConfigMap resource. The data key content of each sub-ConfigMap resource is obtained through the names of each sub-ConfigMap resource. The data key content of each sub-ConfigMap resource is combined into a configuration file according to the order of the names of each sub-ConfigMap resource and saved to the directory of the workload mounted directory on the host machine.

5. The method of claim 1, wherein, In step 3, the content of the `data` key in the main ConfigMap resource configuration information is set to a string formed by merging the names of all child ConfigMap configuration resource types. The specific setting method is as follows: Create string data for each sub-ConfigMap resource name, then merge the string data into a single string, and finally set this string in the data key of the main ConfigMap resource configuration information.

6. A system for storing large AI configuration files based on ConfigMap, characterized in that, The steps of the method for saving large AI configuration files based on ConfigMap as described in any one of claims 1 to 5 are used.

Images