A docker swarm cluster resource cleaning optimization method and system

Abstract

The application discloses a Docker Swarm cluster resource cleaning optimization method and system, which comprises the following steps: an image warehouse receives a container resource cleaning image uploaded by a development node and stores the image; a management node acquires a deployment parameter configuration file of a docker resource cleaning service and deploys the docker resource cleaning service; after the deployment is successful, the docker resource cleaning service automatically pulls and runs the stored container resource cleaning image in the image warehouse according to the configuration in the deployment parameter configuration file of the docker resource cleaning service, and creates and runs a cleaning container instance on all nodes in the Docker Swarm cluster. The method realizes the automatic cleaning service task on the changed nodes when the nodes in the Docker Swarm cluster change.

Description

Technical Field

[0001] This invention relates to the field of disk space cleanup and optimization technology, and in particular to a method and system for cleaning up and optimizing Docker Swarm cluster resources. Background Technology

[0002] The statements in this section merely refer to the background art related to this invention and do not necessarily constitute prior art.

[0003] Typically, the Docker engine provides the `prune` command to clean up resources from various local running instances, container images, storage volumes, and networks. It also provides the `docker system prune` command to clean up all resource types. Users can clean up local resources by running the relevant commands on the host machine where the Docker container resides. For example, running `docker image prune` will delete local Docker image files that are not used by container instances. To clean up invalid and unused resources in a Docker Swarm cluster, you need to manually run commands such as `docker image prune` on all nodes in the cluster to remove unused resources on the nodes.

[0004] However, the above method requires users to manually execute relevant commands on each node of the cluster environment; moreover, it requires the use of a third-party scheduling framework such as Linux's corn scheduling framework to execute commands periodically, and the relevant commands need to be manually added on each host node. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a method and system for cleaning up and optimizing Docker Swarm cluster resources, which enables automatic cleaning service tasks to be performed on the changed nodes when nodes in a Docker Swarm cluster change.

[0006] Firstly, this invention provides a method for cleaning up and optimizing Docker Swarm cluster resources;

[0007] A method for cleaning up and optimizing Docker Swarm cluster resources includes the following steps:

[0008] The image repository receives cleanup images of container resources uploaded by development nodes and stores them;

[0009] The management node acquires a deployment parameter configuration file of the docker resource cleaning service and deploys the docker resource cleaning service.

[0010] Further, the specific steps that the image repository receives the container resource cleaning image uploaded by the development node are as follows:

[0011] The development node acquires a container resource cleaning script.

[0012] The development node acquires an image building command and builds the container resource cleaning image based on the container resource cleaning script.

[0013] The development node acquires an image uploading command and uploads the container resource cleaning image to the image repository.

[0014] Further, the container resource cleaning script comprises:

[0015] Reading configuration;

[0016] According to the configuration, the docker engine is called to execute a command for deleting useless images and / or a command for deleting useless container instances and / or a command for deleting useless storage volumes and / or a command for deleting useless networks.

[0017] Further, the specific steps that the management node deploys the docker resource cleaning service are as follows:

[0018] The management node saves the deployment parameter configuration file of the docker resource cleaning service.

[0019] The management node acquires an execution command, and after the docker engine of the management node receives the execution command, the legality of the deployment parameter configuration file of the docker resource cleaning service is verified.

[0020] Further, the docker resource cleaning service also re-runs the cleaning container instance according to the time interval configured in the deployment parameter configuration file of the docker resource cleaning service.

[0021] In a second aspect, the application further provides a Docker Swarm cluster resource cleaning optimization system.

[0022] The Docker Swarm cluster resource cleaning optimization system comprises:

[0023] The mirror warehouse is configured to receive the container resource cleaning image uploaded by the development node and store the container resource cleaning image.

[0024] The management node is configured to obtain a deployment parameter configuration file of the docker resource cleaning service and deploy the docker resource cleaning service, after successful deployment, the docker resource cleaning service automatically pulls and runs the stored container resource cleaning image in the mirror warehouse according to the configuration in the deployment parameter configuration file of the docker resource cleaning service, and creates and runs the cleaning container instance in all nodes in the DockerSwarm cluster.

[0025] Further, the development node is configured to:

[0026] obtain a container resource cleaning script;

[0027] obtain an image building command, and build the container resource cleaning image based on the container resource cleaning script;

[0028] obtain an image uploading command, and upload the container resource cleaning image to the mirror warehouse.

[0029] Further, the container resource cleaning script comprises:

[0030] read a configuration;

[0031] invoke the docker engine to execute a command for deleting useless images and / or a command for deleting useless container instances and / or a command for deleting useless storage volumes and / or a command for deleting useless networks according to the configuration.

[0032] Further, the management node is further configured to:

[0033] save the deployment parameter configuration file of the docker resource cleaning service;

[0034] obtain an execution command, and the docker engine of the management node checks the legality of the deployment parameter configuration file of the docker resource cleaning service after receiving the execution command, and creates the docker resource cleaning service after passing the check, thereby completing the deployment of the docker resource cleaning service.

[0035] Further, the docker resource cleaning service further runs the cleaning container instance according to the time interval configured in the deployment parameter configuration file of the docker resource cleaning service.

[0036] Compared with the prior art, the present application has the following beneficial effects:

[0037] The Docker Swarm cluster resource cleaning optimization method of the application can automatically run the cleaning service task on the changed (increased or deleted) node without manual intervention when the node in the Docker Swarm cluster changes.

[0038] The Docker Swarm cluster resource cleaning optimization method of the application runs the cleaning service in the container mode, and does not need to install additional software on the running work node.

[0039] The Docker Swarm cluster resource cleaning optimization method of the application supports the environment variable mode configuration, and can flexibly customize the interval time of running, the object to be cleaned and the like according to the environment characteristics of the cluster. BRIEF DESCRIPTION OF DRAWINGS

[0040] The drawings constituting a part of the specification of the application are used to provide further understanding of the application, and the illustrative embodiments of the application and the description thereof are used to explain the application, and do not constitute improper limitation on the application.

[0041] Figure 1 It is a Docker Swarm cluster architecture diagram;

[0042] Figure 2 It is a method flowchart of the first embodiment of the application;

[0043] Figure 3 It is a configuration diagram of the container resource cleaning script of the first embodiment of the application. DETAILED DESCRIPTION

[0044] It should be pointed out that the following detailed description is exemplary, and is intended to provide further description of the application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as generally understood by those skilled in the art to which the application belongs.

[0045] It should be noted that the terms used herein are only for the purpose of describing the specific embodiments, and are not intended to limit the exemplary embodiments according to the application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form, and in addition, it should be understood that when the terms "comprise" and / or "include" are used in the specification, they indicate the presence of a feature, step, operation, device, component and / or their combination.

[0046] Term explanation part:

[0047] Swarm is a container cluster management system integrated in docker engine (Docker Engine), and its overall architecture diagram is as shown in Figure 1 Figure 1 ​The Docker Swarm cluster shown is a cluster including 1 manager node and 3 worker nodes, the nodes in the Swarm cluster are divided into two roles:

[0048] Manager node: receives worker node service definition, sends tasks to worker nodes; maintains cluster desired state and cluster management functions and leader election; by default, the manager node also runs tasks, and can also be configured to only do management tasks;

[0049] Worker node: receives and executes tasks allocated from the manager node, and reports the current state of the task, so that the manager node maintains the desired state of each service.

[0050] Embodiment one, as Figure 2 shown, the embodiment provides a Docker Swarm cluster resource cleaning optimization method;

[0051] A Docker Swarm cluster resource cleaning optimization method solves the problem of cleaning useless resources generated after the nodes in the Docker Swarm cluster run for a period of time, and has a significant efficiency improvement compared with the manual cleaning method, and specifically includes the following steps:

[0052] Step 1: Develop a node to build a container resource cleaning image, specifically:

[0053] Step 101, the development node obtains the container resource cleaning script written by the user, that is, the execution shell script for cleaning the system resources occupied by docker, the main function is to read the configuration, including whether to clean the invalid image, whether to clean the invalid network, whether to clean the invalid storage disk volume, and according to the configuration Call the docker engine to execute the delete useless image command and / or delete useless container instance command and / or delete useless storage volume command and / or delete useless network command, specifically including, according to the configuration Call the docker engine to execute the docker rmi command (delete useless image), call the docker rm–f command (delete useless container instance), call the docker network rm command (delete useless network), call the docker volume rm command (delete useless storage volume), delete the expired image, container, network and data volume generated by the Dockers running that are no longer used. The processing process is shown in Figure 3 , that is, the container resource cleaning script is specifically configured as: read the cleaning configuration parameters; clean the useless container according to the configuration parameters; clean the useless network according to the configuration parameters; clean the useless storage volume according to the configuration parameters; clean the useless image according to the configuration parameters; cleaning check, end cleaning.

[0054] Step 102, the development node obtains an image building command, and builds a container resource cleaning image based on a container resource cleaning script. That is, after the completion of the cleaning script, the container resource cleaning image (docker-clear:v1.0 image) is built by using the command docker build-t docker-clear:v1.0.

[0055] Step 2: submit the image to the warehouse, and the image warehouse receives and stores the container resource cleaning image. Specifically, the development node obtains an image uploading command, and uploads the container resource cleaning image to the image warehouse, that is, the built storage container resource cleaning image is uploaded to the image warehouse by using the command docker push docker-clear:v1.0, so as to be downloaded and run by the nodes in the DockerSwarm cluster.

[0056] Step 3: the management node obtains the deployment parameter configuration file of the docker resource cleaning service, and deploys the docker resource cleaning service. After successful deployment, the docker resource cleaning service automatically pulls and runs the storage container resource cleaning image in the image warehouse according to the configuration in the deployment file, and creates and runs the cleaning container instance in all nodes in the Docker Swarm cluster. Specifically:

[0057] Step 301: the management node obtains the deployment parameter configuration file docker-compose.yaml of the docker resource cleaning service created by the user, and deploys the docker resource optimization service (cleaning expired docker resources) by using the docker stack command; that is, the management node obtains and saves the deployment parameter configuration file (including the information such as the port and image used by the resource optimization service) of the docker resource cleaning service, saves the deployment parameter configuration file of the docker resource cleaning service as docker-compose.yml on the management node of the docker swarm cluster, obtains the docker stack execution command, and deploys by using the command docker stack deploy-c docker-compose.yml swrm-clean. After the docker engine of the docker swarm management node receives the execution command, it will first verify the legality of the deployment parameter configuration file of the docker resource cleaning service. After verification, the docker resource cleaning service with the name swrm-clean is created, the deployment is completed, the service uses the docker-clear:v1.0 image according to the description in the deployment file, and is run again every 24 hours.

[0058] Step 302: After successful deployment, the docker resource cleaning service automatically creates and runs a cleaning container instance on all nodes in the cluster according to the configuration. According to the time interval configured by the deployment script, the cleaning container instance is re-run to clean up all invalid resources on all nodes in the cluster.

[0059] The present application uses the Docker swarm global service mode to clean up useless resources on the cluster nodes, recovers the space occupation of resources, and can automatically create or delete the running instance of the resource cleaning service on the changed nodes when the cluster nodes are added or deleted. This can significantly improve the efficiency of resource space occupation recovery, guarantee the space resource usage during service running in the cluster, and reduce the number of manual execution of necessary node resource space detection tasks by operation and maintenance personnel during service operation and maintenance, greatly improving the reliability of service running and the development and operation efficiency of the system, providing technical support for stable operation of distributed systems under micro-service architecture. In addition, since the container running mode is adopted, there is no invasion to all worker nodes in the cluster, and no additional expenditure is increased.

[0060] In embodiment two, a Docker Swarm cluster resource cleaning optimization system is provided.

[0061] A Docker Swarm cluster resource cleaning optimization system comprises:

[0062] An image repository configured to receive a container resource cleaning image uploaded by a development node and store the container resource cleaning image.

[0063] A management node configured to obtain a deployment parameter configuration file of a docker resource cleaning service and deploy the docker resource cleaning service. After successful deployment, the docker resource cleaning service automatically pulls and runs the stored container resource cleaning image in the image repository according to the configuration in the deployment parameter configuration file of the docker resource cleaning service, creates and runs a cleaning container instance on all nodes in the Docker Swarm cluster.

[0064] As an implementation, the development node is configured to:

[0065] Obtain a container resource cleaning script.

[0066] Obtain an image building command, build a container resource cleaning image based on the container resource cleaning script.

[0067] Obtain an image upload command, and upload the container resource cleaning image to the image repository.

[0068] As an implementation, the container resource cleaning script comprises:

[0069] Read the configuration.

[0070] According to the configuration, the docker engine is invoked to execute a command for deleting useless images and / or a command for deleting useless container instances and / or a command for deleting useless storage volumes and / or a command for deleting useless networks.

[0071] As an implementation, the management node is further configured to:

[0072] save a deployment parameter configuration file of the docker resource cleaning service;

[0073] obtain an execution command, after the docker engine of the management node receives the execution command, verify the legality of the deployment parameter configuration file of the docker resource cleaning service, and after the verification is passed, create the docker resource cleaning service to complete the deployment of the docker resource cleaning service.

[0074] As an implementation, the docker resource cleaning service further re-runs the cleaning container instance according to a time interval configured in the deployment parameter configuration file of the docker resource cleaning service.

[0075] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for cleaning up and optimizing Docker Swarm cluster resources, characterized in that, Includes the following steps: The image repository receives cleanup images of container resources uploaded by development nodes and stores them; The management node obtains the deployment parameter configuration file of the Docker resource cleanup service and deploys the Docker resource cleanup service. After successful deployment, the Docker resource cleanup service automatically pulls and runs the storage container resource cleanup image in the image repository according to the configuration in the Docker resource cleanup service deployment parameter configuration file, and creates and runs cleanup container instances on all nodes in the Docker Swarm cluster. The specific steps for the image repository to receive container resources uploaded by development nodes and clean up the images are as follows: Develop the node to obtain container resource cleanup script; The development node obtains the image build command and builds a container resource cleanup image based on the container resource cleanup script. The development node obtains the image upload command, cleans up the container resources image, and uploads it to the image repository; The specific steps for deploying the Docker resource cleanup service on the management node are as follows: The management node stores the deployment parameter configuration file for the Docker resource cleanup service; The management node receives the execution command. After receiving the execution command, the Docker engine of the management node verifies the validity of the deployment parameter configuration file of the Docker resource cleanup service. If the verification is successful, the Docker resource cleanup service is created and the deployment of the Docker resource cleanup service is completed. After successful deployment, the Docker resource cleanup service will automatically create and run cleanup container instances on all nodes of the cluster according to the configuration. It will re-run the cleanup container instances according to the time interval configured in the deployment script to remove invalid and useless resources on all nodes of the cluster. The container resource cleanup script includes: Read configuration; Based on the configuration, the Docker engine is invoked to execute commands to delete useless images and / or delete useless container instances and / or delete useless storage volumes and / or delete useless networks.

2. The Docker Swarm cluster resource cleanup and optimization method as described in claim 1, characterized in that, The Docker resource cleanup service also re-runs the cleanup container instance according to the time interval configured in the Docker resource cleanup service deployment parameter configuration file.

3. A Docker Swarm cluster resource cleanup and optimization system, characterized in that, include: The image repository is configured to receive and store container resource cleanup images uploaded by development nodes. The management node is configured to: obtain the deployment parameter configuration file of the Docker resource cleanup service, deploy the Docker resource cleanup service, and after successful deployment, the Docker resource cleanup service will automatically pull and run the storage container resource cleanup image in the image repository according to the configuration in the deployment parameter configuration file of the Docker resource cleanup service, and create and run cleanup container instances on all nodes in the Docker Swarm cluster; The specific steps for the image repository to receive container resources uploaded by development nodes and clean up the images are as follows: Develop the node to obtain container resource cleanup script; The development node obtains the image build command and builds a container resource cleanup image based on the container resource cleanup script. The development node obtains the image upload command, cleans up the container resources image, and uploads it to the image repository; The specific steps for deploying the Docker resource cleanup service on the management node are as follows: The management node stores the deployment parameter configuration file for the Docker resource cleanup service; The management node receives the execution command. After receiving the execution command, the Docker engine of the management node verifies the validity of the deployment parameter configuration file of the Docker resource cleanup service. If the verification is successful, the Docker resource cleanup service is created and the deployment of the Docker resource cleanup service is completed. After successful deployment, the Docker resource cleanup service will automatically create and run cleanup container instances on all nodes of the cluster according to the configuration. It will re-run the cleanup container instances according to the time interval configured in the deployment script to remove invalid and useless resources on all nodes of the cluster. The container resource cleanup script includes: Read configuration; Based on the configuration, the Docker engine is invoked to execute commands to delete useless images and / or delete useless container instances and / or delete useless storage volumes and / or delete useless networks.

4. The Docker Swarm cluster resource cleanup and optimization system as described in claim 3, characterized in that, The Docker resource cleanup service also re-runs the cleanup container instance according to the time interval configured in the Docker resource cleanup service deployment parameter configuration file.

Images