Log collection method, device, electronic device and storage medium based on Kubernetes cluster

Abstract

The present application relates to the field of data processing technology, and the present application provides a log collection method, device, electronic device and storage medium based on a Kubernetes cluster. The method comprises: obtaining cluster-level log configuration information from the master node of the Kubernetes cluster through an agent application in an independently constructed information collection container to generate and execute a running configuration file, and then dynamically monitoring the running events of the target container in the working node to generate a corresponding log collection configuration file, the log collection configuration file including the container name, the target container and the association relationship between the log information, and then collecting the target log information from the memory according to the log collection configuration file and sending it to the log service cluster. In the process of collecting the target log information, the present application takes into account the association relationship between the target log information and the container name and the target container, overcomes the problem that Filebeat in the related art cannot know the relationship between the container log and the container, and improves the collection accuracy of the container log.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a log collection method, device, electronic device, and storage medium based on a Kubernetes cluster. Background Art

[0002] Log collection and analysis are crucial for Kubernetes cluster systems. Many logging systems in the industry choose to deploy Filebeat on nodes to collect container logs in Kubernetes clusters. However, Filebeat cannot understand the relationship between container log files and containers, making it difficult for users to query logs for specific containers. Furthermore, since container logs are often needed to troubleshoot issues, the accuracy and importance of container logs are self-evident. Therefore, finding a more efficient and accurate solution to container log collection in Kubernetes clusters has become a pressing issue. Summary of the Invention

[0003] The main purpose of the embodiments of the present application is to propose a log collection method, device, electronic device and storage medium based on a Kubernetes cluster, aiming to improve the accuracy of collecting container logs in a Kubernetes cluster.

[0004] To achieve the above objectives, a first aspect of an embodiment of the present application proposes a log collection method based on a Kubernetes cluster, which is applied to a working node in the Kubernetes cluster. The log collection method includes:

[0005] Building an independent information collection container, wherein the information collection container includes an agent application for collecting log information;

[0006] Starting the proxy application in the information collection container, obtaining cluster-level log configuration information from the master node of the Kubernetes cluster through the proxy application, generating a running configuration file for executing the log collection function based on the log configuration information, and controlling the proxy application to execute the running configuration file;

[0007] Dynamically monitoring the running events of the target container in the working node through the proxy application;

[0008] When the running event is monitored, a log collection configuration file corresponding to the target container is generated, wherein the log collection configuration file includes a container name, an association relationship between the target container and log information generated by the target container;

[0009] Scanning the target file directory of the working node through the proxy application, and loading the target log information generated by the target container in the target file directory into the memory;

[0010] The target log information is collected from the memory according to the log collection configuration file, and the target log information is sent to the log service cluster.

[0011] According to the log collection method provided in some embodiments of the present application, after building the independent information collection container, the log collection method further includes:

[0012] Setting the memory capacity of the information collection container to be less than or equal to a preset capacity, and setting the number of cores of the central processing unit of the information collection container to be less than or equal to a preset number;

[0013] The root path of the working node is mapped to the target path inside the information collection container.

[0014] According to the log collection method provided in some embodiments of the present application, obtaining cluster-level log configuration information from the master node of the Kubernetes cluster through the proxy application includes:

[0015] Accessing the interface server on the master node of the Kubernetes cluster using the Hypertext Transfer Protocol through the proxy application to request cluster-level log configuration information;

[0016] The cluster-level log configuration information sent by the interface server is received through the proxy application.

[0017] According to the log collection method provided in some embodiments of the present application, dynamically monitoring the running events of the target container in the working node through the proxy application includes:

[0018] Dynamically monitor the running events of the target container in the Docker daemon process in the working node through the proxy application;

[0019] The operation event includes any one of a start event, a stop event or a restart event.

[0020] According to the log collection method provided in some embodiments of the present application, after generating the log collection configuration file corresponding to the target container, the log collection method further includes:

[0021] When an update event of the target container is monitored, updating the log collection configuration file corresponding to the target container according to the update event;

[0022] or,

[0023] When a deletion event of the target container is monitored, the log collection configuration file corresponding to the target container is deleted according to the deletion event.

[0024] According to the log collection method provided in some embodiments of the present application, scanning the target file directory of the working node by the proxy application and loading the target log information generated by the target container in the target file directory into the memory includes:

[0025] Scanning the target file directory of the working node at preset intervals by the proxy application;

[0026] When target log information generated by the target container is scanned in the target file directory, the target log information in the target file directory is dynamically loaded into the memory.

[0027] According to the log collection method provided in some embodiments of the present application, collecting the target log information from the memory according to the log collection configuration file and sending the target log information to the log service cluster includes:

[0028] Collecting the target log information from the memory according to the log collection configuration file;

[0029] Performing structured processing on the target log information according to the target container, the namespace corresponding to the target container, the container name corresponding to the target container, and the Kubernetes cluster corresponding to the target container;

[0030] The structured target log information is sent to the log service cluster.

[0031] To achieve the above objectives, a second aspect of an embodiment of the present application provides a log collection device based on a Kubernetes cluster, the log collection device comprising:

[0032] A container construction module, configured to construct an independent information collection container, wherein the information collection container includes an agent application for collecting log information;

[0033] A container operation module is used to start the proxy application in the information collection container, obtain cluster-level log configuration information from the master node of the Kubernetes cluster through the proxy application, generate a running configuration file for executing the log collection function based on the log configuration information, and control the proxy application to execute the running configuration file;

[0034] An event monitoring module, configured to dynamically monitor the running events of the target container in the working node through the proxy application;

[0035] a file generation module, configured to generate a log collection configuration file corresponding to the target container upon monitoring the operation event, wherein the log collection configuration file includes a container name, an association relationship between the target container and log information generated by the target container;

[0036] An information loading module is used to scan the target file directory of the working node through the proxy application, and load the target log information generated by the target container in the target file directory into the memory;

[0037] The information sending module is used to collect the target log information from the memory according to the log collection configuration file, and send the target log information to the log service cluster.

[0038] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory, a processor, and a program stored on the memory and runnable on the processor. When the program is executed by the processor, the log collection method described in the first aspect above is implemented.

[0039] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a storage medium, which is a computer-readable storage medium used for computer-readable storage. The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the log collection method described in the first aspect above.

[0040] The present application proposes a log collection method, device, electronic device and storage medium based on Kubernetes cluster, obtains cluster-level log configuration information from the master node of Kubernetes cluster through an agent application in an independently constructed information collection container to generate and execute a running configuration file, and then dynamically monitors the running events of the target container in the working node to generate a log collection configuration file corresponding to the target container, wherein the log collection configuration file includes the container name, the target container and the association relationship between the log information generated by the target container, and then collects the target log information from the memory according to the log collection configuration file, and sends the target log information to the log service cluster. The target log information sent to the log service cluster obtained by the above method is associated with the container name and the target container, overcomes the problem that Filebeat in the related art cannot know the relationship between the container log and the container, and improves the collection accuracy of the container log in the Kubernetes cluster. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1This is a schematic diagram of an implementation environment for executing a log collection method based on a Kubernetes cluster provided in an embodiment of the present application;

[0042] Figure 2 This is a flow chart of a log collection method based on a Kubernetes cluster provided in an embodiment of the present application;

[0043] Figure 3 This is a flowchart of a log collection method based on a Kubernetes cluster provided by another embodiment of the present application;

[0044] Figure 4 yes Figure 2 A specific method flow chart of step S120;

[0045] Figure 5 yes Figure 2 A specific method flow chart of step S150;

[0046] Figure 6 yes Figure 2 A specific method flow chart of step S160;

[0047] Figure 7 This is a schematic diagram of the structure of a log collection device based on a Kubernetes cluster provided in an embodiment of the present application;

[0048] Figure 8 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0050] It should be noted that, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0051] First, let’s analyze some of the terms used in this application:

[0052] Kubernetes cluster: Kubernetes, also known as K8s, is a containerized application that can be used to manage multiple hosts in a cloud platform. Kubernetes provides a mechanism for application deployment, planning, updating, and maintenance. Application deployment in Kubernetes is achieved by deploying containers. Each container is isolated from each other, each container has its own file system, and processes between containers do not affect each other, and computing resources can be differentiated. Compared to virtual machines, containers can be deployed quickly. Because containers are decoupled from the underlying infrastructure and machine file systems, they can be migrated between different clouds and different versions of operating systems. In a Kubernetes cluster environment, deployed services run on nodes in the form of containers (Pods), where each Pod contains at least one service container.

[0053] Filebeat: is a lightweight delivery tool for forwarding and centralizing log data. Filebeat monitors the specified log files or locations, collects log events, and forwards them to Elasticsearch or Logstash for indexing. Filebeat works as follows: When you start Filebeat, Filebeat starts one or more inputs that look for log data in the specified locations. For each log found by Filebeat, Filebeat starts a collector. Each collector reads a single log to get new content and sends the new log data to Libbeat. Libbeat aggregates the log data and sends the aggregated log data to the output configured for Filebeat.

[0054] Elasticsearch: is a distributed, highly scalable, and highly real-time search and data analysis engine. Elasticsearch can be used to search for various documents. It provides scalable search, has near real-time search capabilities, and supports multi-tenancy. Elasticsearch is distributed, which means that the index can be divided into shards, and each shard can have zero or more replicas. Each node hosts one or more shards and acts as a coordinator to delegate operations to the correct shard. Related data is usually stored in the same index, which consists of one or more primary shards and zero or more replica shards. Once an index is created, the number of primary shards cannot be changed. The implementation principle of Elasticsearch is mainly divided into the following steps: first, the user submits the data to the Elasticsearch database, and then the corresponding sentence is segmented through the segmentation controller, and its weight and segmentation results are stored in the data. When the user searches for data, the results are ranked and scored according to the weight, and then the returned results are presented to the user.

[0055] Docker is an application container engine, not a container itself. Docker is a tool for creating containers. The overall Docker architecture consists of three main components: the client, the Docker daemon, and the registry. The Docker daemon is a key link, connecting the registry and the client. The Docker daemon receives and executes commands from the Docker client. It also manages Docker objects such as images, containers, volumes, and networks. The Docker daemon communicates with other Docker daemons to manage container services. The Docker client provides a set of user interfaces (either command-line or SDK) that allow users to send commands to Docker and receive results. A Docker client can connect to multiple Docker hosts. The Docker client can be installed on the same host as the Docker host or on different hosts. The Docker registry provides the ability to store images. Using commands such as docker pull or docker search actually queries an image repository.

[0056] Currently, many log systems in the industry choose to further deploy Filebeat in deployed containers on the nodes of the Kubernetes cluster to collect container logs in the Kubernetes cluster. However, this will cause some problems: First, this way of deploying Filebeat will result in Filebeat's occupation of host machine resources such as CPU and memory being unable to be effectively restricted, which will affect the safe use of the Kubernetes cluster; second, Filebeat cannot know the relationship between container log files and containers, which is not convenient for subsequent users to query logs for specific containers. However, since container logs are often needed to locate problems, if the obtained container log files cannot be associated with the container, it will affect the convenience and accuracy of locating the problem, making it impossible to quickly locate the problem.

[0057] In order to improve the collection accuracy of container logs in Kubernetes cluster so that problems can be quickly located later, the application proposes a log collection method, device, electronic device and storage medium based on Kubernetes cluster, obtain cluster-level log configuration information from the master node of Kubernetes cluster by the agent application in the information collection container of independent construction to generate and execute operation configuration file, then dynamically monitor the operation events of the target container in the working node to generate the log collection configuration file corresponding to the target container, wherein the log collection configuration file includes the association relationship between the container name, the target container and the log information generated by the target container, then collect the target log information from the memory according to the log collection configuration file, and send the target log information to the log service cluster. The target log information sent to the log service cluster obtained by the above method is associated with the container name and the target container, overcomes the problem that Filebeat cannot know the relationship between the container log and the container in the related art, thereby improving the collection accuracy of the container log in Kubernetes cluster, making it possible to quickly locate problems later.

[0058] Figure 1 This is a schematic diagram of an implementation environment for executing a log collection method based on a Kubernetes cluster provided by an embodiment of the present application. Figure 1 The Kubernetes cluster includes a master node 110 and a worker node 120, and the number of the worker nodes 120 can be multiple. The master node 110 is provided with an interface server (Kube-apiserver) 111, and the Kubernetes cluster can be operated through the Kube-apiserver 111 in the master node 110. In addition, the master node 110 also includes a controller 112, and various configurations for the worker node 120 can be implemented through the controller 112. The worker node 120 includes multiple Pods 121, which are the smallest deployable units of the Kubernetes cluster. A Pod 121 represents a workload running in the cluster, and can include one or more Docker containers, mount the required storage, and have a unique IP address. Multiple containers in Pod 121 will always run on the same node.

[0059] It should be noted that the master node 110 and the working node 120 can be, for example, physical machines in a data center, or virtual machines on a cloud provider, etc. This embodiment does not specifically limit this.

[0060] Figure 2 This is a flow chart of a log collection method based on Kubernetes cluster provided by an embodiment of the present application. Figure 2As shown, the log collection method based on the Kubernetes cluster includes but is not limited to the following steps S110 to S160. It should be noted that the method can be executed by a working node in the Kubernetes cluster, for example, Figure 1 The worker nodes 120 in the illustrated embodiment execute.

[0061] S110: Construct an independent information collection container, wherein the information collection container includes an agent application for collecting log information.

[0062] It is understandable that when executing a log collection method based on a Kubernetes cluster, one can first leverage the unique resource object DaemonSet in Kubernetes to build an independent information collection container on each worker node, and this information collection container includes an agent application for collecting log information. Because this information collection container is independently built and isolated from other containers in the worker node, independent CPU and memory resources can be set for it. Therefore, the CPU and memory resources of the information collection container can be easily limited, thereby reducing the impact on the safe use of the Kubernetes cluster.

[0063] It should be noted that an independent information collection container can be built separately on each working node of the Kubernetes cluster through kubectl, and the information collection container independently built on each working node includes an agent application for collecting log information.

[0064] It should be noted that kubectl is a command-line tool for operating Kubernetes clusters. It can be installed on the master node of Kubernetes. Through kubectl, operations such as adding, deleting, modifying, and querying various resources in the Kubernetes cluster can be implemented.

[0065] S120: Start the proxy application in the information collection container, obtain cluster-level log configuration information from the master node of the Kubernetes cluster through the proxy application, generate a running configuration file for executing the log collection function according to the log configuration information, and control the proxy application to execute the running configuration file.

[0066] In this step, since an independent information collection container is constructed in step S110, and the information collection container includes a proxy application for collecting log information, the proxy application in the information collection container can be started, and the cluster-level log configuration information can be obtained from the master node of the Kubernetes cluster through the proxy application. A running configuration file for executing the log collection function is generated according to the log configuration information, and the proxy application is controlled to execute the running configuration file, so that the log information of the working node can be collected according to the running configuration file in subsequent steps.

[0067] It should be noted that in some feasible implementations, after starting the proxy application in the information collection container, the cluster-level log configuration information (i.e., logconfig configuration) can be obtained from the master node of the Kubernetes cluster through the proxy application, and then a running configuration file (i.e., filebeat.yml file) for executing the log collection function is generated according to the logconfig configuration. The proxy application is then controlled to execute the filebeat.yml file to run the log information collection function of Filebeat.

[0068] In this step, because the log configuration information that the proxy application obtains from the master node of Kubernetes cluster is the configuration information of cluster level, so the running configuration file for executing the log collection function generated according to the log configuration information in the subsequent step is associated with the entire Kubernetes cluster, so after executing the running configuration file, the proxy application in each working node can collect log information based on the angle of cluster level when executing the log information collection function of Filebeat, thereby adapting to the entire Kubernetes cluster. In addition, because the running configuration file is after starting the proxy application, the proxy application automatically obtains the log configuration information of cluster level from the master node of Kubernetes cluster and generates according to the log configuration information, that is to say, the running configuration file for executing the log collection function is automatically generated, does not need the user to manually write, thereby can improve processing efficiency; And, because the running configuration file is automatically generated according to the log configuration information of cluster level, therefore can adapt to the Kubernetes cluster, save the compilation and debugging process that needs to be adapted to execute due to manually writing configuration files, and then can further improve processing efficiency, increase the use value of container.

[0069] S130: Dynamically monitor the running events of the target container in the working node through the proxy application.

[0070] It should be noted that, since the control agent application executes the running configuration file in step S120, the agent application can perform the log collection function (for example, execute the log collection function of Filebeat), but this only implements the conventional log collection function of Filebeat in the related technology, and there is still no way to know the relationship between the container log file and the container. In order to solve this problem, in this step, the running events of the target container in the working node are first dynamically monitored by the agent application, so that the subsequent steps can generate a log collection configuration file including the association relationship between the container log file and the container according to the running events of the target container, thereby solving the problem that Filebeat in the related technology cannot know the relationship between the container log and the container.

[0071] It should be noted that the operation events of the target container may include, for example, a start event, a stop event, or a restart event, and may be appropriately selected according to actual application conditions, and are not specifically limited here.

[0072] S140: When a running event is monitored, a log collection configuration file corresponding to the target container is generated, wherein the log collection configuration file includes an association relationship between the container name, the target container, and the log information generated by the target container.

[0073] In this step, since the running events of the target container in the working node are dynamically monitored by the proxy application in step S130, when the running event is monitored, a log collection configuration file corresponding to the target container can be generated, wherein the log collection configuration file includes the container name, the target container and the association relationship between the log information generated by the target container. Therefore, when the subsequent steps obtain the log information generated by the target container according to the log collection configuration file, the association relationship between the log information and the container name and the target container can be obtained, thereby solving the problem in the related technology that Filebeat cannot know the relationship between the container log and the container, and achieving the purpose of more efficiently and accurately saving the log collection of the container in the Kubernetes cluster.

[0074] S150: Scan the target file directory of the working node through the proxy application, and load the target log information generated by the target container in the target file directory into the memory.

[0075] It should be noted that after the target container in the working node is started, as the target container runs, the working node will save the log information generated by the target container during the operation to the corresponding file directory (for example, / var / log / container / directory). Based on this, in this step, since the startup time of the information collection container is earlier than the startup time of the target container, and the control agent application executes the running configuration file to run the log collection function in step S120, the target file directory of the working node can be scanned by the agent application, and the target log information generated by the target container in the target file directory can be loaded into the memory, so that the subsequent steps can obtain the target log information associated with the container name and the target container from the memory, providing an accurate data basis for subsequent users to query the logs of specific containers.

[0076] S160: Collect target log information from the memory according to the log collection configuration file, and send the target log information to the log service cluster.

[0077] In this step, since a log collection configuration file including the association relationship between the container name, the target container, and the log information generated by the target container is generated in step S140, and the target log information generated by the target container in the target file directory is loaded into the memory in step S150, the target log information can be collected from the memory according to the log collection configuration file, and the target log information is sent to the log service cluster, so that the log service cluster saves the target log information, so that the user can perform accurate log queries for a specific container through the log service cluster.

[0078] In some feasible implementations, the log service cluster may be an Elasticsearch cluster, which may be appropriately selected based on actual application conditions and is not specifically limited herein.

[0079] In the present embodiment, by including the log collection method based on Kubernetes clusters of preceding step S110 to step S160, obtain the log configuration information of cluster level from the master node of Kubernetes cluster by the agent application in the information collection container of independent construction to generate and execute running configuration file, then dynamically monitor the running event of the target container in the working node to generate the log collection configuration file corresponding to the target container, wherein, the log collection configuration file includes the association relationship between the log information generated by container name, target container and target container, then collect target log information from internal memory according to the log collection configuration file, and target log information is sent to log service cluster.The target log information of the log service cluster obtained by said method is associated with container name and target container, overcomes the problem that Filebeat in the related art cannot know the relationship between container log and container, improves the collection accuracy of container log in Kubernetes cluster.

[0080] Reference Figure 3 As shown, an embodiment of the present application further illustrates the log collection method. After executing step S110, the log collection method may further include but is not limited to step S111 and step S112.

[0081] Step S111: setting the memory capacity of the information collection container to be less than or equal to a preset capacity, and setting the number of cores of the central processing unit of the information collection container to be less than or equal to a preset number.

[0082] In this step, since an independent information collection container is constructed in step S110, independent CPU resources and memory resources can be set for the information collection container. Therefore, the memory capacity of the information collection container can be set to be less than or equal to the preset capacity, and the number of cores of the central processing unit of the information collection container can be set to be less than or equal to the preset number, so that the information collection container will not occupy too much CPU resources and memory resources in the working node, thereby reducing the impact on the safe use of the Kubernetes cluster.

[0083] It should be noted that the preset capacity can be appropriately selected according to actual application conditions and is not specifically limited here. For example, the preset capacity can be set to 200M.

[0084] It should be noted that the preset number can be appropriately selected according to actual application conditions and is not specifically limited here. For example, the preset number can be set to 1.

[0085] Step S112: Map the root path of the working node to the target path inside the information collection container.

[0086] It can be understood that by mapping the root path of the working node to the target path inside the information collection container, when the information collection container is started, the file directory corresponding to the root path in the working node can be operated, so that the log information in the file directory can be scanned and collected in subsequent steps.

[0087] In a feasible implementation, the root path of the working node can be mapped to the target path / host inside the information collection container, so that when the information collection container is started, the file directory / var / log / in the working node can be operated to obtain the log information saved in the file directory / var / log / .

[0088] Reference Figure 4 As shown, an embodiment of the present application further explains the "obtaining cluster-level log configuration information from the master node of the Kubernetes cluster through the proxy application" in step S120. The "obtaining cluster-level log configuration information from the master node of the Kubernetes cluster through the proxy application" in step S120 may include but is not limited to steps S121 and S122.

[0089] Step S121: Access the interface server on the master node of the Kubernetes cluster using the Hypertext Transfer Protocol through the proxy application to request cluster-level log configuration information.

[0090] In this step, after starting the proxy application in the information collection container, the proxy application can use the Hypertext Transfer Protocol to access the interface server (Kube-apiserver) on the master node of the Kubernetes cluster, thereby requesting cluster-level log configuration information, so that in subsequent steps, a running configuration file for executing the log collection function can be generated based on the log configuration information and the proxy application can be controlled to execute the running configuration file, thereby achieving the purpose of collecting log information of the working node.

[0091] It should be noted that the Kube-apiserver is equivalent to an entry point to the Kubernetes cluster. Whether controlling the Kubernetes cluster through kubectl or using the remote API, it must go through the Kube-apiserver to listen and control the designated port. The Kube-apiserver is an HTTP / RESTful server that performs the addition, deletion, query, modification, and monitoring of all objects in the Kubernetes system. "Watching" refers to watch operations (such as listening operations). It should be noted that the Kube-apiserver itself is stateless and provides a series of functions such as data access authentication, caching, and API version adaptation and conversion.

[0092] Step S122: Receive cluster-level log configuration information sent by the interface server through the proxy application.

[0093] In this step, since the interface server (Kube-apiserver) on the master node of the Kubernetes cluster is accessed in step S121 to request cluster-level log configuration information, the cluster-level log configuration information sent by the Kube-apiserver can be received through the proxy application, so that a running configuration file for executing the log collection function can be generated according to the log configuration information and the proxy application can be controlled to execute the running configuration file, thereby achieving the purpose of collecting log information of the working node.

[0094] In addition, in one embodiment of the present application, step S130 is further described. Step S130 may include but is not limited to the following steps:

[0095] Dynamically monitor the running events of the target container in the Docker daemon process in the working node through the proxy application.

[0096] It should be noted that in this step, the operation event of the target container includes any one of a start event, a stop event, or a restart event, which is not specifically limited here.

[0097] It should be noted that Docker daemon is the daemon process of Docker. By entering command lines in the Docker Client, the Docker Client and Docker Damon can communicate with each other, thereby completing relevant operations on Docker. Since Docker is a tool for creating containers, multiple containers can be managed through the Docker daemon. In other words, the Docker daemon can manage events such as startup events, shutdown events, or restart events of the target container. Therefore, by dynamically monitoring the running events of the target container in the Docker daemon in the working node through the proxy application, various running conditions of the target container can be captured, so that the subsequent steps can generate corresponding log collection configuration files based on the specific running conditions of the target container. Furthermore, when the subsequent steps obtain the log information generated by the target container according to the log collection configuration file, the association between the log information and the container name and the target container can be obtained, thereby solving the problem in the related art that Filebeat cannot know the relationship between the container log and the container, and achieving the purpose of more efficient and accurate storage of log collection of containers in the Kubernetes cluster.

[0098] In addition, an embodiment of the present application further illustrates the log collection method. After executing step S140, the log collection method may further include but is not limited to the following steps:

[0099] When an update event of the target container is monitored, the log collection configuration file corresponding to the target container is updated according to the update event.

[0100] It should be noted that, since Docker daemon can manage multiple containers, Docker daemon can realize the automatic deployment of target containers in working nodes. When it monitors the update event of the target container, it indicates that the target container has been updated. Since the log collection configuration file corresponds to the target container, when the target container is updated, the log collection configuration file needs to be updated synchronously. Specifically, the log collection configuration file corresponding to the target container can be updated according to the update event.

[0101] In addition, an embodiment of the present application further illustrates the log collection method. After executing step S140, the log collection method may further include but is not limited to the following steps:

[0102] When a deletion event of the target container is monitored, the log collection configuration file corresponding to the target container is deleted according to the deletion event.

[0103] It should be noted that this embodiment is parallel to the above-mentioned embodiment of monitoring the update event of the target container and updating the log collection configuration file.

[0104] It should be noted that, since Docker daemon can manage multiple containers, Docker daemon can realize the automatic deployment of target containers in working nodes. When the deletion event of the target container is monitored, it means that the target container has been deleted from the working node. Since the log collection configuration file corresponds to the target container, when the target container is deleted from the working node, the log collection configuration file needs to be deleted synchronously. Specifically, the log collection configuration file corresponding to the target container can be deleted according to the deletion event.

[0105] Reference Figure 5 As shown, an embodiment of the present application further illustrates step S150, which may include but is not limited to step S151 and step S152.

[0106] Step S151: Scan the target file directory of the working node at preset intervals through the proxy application.

[0107] It should be noted that when the target file directory of the working node is scanned through the proxy application, the target file directory of the working node can be scanned through the proxy application at preset intervals, so that the subsequent steps can incrementally load the target log information generated by the target container in the target file directory into the memory at preset intervals, thereby reducing the processing pressure of loading the target log information into the memory and improving the robustness of the system.

[0108] It should be noted that the preset time can be appropriately selected according to actual application conditions and is not specifically limited here. For example, in a feasible implementation, the target file directory of the working node can be scanned every 5 seconds by the proxy application.

[0109] Step S152: When target log information generated by the target container is scanned in the target file directory, the target log information in the target file directory is dynamically loaded into the memory.

[0110] In this step, since the target file directory of the working node is scanned at preset intervals in step S151, when the target log information generated by the target container is scanned in the target file directory, the target log information in the target file directory can be dynamically loaded into the memory, so that the subsequent steps can obtain the target log information associated with the container name and the target container from the memory, providing an accurate data basis for subsequent users to query the logs of specific containers.

[0111] In this step, since the target log information in the target file directory is dynamically loaded into the memory, the target log information can be incrementally loaded into the memory, thereby reducing the processing pressure of loading the target log information into the memory and improving the robustness of the system.

[0112] Reference Figure 6 As shown, an embodiment of the present application further illustrates step S160, which may include but is not limited to steps S161 to S163.

[0113] Step S161: Collect target log information from the memory according to the log collection configuration file.

[0114] In this step, since a log collection configuration file including the association relationship between the container name, the target container, and the log information generated by the target container is generated in step S140, and the target log information generated by the target container in the target file directory is loaded into the memory in step S150, the target log information can be collected from the memory according to the log collection configuration file, so that the target log information can be sent to the log service cluster for storage in subsequent steps.

[0115] It should be noted that since the log collection configuration file includes the association relationship between the container name, the target container, and the log information generated by the target container, when collecting the target log information from the memory according to the log collection configuration file, the association relationship between the target log information and the container name and the target container can be obtained at the same time, so that the subsequent steps can optimize the structure of the target log information according to the association relationship, thereby achieving the purpose of facilitating the storage and retrieval of the target log information.

[0116] Step S162: Structural processing is performed on the target log information according to the target container, the namespace corresponding to the target container, the container name corresponding to the target container, and the Kubernetes cluster corresponding to the target container.

[0117] In this step, since the target log information is collected from the memory according to the log collection configuration file in step S161, the target log information can be structured according to the target container, the namespace corresponding to the target container, the container name corresponding to the target container, and the Kubernetes cluster corresponding to the target container, so that the target log information after structured processing can be sent to the log service cluster for storage in subsequent steps.

[0118] It should be noted that structuring the target log information specifically refers to structurally associating the target log information with the target container, the namespace corresponding to the target container, the container name corresponding to the target container, and the Kubernetes cluster corresponding to the target container. For example, for the structured target log information, the corresponding target log information can be gradually found through the hierarchical relationship between the Kubernetes cluster, container name, namespace, and target container.

[0119] Step S163: Send the structured target log information to the log service cluster.

[0120] In this step, since the target log information is structured according to the target container, the namespace corresponding to the target container, the container name corresponding to the target container, and the Kubernetes cluster corresponding to the target container in step S162, the structured target log information can be sent to the log service cluster for storage, so that the user can perform accurate log queries for a specific container through the log service cluster.

[0121] It should be noted that, since the target log information saved in the log service cluster is structured, it is possible to save the target log information more efficiently and accurately.

[0122] It should be noted that, in a feasible implementation, after the structured target log information is sent to the log service cluster, the retrieval service unit in the log service cluster can provide an operation interface to the user. The user can use the operation interface provided by the log service cluster to set the conditions for searching the target log information, such as setting the target container, namespace, container name, Kubernetes and other conditions, and quickly find the required very fine-grained log information from the large amount of container log information in the log service cluster, thereby greatly reducing the difficulty of viewing and searching the container log information, reducing the user's viewing cost, improving the user's usage experience, saving the user's search time, improving efficiency, and thereby increasing the use value of the container.

[0123] Reference Figure 7 As shown, the embodiment of the present application further discloses a log collection device based on a Kubernetes cluster. The log collection device 200 can be set on a working node in a Kubernetes cluster, or the log collection device 200 can be a working node in a Kubernetes cluster, and can implement the log collection method based on a Kubernetes cluster as in the previous embodiment. The log collection device 200 includes:

[0124] A container construction module 210 is used to construct an independent information collection container, wherein the information collection container includes an agent application for collecting log information;

[0125] The container operation module 220 is used to start the agent application in the information collection container, obtain cluster-level log configuration information from the master node of the Kubernetes cluster through the agent application, generate a running configuration file for executing the log collection function based on the log configuration information, and control the agent application to execute the running configuration file;

[0126] An event monitoring module 230 is used to dynamically monitor the running events of the target container in the working node through a proxy application;

[0127] A file generation module 240 is configured to generate a log collection configuration file corresponding to a target container upon monitoring a running event, wherein the log collection configuration file includes an association between the container name, the target container, and the log information generated by the target container;

[0128] The information loading module 250 is used to scan the target file directory of the working node through the proxy application and load the target log information generated by the target container in the target file directory into the memory;

[0129] The information sending module 260 is used to collect target log information from the memory according to the log collection configuration file and send the target log information to the log service cluster.

[0130] In one embodiment, the log collection device 200 further includes:

[0131] A resource setting module, configured to set the memory capacity of the information collection container to be less than or equal to a preset capacity, and to set the number of cores of the central processing unit of the information collection container to be less than or equal to a preset number;

[0132] The path mapping module is used to map the root path of the working node to the target path inside the information collection container.

[0133] In one embodiment, the container execution module 220 includes:

[0134] An information request module, which is used to access the interface server on the master node of the Kubernetes cluster using the Hypertext Transfer Protocol through a proxy application to request cluster-level log configuration information;

[0135] The information receiving module is used to receive cluster-level log configuration information sent by the interface server through the proxy application.

[0136] In one embodiment, the event monitoring module 230 includes:

[0137] The information monitoring module is used to dynamically monitor the running events of the target container in the Docker daemon process in the working node through the proxy application; wherein the running event includes any one of the startup event, shutdown event or restart event.

[0138] In one embodiment, the log collection device 200 further includes:

[0139] An information update module is used to update the log collection configuration file corresponding to the target container according to the update event when monitoring the update event of the target container;

[0140] or,

[0141] The information deletion module is used to delete the log collection configuration file corresponding to the target container according to the deletion event when monitoring the deletion event of the target container.

[0142] In one embodiment, the information loading module 250 includes:

[0143] An information scanning module is used to scan the target file directory of the working node at preset intervals through an agent application;

[0144] The information loading submodule is used to dynamically load the target log information in the target file directory into the memory when the target log information generated by the target container is scanned in the target file directory.

[0145] In one embodiment, the information sending module 260 includes:

[0146] The information collection module is used to collect target log information from the memory according to the log collection configuration file;

[0147] The structured processing module is used to perform structured processing on the target log information based on the target container, the namespace corresponding to the target container, the container name corresponding to the target container, and the Kubernetes cluster corresponding to the target container;

[0148] The information sending submodule is used to send the structured target log information to the log service cluster.

[0149] Reference Figure 8 As shown, Figure 8 The following is a schematic diagram of the hardware structure of an electronic device provided in one embodiment of the present application, wherein the electronic device includes:

[0150] The processor 310 may be implemented using a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is configured to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0151] The memory 320 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 320 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 320 and is called by the processor 310 to execute the log collection method of the embodiments of this application.

[0152] Input / output interface 330, used to implement information input and output;

[0153] Communication interface 340, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.); and

[0154] bus 350 , which transmits information between each component of the device (e.g., processor 310 , memory 320 , input / output interface 330 , and communication interface 340 );

[0155] The processor 310 , the memory 320 , the input / output interface 330 and the communication interface 340 are connected to each other in communication within the device via the bus 350 .

[0156] An embodiment of the present application also provides a storage medium, which is a computer-readable storage medium used for computer-readable storage. The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the above-mentioned log collection method.

[0157] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0158] The embodiments described in the embodiments of this application are intended to more clearly illustrate the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0159] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0160] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0161] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0162] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0163] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the above-mentioned units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or can be integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0164] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0165] In addition, the functional units in each embodiment of the present application may be integrated into a processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The above-mentioned integrated units may be implemented in the form of hardware or software functional units.

[0166] If the assembled units are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method of each embodiment of the present application. The aforementioned storage medium includes various media that can store programs, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0167] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. A log collection method based on Kubernetes cluster, characterized in that: Applied to a working node in the Kubernetes cluster, the log collection method includes: Building an independent information collection container, wherein the information collection container includes an agent application for collecting log information; Starting the proxy application in the information collection container, obtaining cluster-level log configuration information from the master node of the Kubernetes cluster through the proxy application, generating a running configuration file for executing the log collection function based on the log configuration information, and controlling the proxy application to execute the running configuration file; Dynamically monitoring the running events of the target container in the working node through the proxy application; When the running event is monitored, a log collection configuration file corresponding to the target container is generated, wherein the log collection configuration file includes a container name of the target container and an association relationship between the target container and log information generated by the target container; Scanning the target file directory of the working node through the proxy application, and loading the target log information generated by the target container in the target file directory into the memory; The target log information is collected from the memory according to the log collection configuration file, and the target log information is sent to the log service cluster.

2. The log collection method according to claim 1, characterized in that: After the independent information collection container is constructed, the log collection method further includes: Setting the memory capacity of the information collection container to be less than or equal to a preset capacity, and setting the number of cores of the central processing unit of the information collection container to be less than or equal to a preset number; The root path of the working node is mapped to the target path inside the information collection container.

3. The log collection method according to claim 1, wherein: Obtaining cluster-level log configuration information from the master node of the Kubernetes cluster through the proxy application includes: Accessing the interface server on the master node of the Kubernetes cluster using the Hypertext Transfer Protocol through the proxy application to request cluster-level log configuration information; The cluster-level log configuration information sent by the interface server is received through the proxy application.

4. The log collection method according to claim 1, wherein: Dynamically monitoring the running events of the target container in the working node through the proxy application includes: Dynamically monitor the running events of the target container in the Docker daemon process in the working node through the proxy application; The operation event includes any one of a start event, a stop event or a restart event.

5. The log collection method according to claim 1, wherein: After generating the log collection configuration file corresponding to the target container, the log collection method further includes: When an update event of the target container is monitored, updating the log collection configuration file corresponding to the target container according to the update event; or, When a deletion event of the target container is monitored, the log collection configuration file corresponding to the target container is deleted according to the deletion event.

6. The log collection method according to claim 1, wherein: Scanning the target file directory of the working node by the proxy application and loading the target log information generated by the target container in the target file directory into the memory includes: Scanning the target file directory of the working node at preset intervals by the proxy application; When target log information generated by the target container is scanned in the target file directory, the target log information in the target file directory is dynamically loaded into the memory.

7. The log collection method according to claim 1, wherein: The collecting the target log information from the memory according to the log collection configuration file and sending the target log information to the log service cluster includes: Collecting the target log information from the memory according to the log collection configuration file; Performing structured processing on the target log information according to the target container, the namespace corresponding to the target container, the container name corresponding to the target container, and the Kubernetes cluster corresponding to the target container; The structured target log information is sent to the log service cluster.

8. A log collection device based on Kubernetes cluster, characterized in that: include: A container construction module, configured to construct an independent information collection container, wherein the information collection container includes an agent application for collecting log information; A container operation module is used to start the proxy application in the information collection container, obtain cluster-level log configuration information from the master node of the Kubernetes cluster through the proxy application, generate a running configuration file for executing the log collection function based on the log configuration information, and control the proxy application to execute the running configuration file; An event monitoring module, configured to dynamically monitor the running events of the target container in the working node through the proxy application; a file generation module, configured to generate a log collection configuration file corresponding to the target container upon monitoring the operation event, wherein the log collection configuration file includes a container name of the target container and an association relationship between the target container and log information generated by the target container; An information loading module is used to scan the target file directory of the working node through the proxy application, and load the target log information generated by the target container in the target file directory into the memory; The information sending module is used to collect the target log information from the memory according to the log collection configuration file, and send the target log information to the log service cluster.

9. An electronic device, characterized in that: The electronic device includes a memory, a processor, and a program stored on the memory and executable on the processor. When the program is executed by the processor, the steps of the log collection method based on the Kubernetes cluster are implemented.

10. A storage medium, which is a computer-readable storage medium and is used for computer-readable storage, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the log collection method based on the Kubernetes cluster according to any one of claims 1 to 7.

Images