Container group log collection method and system based on kubernetes, electronic equipment and storage medium

Abstract

The invention discloses a kubernetes-based container group log collection method and system, electronic equipment and a storage medium, and the method comprises the steps: defining a collection rule which comprises a tag, a log path, a neglect file, a partition in kafka, a storage database and a log screening key value; configuring a collector based on the collection rule; the collector carries out collection based on configuration; configuring the collector based on the collection rule; judging whether the collection rule is added with a terminator or not; if not, adding a terminator; the terminator matches a corresponding pod through a label selector based on a label in the acquisition rule; analyzing the collection rule to construct a structure used for rendering a template, and adding pod into a structural body; and rendering a configuration file used by the collector based on the structural body. According to the method, log collection of the application is configured in a general mode, the application logs are dynamically collected in a container environment, and meanwhile, the configuration difference between different collectors is shielded.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a container group log collection method, system, electronic device and storage medium based on Kubernetes. Background Art

[0002] The logs printed by the application are very important information. When we need to analyze the operation of the application or find the cause of a fault, viewing the logs is a critical step. In a non-container environment, the application log path is static and fixed. Even if the application is restarted, its path will not change. Therefore, the log collector only needs to configure a static path. However, in a container environment, the application log is dynamic and uncertain. Not only do we need to consider the differences in the standard output log directories at different runtimes, but we also need to consider the differences in the volumes mounted by the application itself. At the same time, if the application pod is restarted, the path will also change. In addition, different log collectors have different configuration methods. If they are not unified, it will lead to additional learning and operation and maintenance costs. Summary of the invention

[0003] In view of the shortcomings existing in the above problems, the present invention provides a container group log collection method, system, electronic device and storage medium based on Kubernetes.

[0004] To achieve the above object, the present invention provides a container group log collection method based on Kubernetes, comprising:

[0005] Define collection rules, including tags, log paths, ignored files, partitions in Kafka, storage databases, and log filter keys;

[0006] Configuring a collector based on the collection rules;

[0007] The collector performs collection based on the configuration;

[0008] Wherein, configuring the collector based on the collection rule includes:

[0009] Determine whether a terminator is added to the collection rule; if not, add the terminator;

[0010] The terminator matches the corresponding pod based on the label in the collection rule through the label selector;

[0011] Parsing the collection rules to construct a structure for rendering a template and adding the pod to the structure;

[0012] A configuration file used by the collector is rendered based on the structure.

[0013] Preferably, the collection rules include application collection rules and host collection rules.

[0014] Preferably, the application collection rules include tags, standard output log paths, log paths, ignored files, partitions in Kafka, storage databases, and log filtering keys.

[0015] Preferably, the host collection rules include log paths, ignored files, partitions in Kafka, storage databases and log filtering keys.

[0016] Preferably, parsing the collection rules includes:

[0017] Based on the container ID, obtain the corresponding container information and container environment variable information;

[0018] Use regular expressions to match whether an environment variable exists in the path, and then search the container environment variables to see whether the environment variable exists. If so, replace the path with the real path.

[0019] Find the corresponding mount point based on the real path.

[0020] Preferably, if a standard output log path is defined, the corresponding standard output log path is found according to different operations.

[0021] The present application also provides a container group log collection system based on Kubernetes, including:

[0022] A definition module is used to define collection rules, including tags, log paths, ignored files, partitions in Kafka, storage databases, and log filtering keys;

[0023] A configuration module, used for configuring a collector based on the collection rules;

[0024] A collection module, used for the collector to collect data based on the configuration;

[0025] Wherein, configuring the collector based on the collection rule includes:

[0026] Determine whether a terminator is added to the collection rule; if not, add the terminator;

[0027] The terminator matches the corresponding pod based on the label in the collection rule through the label selector;

[0028] Parsing the collection rules to construct a structure for rendering a template and adding the pod to the structure;

[0029] A configuration file used by the collector is rendered based on the structure.

[0030] The present invention also provides an electronic device, comprising at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program, and when the program is executed by the processing unit, the processing unit executes the above method.

[0031] The present invention also provides a storage medium storing a computer program executable by an electronic device, and when the program runs on the electronic device, the electronic device executes the above method.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] The present invention adopts a universal method to configure application log collection, dynamically collects application logs in a container environment, and shields configuration differences between different collectors. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a flow chart of the container group log collection method based on Kubernetes of the present invention. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] Reference Figure 1 The present invention provides a container group log collection method based on kubernetes, comprising:

[0037] Define collection rules and apply them to the cluster. The collection rules include tags, log paths, ignored files, partitions in Kafka, storage databases, and log filter keys.

[0038] Configure collectors based on collection rules;

[0039] Specifically, the controller listens to the creation of collection rules and creates the underlying collector configuration;

[0040] The collector collects data based on the configuration;

[0041] Specifically, after the underlying collector configuration is created, the collector will collect according to the configuration;

[0042] In this embodiment, the controller refers to the component responsible for parsing the collection rules created by the user, and the collector refers to the component specifically responsible for collecting logs. The controller parses the collection rules created by the user according to the configuration format of different collectors, and generates the collector configuration, so that the collector works in a specified manner.

[0043] Specifically, the collection rules include application collection rules LogCollectionRule and host collection rules HostLogCollectionRule;

[0044] For the application collection rule LogCollectionRule, the definition is as follows:

[0045] matchLabels: the labels to be matched. Based on the labels defined by matchLabels, the workload to which the rule is to be applied is found through the label selector. When the controller parses the rules, the necessary related information, such as environment variables and mounting information, is only read from the workload, so that a single rule is effective only for a specific application.

[0046] stdout: standard output log, that is, when the application does not specifically output logs to a certain path, the default output location generally varies according to the different paths at runtime. For example, the default path at Docker runtime is / var / lib / docker / container, while that at containerd runtime is / var / log / pods. When configured to true, the corresponding standard output log path will be automatically parsed according to the runtime used by the current cluster and added to the generated collector configuration.

[0047] Path: Some applications configure a unified output path for logs to facilitate log classification and archiving. In this case, you need to configure a specific path and file name. The file name can be matched with wildcards. For example, *.log will collect all logs with the suffix .log. In addition, the path can include environment variables, such as / var / $(HOSTNAME). If the HOSTNAME environment variable is configured in the container, it will eventually be replaced by the value of the environment variable. The resolved path will be added to the generated collector configuration.

[0048] excludeFiles: Under the path configured in path, not all logs may need to be collected. For example, if *.log is configured but error.log is not desired to be collected, then error.log can be configured in excludeFiles to ignore the file during collection. This configuration requires the support of the collector itself. For example, the filebeat collector can decide the ignored files by configuring exclude_files.

[0049] kafkaTopic: After application logs are collected, they can be transferred through the Kafka message queue. You need to define which Kafkatopic to send them to. Kafkatopic can be understood as a partition in Kafka. The collector will have input and output configurations. The path configured above is the input, and Kafka is the output. After configuring this field, the collector will add topic information to the log after the collection is completed, and the output will be sent to the corresponding topic.

[0050] esIndex: After application logs are collected, they are finally stored in the elasticsearch database. You need to define which elasticsearch index to send them to. The elasticsearch index can be understood as a partition in elasticsearch. As above, after configuring this field, the collector will add index information to the log after the collection is completed, and it will be sent to the corresponding topic when outputting.

[0051] If the log volume is too large, you can transfer it through Kafka first, and add consumers, such as the logstash component (log processing component), to actively consume the data in Kafka and send the consumed data to elasticsearch. This can avoid sending a large number of logs to elasticsearch at the same time, causing excessive load;

[0052] fields: fields are composed of multiple key-value pairs. After fields are configured in the rule, the collector will add the key-value pairs defined by fields to the collection results after the collection is completed. After the collection is completed and stored in the database, these key-value pairs can be used to filter logs.

[0053] tags: tags are composed of multiple strings. After configuring tags in the rule, the collector will add tags such as tags: ['test1', 'test2'] to the collection results after the collection is completed. This configuration is mainly used to add tags to application logs for differentiation according to needs;

[0054] rootConfiguration: When collecting logs, you need to let the collector apply additional configurations. These configurations will affect the behavior of the collector. For example, if the log encoding is utf-8, you need to add encode=utf-8 to let the collector parse the log in the utf-8 encoding format. The configuration here is the native configuration parameter of the collector, which needs to be supported by the collector itself.

[0055] The host collection rule HostLogCollectionRule is defined as follows:

[0056] path: the log path to be collected on the host. The configuration method is the same as the path in LogCollectionRule;

[0057] excludeFiles: log files that do not need to be collected in the path configured under path;

[0058] kafkaTopic: After application logs are collected, they can be transferred through the Kafka message queue. You need to define which Kafkatopic to send them to, which is the same as the KafkaTopic in LogCollectionRule;

[0059] esIndex: After application logs are collected, they are finally stored in the elasticsearch database. You need to define which elasticsearch index they are sent to, which is the same as esIndex in LogCollectionRule;

[0060] fields: fields are composed of multiple key-value pairs. After fields are configured in the rule, the collector will add the key-value pairs defined by fields to the collection results after the collection is completed. After the collection is completed and stored in the database, these key-value pairs can be used to filter logs.

[0061] tags: tags are composed of multiple strings. After configuring tags in the rule, the collector will add tags such as tags: ['test1', 'test2'] to the collection results after the collection is completed. This configuration is mainly used to add tags to application logs for differentiation according to needs;

[0062] rootConfiguration: When collecting logs, you need to let the collector apply additional configurations. These configurations will affect the behavior of the collector. For example, if the log encoding is utf-8, you need to add encode=utf-8 to let the collector parse the log in the utf-8 encoding format. The configuration here is the native configuration parameter of the collector and needs to be supported by the collector itself.

[0063] When a collection rule is created or updated, the specific behavior of the controller is as follows:

[0064] If the rule has not been added with the finalizer "log.controller / finalizer", then add a finalizer for it and re-queue it; when the collection rule is deleted, all configurations generated by the rule will be cleaned up. Only after the cleanup is completed, the terminator will be removed and the rule can be deleted;

[0065] Use labelSelector to find the pods that match matchLabels in the collection rule. A pod is an instance of an application running on Kubernetes, also known as a workload. Each pod defines its own label. Use labelSelector to match pods that contain labels defined by the label selector.

[0066] Parse the collection rules, build a structure for rendering templates, and add pod metadata to the structure.

[0067] In this embodiment, during parsing, the corresponding log path mounting information will be found, which specifically includes the following steps:

[0068] According to the container ID, find the corresponding container information;

[0069] Find the environment variable information of the container;

[0070] For the log path saved when parsing the rules, a regular expression is used to match whether there is an environment variable such as $(HOSTNAME) in the path, and the container environment variable is searched for the environment variable. If it exists, the path is replaced with the real path. In this way, the log path can be dynamically configured through environment variables.

[0071] Find the container's mount point information;

[0072] If standard output is configured, find the corresponding standard output log path according to different runtimes. For docker runtime, its standard output log path is under / var / lib / docker / containers, while for containerd runtime, its standard output log path is generally under / var / log / pods.

[0073] Find the corresponding mount point on the host for the configured collection log path;

[0074] If the mount point is not found, you can use the container's upperdir to find the log. The upperdir is a concept of the container file system. The creation, modification, and deletion of all files in the container are reflected in the upperdir layer. Therefore, finding the upperdir can realize the collection of logs in the container without mounting. The upperdir of the docker runtime and the containerd runtime are also different. The upperdir of the docker runtime is / var / lib / docker / overlay2 / xxx / diff, and the upperdir of the containerd runtime is

[0075] / var / lib / containerd / io.containerd.snapshotter.v1.overlayfs / snapshots / xxx / fs. The specific path can be obtained from the obtained container information.

[0076] After parsing is completed, the configuration file used by the underlying collector is rendered through the template file. The template follows the go-template syntax. Even for different collectors with different configurations, the same collection rules can be rendered with different configurations by customizing the go-template template file. Since the go-template template is generally configured by R&D personnel, the differences between different collectors are shielded from users.

[0077] In addition, you can add the annotation "log.harmonycloud.cn / globalConfig" to the collection rules to specify the template file to be used. Therefore, you can adjust the generated configuration according to different collection scenarios without adjusting the collection rules, so that users will not feel it. A typical scenario is: In the cluster, there are already a number of applications that collect logs normally. At this time, you need to deploy another batch of applications for traffic playback. The logs they print need to be processed by special plug-ins, so the input, filter, and output plug-ins in the configuration file need to be adjusted. At this time, you do not need to make too many adjustments to the collection rules. After configuring the collection rules normally, you only need to add the annotation "log.harmonycloud.cn / globalConfig" = TrafficPlayBack.conf, and the template file TrafficPlayBack.conf will be used when rendering. For products, you only need to select the traffic playback option when creating rules. Users do not need to pay attention to the underlying layer and can adjust the template to generate the corresponding configuration.

[0078] After the configuration is generated, the collection work is handed over to the underlying collector. For example, for the filebeat collector, it will periodically read the configuration file in a fixed directory, so when a new configuration is generated or modified, it can be automatically loaded, and the controller does not need to do additional operations. For the fluentd collector, you need to find the child process generated by the fluentd process and restart it before you can load the latest configuration. In short, for different collectors, you only need to adapt at the controller code level to ensure that the collector can load the latest configuration in time when the configuration is generated or modified.

[0079] The controller monitors the pod and processes it, including the following logic:

[0080] For pod creation events, the container is not ready at this time and no processing is performed;

[0081] For pod update events, check whether the container is ready;

[0082] If the container is not ready, skip this event and wait for the next pod update event to be queued by the apiserver;

[0083] If the container is ready, traverse the collection rules under the pod command space to find out whether there is a collection rule that matches the pod label. If there is, enter the creation configuration logic, otherwise skip;

[0084] For pod deletion events, the collection rules under the pod command space are traversed to find out whether there is a collection rule that matches the podlabel. If there is, the remaining configuration of the pod is cleaned up.

[0085] The present application also provides a container group log collection system based on Kubernetes, including:

[0086] Definition module, used to define collection rules, including tags, log paths, ignored files, partitions in Kafka, storage databases, and log filtering keys;

[0087] A configuration module, used to configure the collector based on the collection rules;

[0088] The collection module is used by the collector to collect data based on the configuration;

[0089] The configuration of the collector based on the collection rules includes:

[0090] Determine whether a terminator is added to the collection rule; if not, add a terminator;

[0091] The terminator matches the corresponding pod based on the label in the collection rule through the label selector;

[0092] Parse the collection rules to build a structure for rendering templates and add pods to the structure;

[0093] Renders the configuration file used by the collector based on the structure.

[0094] The present invention also provides an electronic device, comprising at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program, and when the program is executed by the processing unit, the processing unit executes the above method.

[0095] The present invention also provides a storage medium storing a computer program executable by an electronic device. When the program runs on the electronic device, the electronic device executes the above method.

[0096] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A container group log collection method based on kubernetes, characterized in that: include: Define collection rules, including tags, log paths, ignored files, partitions in Kafka, storage databases, and log filter keys; Configuring a collector based on the collection rules; The collector performs collection based on the configuration; Wherein, configuring the collector based on the collection rule includes: Determine whether a terminator is added to the collection rule; if not, add the terminator; The terminator matches the corresponding pod based on the label in the collection rule through the label selector; Parsing the collection rules to construct a structure for rendering a template and adding the pod to the structure; A configuration file used by the collector is rendered based on the structure.

2. According to the Kubernetes-based container group log collection method according to claim 1, it is characterized in that: The collection rules include application collection rules and host collection rules.

3. The container group log collection method based on kubernetes according to claim 2 is characterized in that: The application collection rules include tags, standard output log path, log path, ignored files, partitions in Kafka, storage databases, and log filtering keys.

4. According to the Kubernetes-based container group log collection method according to claim 3, it is characterized in that: The host collection rules include log path, ignored files, partitions in Kafka, storage databases and log filtering keys.

5. The container group log collection method based on kubernetes according to claim 4 is characterized in that: Parsing the collection rules includes: Based on the container ID, obtain the corresponding container information and container environment variable information; Use regular expressions to match whether an environment variable exists in the path, and then search the container environment variables to see whether the environment variable exists. If so, replace the path with the real path. Find the corresponding mount point based on the real path.

6. The container group log collection method based on kubernetes according to claim 5 is characterized in that: If a standard output log path is defined, the corresponding standard output log path is found according to different operations.

7. A container group log collection system based on kubernetes, characterized in that: include: A definition module is used to define collection rules, including tags, log paths, ignored files, partitions in Kafka, storage databases, and log filtering keys; A configuration module, used for configuring a collector based on the collection rules; A collection module, used for the collector to collect data based on the configuration; Wherein, configuring the collector based on the collection rule includes: Determine whether a terminator is added to the collection rule; if not, add the terminator; The terminator matches the corresponding pod based on the label in the collection rule through the label selector; Parsing the collection rules to construct a structure for rendering a template and adding the pod to the structure; A configuration file used by the collector is rendered based on the structure.

8. An electronic device, characterized in that: The method comprises at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program, and when the program is executed by the processing unit, the processing unit executes the method according to any one of claims 1 to 6.

9. A storage medium, characterized in that: It stores a computer program executable by an electronic device. When the program runs on the electronic device, the electronic device executes the method according to any one of claims 1 to 6.

Images