A Kubernetes container log data processing method and device

Abstract

The application discloses a Kubernetes container log data processing method and device, which comprises the following steps: obtaining log files generated by a container runtime in a Kubernetes cluster; extracting an identifier of the log files according to a fixed naming rule of Kubernetes log files; attaching the identifier to an original log file and sending the original log file to a log content storage through a log content transmission channel; listening to resource changes in a Kubernetes API Server and collecting metadata; and sending the metadata to a log metadata storage through a metadata transmission channel with a four-tuple as a primary key. The application generates a lightweight mark through log file name analysis, thereby realizing separate transmission of logs and metadata, reducing log data redundancy, significantly reducing computing resource overhead, transmission and storage costs, and avoiding the problem of processing flow dependence on cluster availability.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of container log collection, and particularly relates to a Kubernetes container log data processing method and device. BACKGROUND

[0002] Kubernetes (K8s) as an open-source container orchestration platform has become the most important part of modern cloud-native architecture. With the widespread application of microservice architecture and containerized applications, enterprises have increasingly high demands for container management and monitoring, especially the collection, storage, and analysis of logs, which have become the key link to improve system stability and business monitoring. In Kubernetes clusters, container log management is not just a simple replacement of traditional log collection tools, it involves the complexity of container life cycle and efficient, scalable data processing capabilities. In order to ensure that problems can be quickly located when applications and services in the cluster have problems, Kubernetes container log collection technology has gradually developed a complete solution. Through a distributed log system, combined with internal and external monitoring tools, centralized management and efficient analysis of logs can be achieved, thereby improving the overall operation and fault diagnosis capabilities.

[0003] In the prior art, the collection and enhanced processing of Kubernetes container logs generally follow the following process:

[0004] 1. Log generation and storage

[0005] The container runtime writes the standard output (stdout) and standard error (stderr) log streams of processes within the container to the host file system. These log files are stored in a specific directory on the host (usually / var / log / containers / ), and their file names follow a fixed naming format: <pod_name>_<namespace_name>_<container_name>_<container_id>.log.

[0006] 2. Log content collection

[0007] A dedicated log collector deployed on the host continuously monitors the / var / log / containers / directory. When new log files or file content updates are detected, the collector reads the contents of these files.

[0008] 3. Metadata association processing

[0009] The log content read by the collector is only the original information output by the container application (example: 2025-08-0706:53:28.971INFO helloworld), and does not contain resource information associated with the container in the Kubernetes cluster (such as: Pod UID, IP, label, Namespace UID / label, associated ReplicaSet, running node information, and container image, etc.).

[0010] To supplement the missing cluster resource information, the log collector is usually configured with a special processor to supplement the metadata. The processor performs the following key operations:

[0011] (1) File name resolution: based on the fixed naming format of the log file (<pod name>_<namespace name>_<container name>_<container id>.log), the Pod name, Namespace name, container name and container ID are extracted.

[0012] (2) API query: using the key identification information obtained by parsing (mainly the Pod name), a query request is initiated to the Kubernetes APIServer to obtain the complete metadata of the Pod and its associated resources (including but not limited to Pod detailed information, associated Namespace, associated ReplicaSet, running node Node, etc.).

[0013] (3) Log enhancement: the Kubernetes resource metadata information obtained from the API Server is organized into a structured additional field (usually in the form of a nested object, for example: named kubernetes), and this additional field is merged with the original collected log message content, to finally generate an enhanced log record containing rich context information. The code example is as follows:

[0014]

[0015] However, the above Kubernetes container log collection and enhancement processing scheme has the following objective technical defects:

[0016] 1. Log data volume significantly expands

[0017] The byte length of the associated added Kubernetes resource metadata (such as: Pod, Node, Namespace detailed information, etc.) far exceeds the byte length of the original log message itself. The volume of these additional metadata can be several times or even tens of times the volume of the original log message, resulting in a significant increase in the overall data volume of a single enhanced log record.

[0018] 2. Metadata is highly redundant

[0019] The associated Kubernetes resource metadata (e.g. Pod UID, IP, labels, etc.) of a container running in the same pod remains unchanged during its life cycle. However, the prior art solution attaches these same metadata on each raw log record, resulting in a large amount of data fields that are completely duplicated.

[0020] 3. Increased computational resource overhead

[0021] It is a common practice to compress log data before transmission. Due to the significant expansion of data volume (shortcoming 1) and high redundancy (shortcoming 2), the amount of data to be processed during compression (at the sending end) and decompression (at the receiving end) increases significantly. This directly leads to an increase in CPU computational overhead for compression and decompression operations, increasing the processing burden of the system.

[0022] 4. Increased transmission and storage costs

[0023] The expansion and redundancy of data volume directly lead to:

[0024] Increased network transmission overhead: When log data is transmitted from the collector to the backend storage or analysis system, more network bandwidth is required.

[0025] Increased storage overhead: Storing expanded log data in the backend system requires more disk space or cloud storage resources.

[0026] 5. Processing flow relies on cluster availability

[0027] The processor of the log collector usually caches queried Kubernetes resource metadata to improve efficiency. However, when processing logs for new containers or cache entries expire / mismatch, the processor still needs to query the Kubernetes API server in real time to obtain the metadata.

[0028] In this scenario, if the Kubernetes API Server is temporarily unavailable (e.g. network interruption, service overload or fault restart, etc.), the processor is in a blocked waiting state (or needs to be repeatedly retried) due to the inability to obtain the required metadata, causing the processing flow associated with this part of the log to stall, thereby affecting the real-time collection, processing and transmission of log data. SUMMARY

[0029] To this end, the application provides a Kubernetes container log data processing method and device to solve the problems of high redundancy of log data collected by the existing container log collection method, large computing resource overhead, high transmission and storage cost, and processing flow dependent on cluster availability.

[0030] To achieve the above object, the application provides the following technical solutions:

[0031] In a first aspect, a Kubernetes container log data processing method is provided, which is applied to a log collector and includes the following steps:

[0032] Step 1: obtaining log files generated by a container runtime in a Kubernetes cluster;

[0033] Step 2: extracting an identifier of the log file according to a fixed naming rule of a Kubernetes log file;

[0034] Step 3: attaching the identifier to the original log file and sending it to a log content storage through a log content transmission channel for storage;

[0035] Step 4: listening to resource changes in a Kubernetes API server and collecting metadata;

[0036] Step 5: sending the metadata with a four-tuple as a primary key to a log metadata storage through a metadata transmission channel.

[0037] As a preferred, the method further includes the following steps:

[0038] The log queryer receives a query request of a user and submits a retrieval condition to the log metadata storage; the log metadata storage retrieves a matched four-tuple list according to the retrieval condition and returns it to the log queryer;

[0039] The log queryer receives the matched four-tuple list returned by the log metadata storage according to the retrieval condition and associates complete metadata according to the four-tuple list;

[0040] The log queryer queries original log content from the log content storage according to the four-tuple list and receives the original log content returned by the log content storage;

[0041] The log queryer returns the complete metadata and the original log content to the user after merging them.

[0042] As a preference, in step 2, the identifier adopts a four-tuple identifier, an identifier replacing a complete ID with a short container ID, an identifier of a hash value calculated according to a four-tuple, an identifier of a four-tuple converted into Base64, an identifier of an increased node name, or an identifier of an increased mirror version.

[0043] As a preference, in step 2, when the identifier adopts a four-tuple identifier, the four-tuple identifier is extracted after the file name is split by an underscore when extracting the identifier of the log file.

[0044] As a preference, the log collector adopts Filebeat, Fluent Bit, Fluentd, Logstash, Vector, Splunk Forwarder, or Datadog Agent.

[0045] As a preference, the log content transmission channel and the metadata transmission channel adopt a message queue or a streaming component.

[0046] As a preference, the message queue adopts Kafka, RabbitMQ, RocketMQ, AWS Kinesis, or Google Pub / Sub.

[0047] As a preference, the streaming component adopts an HTTP API, gRPC, or Syslog.

[0048] As a preference, the log content repository and the log metadata repository adopt Elasticsearch, ClickHouse, Loki, Splunk, PostgreSQL, MySQL, Oracle, Cassandra, or Redis.

[0049] In a second aspect, a Kubernetes container log data processing apparatus comprises:

[0050] A log file collection module is configured to acquire log files generated by a container runtime in a Kubernetes cluster.

[0051] A lightweight tag generation module is configured to extract an identifier of a Kubernetes log file according to a fixed naming rule of the log file.

[0052] A log content transmission module is configured to attach the identifier to an original log file and send the original log file to a log content repository for storage through a log content transmission channel.

[0053] A resource listening module is configured to listen to resource changes in a Kubernetes API Server and collect metadata.

[0054] a metadata transmission module, configured to send the metadata to a log metadata repository for storage through a metadata transmission channel with a four-tuple as a primary key.

[0055] Compared with the prior art, the present application has at least the following beneficial effects:

[0056] The present application provides a Kubernetes container log data processing method, which comprises the following steps: obtaining log files generated by a container runtime in a Kubernetes cluster; extracting an identifier of the log files according to a fixed naming rule of the Kubernetes log files; attaching the identifier to the original log files and sending the original log files to a log content repository for storage through a log content transmission channel; listening to resource changes in a Kubernetes API Server and collecting metadata; and sending the metadata to a log metadata repository for storage through a metadata transmission channel with a four-tuple as a primary key. The present application generates a lightweight tag through log file name analysis, thereby realizing separate transmission of logs and metadata, reducing log data redundancy, significantly reducing computing resource overhead, transmission and storage costs, and avoiding the problem of dependence of the processing flow on cluster availability. BRIEF DESCRIPTION OF DRAWINGS

[0057] In order to more intuitively illustrate the prior art and the present application, exemplary drawings are given below. It should be understood that the specific shapes, structures shown in the drawings should not be regarded as limiting conditions in the implementation of the present application; for example, based on the technical concepts disclosed in the present application and the exemplary drawings, those skilled in the art can easily make routine adjustments or further optimization of the increase / decrease / ownership division of certain units (components), specific shapes, positional relationships, connection modes, size ratio relationships, etc.

[0058] Figure 1 A Kubernetes container log data processing method flow chart is provided for the first embodiment of the present application.

[0059] Figure 2 A structure schematic diagram of a Kubernetes container log data processing method is provided for the first embodiment of the present application. DETAILED DESCRIPTION

[0060] The present application will be further described in detail below with reference to the accompanying drawings.

[0061] In the description of the present application: unless otherwise specified, the meaning of "a plurality of" is two or more. The terms "first", "second", "third" and the like in the present application are intended to distinguish the objects referred to, and do not have a special meaning in the technical connotation aspect (for example, it should not be understood as emphasizing importance or order, etc.). The expressions "include", "contain", "have" and the like also mean "not limited to" (certain units, components, materials, steps, etc.).

[0062] The terms such as "upper", "lower", "left", "right", "middle" and the like cited in the present application are generally indications for the intuitive understanding of the relative positional relationship according to the drawings, and are not absolute limitations on the positional relationship in the actual product.

[0063] Embodiment one

[0064] Please refer to Figure 1 and Figure 2 The embodiment provides a Kubernetes container log data processing method, which is applied to a log collector and comprises the following steps:

[0065] S1: acquiring log files generated by a container runtime in a Kubernetes cluster, that is, log content collection;

[0066] S2: extracting an identifier of the log file according to a fixed naming rule of the Kubernetes log file;

[0067] Specifically, the embodiment can utilize the fixed naming rule of the Kubernetes log file (<pod name>_<namespace name>_<container name>_<container id>.log), and the log collector automatically parses the identifier when reading the file. The identifier (that is, the lightweight mark) can be a four-tuple identifier, an identifier in which a short container ID replaces a complete ID, an identifier in which a hash value is calculated according to the four-tuple, an identifier in which the four-tuple is converted into Base64, an identifier in which a node name is added, or an identifier in which an image version is added, as shown in Table 1.

[0068] Table 1

[0069]

[0070] Among the above-mentioned identifiers, the four-tuple identifier is the preferred solution. When extracting the four-tuple identifier of the log file, the four-tuple component identifier is extracted by splitting the file name with an underscore, and is attached to the original log as a mark. Compared with the traditional solution, the data volume of the attached content is greatly reduced. Specifically, the four-tuple includes: pod_name (Pod name), namespace (namespace), container_name (container name), and container_id (container ID).

[0071] S3: Attach the identifier to the original log file and send it to the log content storage through the log content transmission channel for storage;

[0072] Specifically, the log content transmission channel can use a message queue or a streaming component. The message queue can use Kafka, RabbitMQ, RocketMQ, AWS Kinesis, or Google Pub / Sub. The streaming component (which can directly transmit) can use HTTP API, gRPC, or Syslog.

[0073] The log content storage can use any system that supports independent storage of original logs and metadata (as long as it supports separate storage of log content and metadata index), such as: log-specific storage: Elasticsearch, ClickHouse, Loki, or Splunk, etc.; general-purpose database: PostgreSQL, MySQL, Oracle, Cassandra, Redis, etc.

[0074] S4: Listen to the resource changes in the Kubernetes API Server and collect metadata, i.e., metadata asynchronous collection;

[0075] Specifically, the log collector listens to the changes of resources on the Kubernetes API Server in the background service, and collects the changes of all Pods, namespaces, and nodes in the cluster, etc. From which, the metadata is extracted, and the Pod-related metadata is taken as the primary key of the four-tuple, which is transmitted through a dedicated channel and stored in the database for subsequent query.

[0076] This process (i.e., metadata asynchronous collection) is completely decoupled from the log content collection process, so even if the Kubernetes API Server is temporarily unavailable, it will not affect the real-time storage of log content.

[0077] S5: Store the metadata with the four-tuple as the primary key in the log metadata storage through the metadata transmission channel.

[0078] Specifically, the metadata transmission channel can also use a message queue or a streaming component. The message queue can use Kafka, RabbitMQ, RocketMQ, AWS Kinesis, or Google Pub / Sub. The streaming component can use an HTTP API, gRPC, or Syslog2.

[0079] In this embodiment, the log collector splits the metadata and log data into two independent transmission channels. The log content transmission channel transmits the original log content with additional lightweight tags, and the metadata transmission channel transmits the metadata information collected from the Kubernetes API server. The two channels are sent through different message queues or transmission paths.

[0080] It should be noted that the mainstream log collectors currently provide related configuration capabilities, and no additional development is required to achieve data separation transmission.

[0081] The log metadata repository can use any system that supports independent storage of original logs and metadata (as long as it supports separate storage of log content and metadata index), such as log-specific storage: Elasticsearch, ClickHouse, Loki, or Splunk, etc.; general-purpose databases: PostgreSQL, MySQL, Oracle, Cassandra, Redis, etc.

[0082] The Kubernetes container log data processing method provided in this embodiment also includes dynamic association during query, that is, when a user needs to retrieve logs, the following steps are performed:

[0083] Step A: The log queryer receives the user's query request and submits the retrieval condition (such as app = nginx) to the log metadata repository. The log metadata repository retrieves the matching four-tuple list according to the retrieval condition and returns it to the log queryer;

[0084] Step B: The log queryer receives the matching four-tuple list returned by the log metadata repository according to the retrieval condition, and associates complete metadata according to the four-tuple list;

[0085] Step C: The log queryer queries the original log content from the log content repository according to the four-tuple list and receives the original log content returned by the log content repository;

[0086] Step D: The log queryer combines the complete metadata and the original log content and returns them to the user.

[0087] It should be noted that the metadata association operation (i.e. dynamically associating log content with metadata through an external service) is not limited to query phase triggering, and can be triggered in pre-association batch processing, stream delay association or hybrid mode:

[0088] Pre-association batch processing: periodically batch-inject new metadata into historical logs;

[0089] Stream delay association: asynchronous association is completed within a few seconds after the log is stored;

[0090] Hybrid mode: pre-association of high-frequency query fields, and on-demand association of low-frequency fields.

[0091] The log collector in the Kubernetes container log data processing method provided in this embodiment can use open source tools or commercial products. Among them, the open source tools can use Filebeat, Fluent Bit, Fluentd, Logstash or Vector, etc., and the commercial products can use Splunk Forwarder or Datadog Agent, etc. It should be noted that any log collection tool with "file name parsing capability" and "data shunting function" can implement the present solution, that is, the key capability requirements are: parsing the file name field + shunting the parsing result and the original log.

[0092] The Kubernetes container log data processing method provided in this embodiment separates the log and the metadata for processing, and its core idea is to reuse the inherent information of the log file name as a lightweight tag to realize separation during transmission and association during query, so it has the following advantages:

[0093] 1. Eliminate data expansion and redundancy

[0094] The original log remains simple, and only one copy of the metadata is stored globally. Compared with the traditional solution, the network bandwidth and storage cost are significantly reduced.

[0095] 2. Eliminate API dependency risk completely

[0096] The log content collection process is completely independent of the API Server, so even if the cluster control plane fails, the log content can still be continuously collected and stored (the metadata can be recorded later).

[0097] 3. Significant reduction in computing overhead

[0098] The compressed object is mainly the original log, and the amount of data processed is greatly reduced, so the CPU load is significantly reduced when processing at full speed.

[0099] 4. Compatible with existing ecosystem

[0100] The main stream collector can realize the data collection of the technical solution by simple modification through configuration collection process, plug-in mechanism or addition of external adapter.

[0101] The experimental data comparison (as shown in Table 2) shows that, under the same resource configuration:

[0102] Table 2

[0103] Each log occupies storage Log processing speed Conventional scheme 3.80 KB 32,410 per minute The scheme provided in this embodiment 1.03 KB 95,898 per minute

[0104] As can be seen from Table 2, the method provided in the embodiment has obvious advantages compared with the traditional scheme, and the log processing speed is about 3 times of the traditional scheme, and the storage overhead is less than 1 / 3 of the traditional scheme.

[0105] Embodiment two

[0106] The embodiment provides a Kubernetes container log data processing device, which comprises:

[0107] A log file collection module is configured to acquire log files generated by a container runtime in a Kubernetes cluster.

[0108] A lightweight tag generation module is configured to extract an identifier of a Kubernetes log file according to a fixed naming rule of the log file.

[0109] A log content transmission module is configured to attach the identifier to an original log file and send the original log file to a log content storage through a log content transmission channel.

[0110] A resource listening module is configured to listen to resource changes in a Kubernetes API Server and collect metadata.

[0111] A metadata transmission module is configured to send the metadata to a log metadata storage through a metadata transmission channel with a four-tuple as a primary key.

[0112] The specific implementation content of each module in the Kubernetes container log data processing device can be referred to the limitation of the Kubernetes container log data processing method in the foregoing description, and will not be described here.

[0113] The technical features of the above embodiments can be combined in any manner (as long as the combination of the technical features does not exist contradiction), in order to make the description simple, not all possible combinations of the technical features in the above embodiments are described; these embodiments not explicitly written should also be considered as the scope of the present disclosure.

Claims

1. A Kubernetes container log data processing method, characterized in that, The method is applied to a log collector, and includes: Step 1: obtaining a log file generated by a container runtime in a Kubernetes cluster; Step 2: extracting an identifier of the log file according to a fixed naming rule of a Kubernetes log file; specifically, the log collector automatically parses the identifier when reading the log file by using the fixed naming rule <pod name> _ <namespace name> _ <container name> _ <container id>. log of the Kubernetes log file; wherein the identifier is a four-tuple identifier, a short container ID instead of a complete ID identifier, a hash value identifier calculated according to the four-tuple, a Base64 identifier converted from the four-tuple, an added node name identifier, or an added image version identifier; when the identifier is the four-tuple identifier, the four-tuple identifier is extracted by splitting the file name by an underscore when extracting the identifier of the log file; Step 3: attaching the identifier to the original log file and sending to a log content storage through a log content transmission channel for storage; Step 4: listening to resource changes in a Kubernetes API Server and collecting metadata; Step 5: sending the metadata with the four-tuple as the primary key to a log metadata storage through a metadata transmission channel for storage.

2. The Kubernetes container log data processing method of claim 1, wherein, Further comprising: The log queryer receives a query request of a user and submits a retrieval condition to the log metadata storage; The log metadata storage retrieves a matched four-tuple list according to the retrieval condition and returns to the log queryer; The log queryer receives the matched four-tuple list returned by the log metadata storage according to the retrieval condition, and associates complete metadata according to the four-tuple list; The log queryer queries original log content from the log content storage according to the four-tuple list and receives the original log content returned by the log content storage; The log queryer returns the complete metadata and the original log content to the user after merging.

3. The Kubernetes container log data processing method of claim 1, wherein, The log collector uses Filebeat, Fluent Bit, Fluentd, Logstash, Vector, Splunk Forwarder, or DatadogAgent.

4. The Kubernetes container log data processing method of claim 1, wherein, The log content transmission channel and the metadata transmission channel use a message queue or a stream processing component.

5. The Kubernetes container log data processing method of claim 4, wherein, The message queue uses Kafka, RabbitMQ, RocketMQ, AWS Kinesis, or Google Pub / Sub.

6. The Kubernetes container log data processing method of claim 4, wherein, The stream processing component uses an HTTP API, gRPC, or Syslog.

7. The Kubernetes container log data processing method of claim 1, wherein, The log content storage and the log metadata storage use Elasticsearch, ClickHouse, Loki, Splunk, PostgreSQL, MySQL, Oracle, Cassandra, or Redis. 8.A Kubernetes container log data processing apparatus, characterized by, Comprising: A log file collection module is configured to acquire log files generated by a container runtime in a Kubernetes cluster. A lightweight label generation module is configured to extract an identifier of the log file according to a fixed naming rule of a Kubernetes log file. Specifically, the fixed naming rule of the Kubernetes log file is <pod name> _ <namespace name> _ <container name> _ <container id>.log, and the log collector automatically parses the identifier when reading the log file. The identifier is a four-tuple identifier, a short container ID instead of a complete ID identifier, a hash value identifier calculated according to the four-tuple, a Base64 identifier converted from the four-tuple, an added node name identifier, or an added image version identifier. When the identifier is the four-tuple identifier, the four-tuple identifier is extracted by separating the file name by an underscore when extracting the identifier of the log file. A log content transmission module is configured to attach the identifier to the original log file and send it to a log content storage through a log content transmission channel. A resource listening module is configured to listen to resource changes in the Kubernetes API Server and collect metadata. A metadata transmission module is configured to send the metadata to a log metadata storage through a metadata transmission channel with the four-tuple as the primary key.

Images