Log collection method, log collector, program product, electronic equipment and storage medium

Abstract

The invention provides a log collection method, a log collector, a program product, electronic equipment and a storage medium, and is applied to the technical field of data processing.The log collection method comprises the steps that a log directory is scanned according to a preset hierarchical directory structure, and collection information corresponding to an application to be collected is obtained, the preset hierarchical directory structure comprises an application name, an environment identifier, a node IP and a log file name; configuration information corresponding to the to-be-collected application is obtained according to the collection information, and the configuration information comprises basic configuration and extension configuration; and collecting an application log corresponding to the to-be-collected application according to the configuration information. In the above scheme, the configuration information of each application is stored in the server, and service centralization is realized, so that the configuration information corresponding to the to-be-acquired application can be acquired from the server through the acquisition information, and the delay and complexity of log acquisition can be effectively reduced.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and more specifically, to a log collection method, a log collector, a program product, an electronic device, and a storage medium. Background Technology

[0002] Log collection is a core component of ensuring service observability in distributed systems and microservice architectures. Services are typically deployed in a distributed, multi-instance manner, and log data is generated in local files on different physical machines, containers, or nodes. These logs record all the details of service operation and are the core basis for problem localization, performance monitoring, and anomaly warning.

[0003] The existing log collection process typically involves: first, collecting service log files and forwarding them to Kafka; then, consuming the logs from Kafka into a data warehouse; and finally, querying the log data in the data warehouse through a visual interface. However, the complex configuration of existing log collection technologies leads to high latency and complexity in the process. Summary of the Invention

[0004] The purpose of this application is to provide a log collection method, log collector, program product, electronic device and storage medium to solve the technical problems of high latency and complexity in log collection in the prior art.

[0005] In a first aspect, embodiments of this application provide a log collection method, comprising: scanning a log directory according to a preset hierarchical directory structure to obtain collection information corresponding to an application to be collected, wherein the preset hierarchical directory structure includes an application name, an environment identifier, a node IP, and a log file name; obtaining configuration information corresponding to the application to be collected based on the collection information, wherein the configuration information includes basic configuration and extended configuration; and collecting application logs corresponding to the application to be collected based on the configuration information.

[0006] In the above scheme, a pre-defined hierarchical directory structure enables the collector to automatically identify newly added or modified applications. Therefore, the collection information of the application to be collected can be obtained by scanning the log directory. Furthermore, the configuration information of each application is stored on the server, realizing service centralization. Thus, the configuration information corresponding to the application to be collected can be obtained from the server through the collection information, replacing the local scattered configuration files. Compared with the prior art where one application attaches one configuration file, the log collection method provided in this application embodiment can effectively reduce the latency and complexity of log collection.

[0007] In an optional implementation, after collecting application logs corresponding to the application to be collected according to the configuration information, the log collection method further includes: receiving a backtracking instruction, wherein the backtracking instruction includes a backtracking time interval and a backtracking application configuration; and collecting historical logs within the backtracking time interval according to the backtracking application configuration. In the above scheme, by using the backtracking time interval and backtracking application configuration in the backtracking instruction, historical logs for a specific period can be collected in a targeted manner without full scanning, reducing resource consumption; in addition, when downstream Kafka consumer fails or data warehouse storage is abnormal, missing logs can be quickly collected to ensure data integrity.

[0008] In an optional implementation, the configuration information includes a target Kafka address and a target Topic. After obtaining the configuration information corresponding to the application to be collected based on the collection information, the log collection method further includes: creating a Kafka client based on the target Kafka address and the target Topic; after collecting application logs corresponding to the application to be collected based on the configuration information, the method further includes: sending the application logs to the Kafka client. In the above scheme, the client is dynamically created based on the target Kafka address and the target Topic, avoiding the long-term occupation of connection and memory resources by static Kafka clients; at the same time, the Kafka client is automatically created as the configuration changes, without the need to restart the collector, adapting to real-time updates of the server configuration and improving system flexibility.

[0009] In an optional implementation, collecting application logs corresponding to the application to be collected based on the configuration information includes: recording the collection progress information corresponding to the application logs and writing the collection progress information into a metadata file; reading the metadata file upon restart; and starting to collect the application logs from the progress corresponding to the collection progress information according to the metadata file. In the above scheme, the metadata file records the collection progress information, and collection resumes from the breakpoint after restarting, avoiding repeated collection of historical data or omission of new data, and reducing the pressure on downstream storage and processing; in addition, the collection progress information is stored in the metadata file, so even if the collector exits abnormally, the collection state can still be restored through the metadata, reducing the risk of log data loss.

[0010] Secondly, embodiments of this application provide a log collector, comprising: an application discovery module, configured to scan a log directory according to a preset hierarchical directory structure, wherein the preset hierarchical directory structure includes an application name, an environment identifier, a node IP, and a log file name; a configuration management module, connected to the application discovery module, configured to manage application configuration; an application management module, connected to the configuration management module, configured to obtain application configuration from the configuration management module; and a log collection module, connected to the application management module, configured to collect application logs according to the application configuration.

[0011] In the above scheme, a pre-defined hierarchical directory structure enables the collector to automatically identify newly added or modified applications. Therefore, the collection information of the application to be collected can be obtained by scanning the log directory. Furthermore, the configuration information of each application is stored on the server, realizing service centralization. Thus, the configuration information corresponding to the application to be collected can be obtained from the server through the collection information, replacing the local scattered configuration files. Compared with the prior art where one application attaches one configuration file, the log collection method provided in this application embodiment can effectively reduce the latency and complexity of log collection.

[0012] In an optional implementation, the log collector further includes a publishing module, connected to the application management module and the log collection module, used to create a Kafka client according to the application configuration and send the application logs to the Kafka client. In the above scheme, the client is dynamically created based on the target Kafka address and target Topic, avoiding the long-term occupation of connection and memory resources by static Kafka clients; simultaneously, the Kafka client is automatically created as the configuration changes, without requiring a collector restart, adapting to real-time updates of the server configuration and improving system flexibility.

[0013] In an optional implementation, the log collector further includes a status management module connected to the log collection module, used to record the collection progress information of the application logs and persistently store the collection progress information. In the above scheme, the metadata file records the collection progress information, and the collection can be resumed from the breakpoint after restarting, avoiding repeated collection of historical data or omission of new data, and reducing the pressure on downstream storage and processing; in addition, the collection progress information is stored in the metadata file, so even if the collector exits abnormally, the collection status can still be restored through the metadata, reducing the risk of log data loss.

[0014] Thirdly, embodiments of this application provide a computer program product, including computer program instructions, which are read and executed by a processor to perform the log collection method as described in the first aspect.

[0015] Fourthly, embodiments of this application provide an electronic device, including: a processor, a memory, and a bus; the processor and the memory communicate with each other via the bus; the memory stores computer program instructions executable by the processor, and the processor can execute the log collection method as described in the first aspect by calling the computer program instructions.

[0016] Fifthly, embodiments of this application provide a computer-readable storage medium that stores computer program instructions, which, when executed by a computer, cause the computer to perform the log collection method as described in the first aspect.

[0017] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, embodiments of this application are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 An architecture diagram of a log collector provided in an embodiment of this application; Figure 2 A flowchart illustrating a log collection method provided in this application embodiment; Figure 3 This is a structural block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0020] Before introducing the log collection method and log collector provided in the embodiments of this application, some terms in this field will be explained first.

[0021] Log collector: A software tool used to collect, process, and transmit log data from various sources.

[0022] Log rotation refers to automatically creating new log files and archiving or deleting old log files after the log files reach a certain size or time, in order to prevent the log files from growing indefinitely.

[0023] Log level: refers to the importance level of log information. Common log levels include DEBUG, INFO, WARNING, ERROR, and CRITICAL.

[0024] Kafka & Topic: Apache Kafka is a distributed stream processing platform primarily used for building real-time data pipelines and streaming applications. It is capable of processing real-time data with high throughput. In Kafka, messages are stored as streams in categories called Topics. A Topic can be viewed as a message queue; producers send messages to Topics, and consumers read messages from Topics.

[0025] Kubernetes is an open-source container orchestration platform used to automate the deployment, scaling, and management of applications. The DaemonSet deployment mode ensures that a Pod controller runs on all (or some) Nodes, typically used for log collection, monitoring, etc.

[0026] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0027] This application first introduces a log collection system provided in its embodiments. This system may include a client, a server, a Kafka platform, a log collector, and an application. Users can interact with the log collection system through the client, for example, by modifying application configuration information. The server, which can be understood as a log backend, stores application configuration information and other data. The Kafka platform stores the collected application logs. The log collector collects application logs. The application prints logs and generates corresponding log directories.

[0028] The following describes the specific implementation of the log collector in the above log collection system. The log collector may include: an application discovery module, a configuration management module, an application management module, and a log collection module.

[0029] Specifically, the application discovery module scans the log directory according to a preset hierarchical directory structure. This preset hierarchical directory structure refers to the standardized path format for storing log files. As one implementation, the preset hierarchical directory structure may include the application name, environment identifier, node IP, subdirectories, and log file names. For example, the preset hierarchical directory structure can be represented as / {application name} / {environment identifier} / {node IP} / {subdirectory} / {log file name}, such as / app1 / t1 / 127.0.0.1 / tmp / run.log.

[0030] Among them, the application name refers to the unique identifier of the business service (e.g., app1), used to distinguish different business systems; the environment identifier refers to the environment in which the service is deployed (e.g., test environment t1, production environment prod), supporting multi-environment log isolation; the node Internet Protocol (IP) refers to the IP address of the physical machine or container node where the service runs (e.g., 127.0.0.1), used to locate the specific running node; and the log file name indicates the specific file where logs are stored (e.g., run.log, info.log), carrying different types of log content.

[0031] As one implementation method, the application discovery module can periodically scan the log directory, parse the application name, environment identifier, and node IP in the path, and output a list of applications to be collected; for example, when scanning / app2 / t1 / 127.0.0.1 / log / info.log, the output is {name: "app2", env: "t1", ip: "127.0.0.1", file: "log / info.log"}.

[0032] The configuration management module connects to the application discovery module and is used to manage application configurations. Specifically, the configuration management module can integrate multi-source basic configurations and extended configurations, and receive information collected by the application discovery module, caching the returned application configurations.

[0033] The basic configuration refers to the core rules for log collection, including the collection switch (enabled), log paths (paths), multiline matching rules (multiline.pattern), and the ignore_older threshold (ignore_older). The extended configuration refers to the rules related to log distribution, which can be fields appearing in the form of a map[string]any, and may include the target Kafka address (Kafka_server), the target Topic (Topic), and the discard switch (discard), etc. It is understood that this application does not specifically limit the specific implementation methods of the above-mentioned basic and extended configurations, and those skilled in the art can make appropriate selections according to the actual situation.

[0034] The application management module connects with the configuration management module to obtain application configurations. Specifically, the configuration management module can obtain application configurations from the configuration management module, parse them, generate collection tasks (including log paths, Kafka information, etc.), check for configuration updates every 5 seconds, and dynamically adjust tasks (such as adding / deleting collection targets).

[0035] The log collection module connects to the application management module and is used to collect application logs according to the application configuration. Specifically, the log collection module can execute collection tasks generated by the application management module, monitor for new content in log files, process multiple lines of logs according to configuration, and filter invalid data, etc.

[0036] In the above scheme, a pre-defined hierarchical directory structure enables the collector to automatically identify newly added or modified applications. Therefore, the collection information of the application to be collected can be obtained by scanning the log directory. Furthermore, the configuration information of each application is stored on the server, realizing service centralization. Thus, the configuration information corresponding to the application to be collected can be obtained from the server through the collection information, replacing the local scattered configuration files. Compared with the prior art where one application attaches one configuration file, the log collection method provided in this application embodiment can effectively reduce the latency and complexity of log collection.

[0037] Furthermore, based on the above embodiments, the log collector provided in this application embodiment may also include: a publishing module.

[0038] Specifically, the publishing module connects with the application management module and the log collection module to create Kafka clients based on application configurations and send application logs to the Kafka clients.

[0039] As one implementation, the publishing module can dynamically create Kafka clients: create an independent client for each task, and record the client ID and last active time; as another implementation, the publishing module can encapsulate the collected log content and send it to the target Kafka Topic through the client; as yet another implementation, the publishing module can iterate through the client list every minute and destroy clients whose last active time plus 15 minutes is less than the current time. For example, after app2 goes offline, its client will not send anything for 15 minutes and will be automatically destroyed.

[0040] In the above solution, clients are dynamically created based on the target Kafka address and target Topic, avoiding the long-term occupation of connection and memory resources by static Kafka clients. At the same time, Kafka clients are automatically created when configuration changes occur, without the need to restart the collector, adapting to real-time updates of server configurations and improving system flexibility.

[0041] Furthermore, based on the above embodiments, the log collector provided in this application embodiment may also include: a status management module.

[0042] Specifically, the status management module is connected to the log collection module to record the progress information of application log collection and to persistently store the collection progress information.

[0043] As one implementation, the status management module can receive the collection progress (such as the current line number in run.log) from the log collection module in real time, and then write the progress information to a metadata file (JSON format) in the format {log file path: line number}, supporting local persistence. This way, when the log collector starts, it can read the metadata file and pass the breakpoint progress of each file to the log collection module, ensuring that the resume collection function is effective. As another implementation, if the metadata file is corrupted, the status management module can automatically start collecting from the beginning of the log file and record exception logs.

[0044] In the above scheme, the metadata file records the collection progress information. After restarting, the collection can resume from the breakpoint, avoiding the repeated collection of historical data or the omission of new data, and reducing the pressure on downstream storage and processing. In addition, the collection progress information is stored in the metadata file. Even if the collector exits abnormally, the collection status can still be restored through the metadata, reducing the risk of log data loss.

[0045] Please refer to Figure 1 , Figure 1 The following is an architecture diagram of a log collector provided in an embodiment of this application. The modules in the log collector are described in turn below.

[0046] Input layer (Ark, File, CLI, Env): Used to provide multi-source configuration input (framework integration, file, command line, environment variables) and provide collection rules for Config Manager.

[0047] Config Manager: This is the basic configuration for the data collector. Note the difference between this and application configuration. Application configuration stores the collection rules in the log center and retrieves them in real time, while this part of the configuration includes basic settings such as collection mode, log center address, and default Kafka address. Application configuration can only be dynamically retrieved from the log center if the log center address is known, while basic configuration can be injected and modified through four methods: environment variables (env), startup command (cli), configuration file (file), and configuration center (ark).

[0048] FS Scanner: This refers to a file scanner used to periodically scan the directory / var / lib / kubelet / pods / {PodID} / volumes / kubernetes.io~empty-dir / tmp-dir / {name} / {env} / {ip} / ... Once a new service is deployed on a node, a new directory / {name} / {env} / {ip} will be added to the log directory.

[0049] FS Inotify Mod: Used to listen for file system events (such as additions and modifications) to achieve real-time monitoring.

[0050] K8s App Discover: refers to the application discoverer, which is used to abstract the scanned log directory into an application by calling the file scanner.

[0051] App Manager: This refers to the Log Task Manager, which retrieves application configurations from the log center based on applications discovered by the application discoverer, and generates a log task to collect logs. Application logs can be stored on a server for centralized management, and the Log Task Manager can perform scheduled scans and configuration updates of application logs, for example, scanning and updating every 5 seconds.

[0052] HTTP Server: This provides an HTTP service that allows users to query data collection status, perform control functions, stop or restart tasks, and modify task configurations. It is primarily used for debugging and provides an entry point for manual intervention.

[0053] State Manager: This refers to the state manager, which manages metadata. It is mainly used to record xx1.log (collected up to line xx) and xx2.log (collected up to line xx). It records this data and stores it in a JSON file periodically. This way, when the service restarts, it does not have to start collecting from the beginning, but can continue collecting from the previously collected line xx, thereby reducing log duplication.

[0054] Publisher: This refers to the sender, which is used to send the collected log data to a specified topic in a specified Kafka instance. It can include Kafka Publisher and Console Publisher. Manager GUI: This refers to the visual interface of the log backend, which supports configuration submission and log querying, and serves as the entry point for user interaction.

[0055] Controller: refers to the controller; Metrics Exporter: refers to the metrics exporter; File: refers to the file; Collector: refers to the collector; MD Redis: refers to the Redis instance used in the master data; Filesystem: refers to the file system; Terminal Manager Services: refers to the terminal management services.

[0056] Based on the above-described log collection system, this application also provides a log collection method, which can be applied to the log collector in the above-described log collection system. Please refer to... Figure 2 , Figure 2A flowchart of a log collection method provided in this application embodiment is shown. This method can be applied to situations where it can be, but is not limited to, executed by an electronic device. Figure 3 The possible structure of this electronic device is shown below; for details, please refer to the following section. Figure 3 The above-mentioned log collection method may specifically include the following steps: S101: Scan the log directory according to the preset hierarchical directory structure to obtain the collection information corresponding to the application to be collected.

[0057] S102: Obtain the configuration information corresponding to the application to be collected based on the collected information.

[0058] S103: Collect application logs corresponding to the application to be collected based on the configuration information.

[0059] Specifically, in S101 above, the preset hierarchical directory structure refers to the standardized path format for storing log files. As one implementation, the preset hierarchical directory structure may include the application name, environment identifier, node IP, and log file name. For example, the preset hierarchical directory structure can be represented as / {application name} / {environment identifier} / {node IP} / {subdirectory} / {log file name}, such as: / app1 / t1 / 127.0.0.1 / tmp / run.log.

[0060] Among them, the application name refers to the unique identifier of the business service (e.g., app1), used to distinguish different business systems; the environment identifier refers to the environment in which the service is deployed (e.g., test environment t1, production environment prod), supporting multi-environment log isolation; the node Internet Protocol (IP) refers to the IP address of the physical machine or container node where the service runs (e.g., 127.0.0.1), used to locate the specific running node; and the log file name indicates the specific file where logs are stored (e.g., run.log, info.log), carrying different types of log content.

[0061] The log directory refers to the file system path where application log files are stored. This path follows a preset hierarchical directory structure and is the core data source for the log collector to scan log files. By scanning the log directory according to the preset hierarchical directory structure, the log collector can automatically extract key information such as application name and environment identifier through directory scanning, without the need for manual configuration of the service list.

[0062] The application to be collected refers to the business service or application that needs to have its logs collected. It is automatically identified and abstracted by the log collector by scanning the log directory. Each application to be collected corresponds to a unique combination of application name and environment identifier.

[0063] As one implementation method, when the log collector scans the log directory, it parses the application name and environment identifier in the path to abstract a collection of log files of the same service in the same environment into a single application to be collected. For example, when scanning the paths / app1 / t1 / 127.0.0.1 / tmp / run.log and / app1 / t1 / 192.168.1.1 / log / error.log, since they have the same application name, they can be abstracted into the same application to be collected.

[0064] In this embodiment, the log collector can scan the log directory according to a preset hierarchical directory structure using the FS Scanner module, thereby obtaining the collection information corresponding to the application to be collected. This collection information corresponds to the preset hierarchical directory structure and may include the application name, environment identifier, node IP, and log file name of the application to be collected.

[0065] In S102 above, the configuration information may include basic configuration and extended configuration. Basic configuration refers to the core rules for log collection, including the collection switch (enabled), log paths (paths), multiline matching rules (multiline.pattern), and the ignore old file threshold (ignore_older), etc. Extended configuration refers to the rules related to log distribution, which can be fields appearing in the form of a map[string]any, and may include the target Kafka address (Kafka_server), target Topic (Topic), and log discard switch (discard), etc. It is understood that the embodiments of this application do not specifically limit the specific implementation of the above-mentioned basic and extended configurations, and those skilled in the art can make appropriate selections according to the actual situation.

[0066] In this embodiment, the log collector can use the App Manager module to send a configuration request to the log center using the application name and environment identifier as query keys, thereby obtaining the configuration information corresponding to the application to be collected based on the collected information. In S103 above, the log collector can collect application logs corresponding to the application to be collected based on the configuration information obtained in S102 above. That is to say, the configuration information of all services is stored on the server side, and the user side can automatically modify it through work orders; the log collector discovers the application name and environment identifier of the service through directory scanning, and then pulls the corresponding configuration information from the service through the application name and environment identifier, realizing service centralization and dynamic configuration.

[0067] In the above scheme, a pre-defined hierarchical directory structure enables the collector to automatically identify newly added or modified applications. Therefore, the collection information of the application to be collected can be obtained by scanning the log directory. Furthermore, the configuration information of each application is stored on the server, realizing service centralization. Thus, the configuration information corresponding to the application to be collected can be obtained from the server through the collection information, replacing the local scattered configuration files. Compared with the prior art where one application attaches one configuration file, the log collection method provided in this application embodiment can effectively reduce the latency and complexity of log collection.

[0068] Furthermore, based on the above embodiments, after S103, the log collection method provided in this application embodiment may further include the following steps: S201: Receive backtracking command.

[0069] S202: Collect historical logs within the backtracking time interval according to the backtracking application configuration.

[0070] Specifically, in S201 above, the backtracking instruction includes the backtracking time interval and the backtracking application configuration; in S202 above, historical logs within the backtracking time interval can be further collected according to the backtracking application configuration.

[0071] Therefore, this application embodiment supports log backtracking functionality. By configuring backtrack[time1, time2], the log collector can start an additional program to scan log files within the corresponding time interval, and then perform additional sampling on the scanned files. This can be used in scenarios where log backtracking is needed due to Kafka anomalies or downstream consumer scenario anomalies.

[0072] In the above solution, by configuring the backtracking time range and backtracking application in the backtracking command, historical logs for a specific period can be collected in a targeted manner without full scanning, thus reducing resource consumption. In addition, when downstream Kafka consumer fails or data warehouse storage is abnormal, missing logs can be quickly collected to ensure data integrity.

[0073] Furthermore, based on the above embodiments, the configuration information may include the target Kafka address and the target Topic. In this case, after S102 above, the log collection method provided in this application embodiment may further include the following steps: S301: Create a Kafka client based on the target Kafka address and the target Topic.

[0074] Accordingly, after S103 above, the log collection method provided in this application embodiment may further include the following steps: S302: Send application logs to the Kafka client.

[0075] Specifically, because services on Kubernetes nodes are dynamically scheduled, for example, node A might have three services (x1, x2, and x3) at point 1, but at point 2, with services coming online and going offline, the services might become x1, y2, and y3. Therefore, services are dynamically created, while log collectors are not bound to the lifecycle of services; a log collector might not restart for an entire quarter.

[0076] However, the Kafka and Topic configurations may differ for each service. Creating a Kafka client for each service would waste significant resources. Therefore, the Kafka client in this embodiment is dynamically created, and this embodiment supports multi-Kafka, multi-Topic technology.

[0077] As one implementation method, if a Kafka client does not write logs within a preset time period (e.g., 15 minutes), the Kafka client will be destroyed and its resources will be reclaimed.

[0078] As another implementation method, multiple Kafka single Topic technology or single Kafka multiple Topic technology can also be used. This application does not make specific limitations on this, and those skilled in the art can make appropriate adjustments according to the actual situation.

[0079] In the above solution, clients are dynamically created based on the target Kafka address and target Topic, avoiding the long-term occupation of connection and memory resources by static Kafka clients. At the same time, Kafka clients are automatically created when configuration changes occur, without the need to restart the collector, adapting to real-time updates of server configurations and improving system flexibility.

[0080] Furthermore, based on the above embodiments, S103 may specifically include the following steps: S401: Record the collection progress information corresponding to the application log and write the collection progress information to the metadata file.

[0081] S402: Read the metadata file upon reboot.

[0082] S403: Start collecting application logs from the progress corresponding to the collection progress information based on the metadata file.

[0083] Specifically, in S401 above, the acquisition progress information can be used to record how many lines have been acquired for each log file, for example: recording that xx lines of log file a have been acquired and xx lines of log file b have been acquired. This acquisition progress information is stored in memory in real time; for each additional line of data acquired, this field is updated and incremented by one.

[0084] In the above S402, in order to reduce duplicate collection, the collection progress information can be written to a JSON file (i.e., metadata file) periodically. Normal exit will also write to the JSON file. When the log collector restarts, it will read the metadata from the JSON file and continue reading the previously read values.

[0085] In the above S403, the metadata file stores the collection progress information; if the log collector restarts, it will automatically load the collection progress information stored in the metadata file, so that it can continue to collect based on the collection progress information without re-collecting.

[0086] As one implementation method, the metadata file can be mounted as directory information to avoid log resampling. Since the log collector is deployed as a pod, the normal directory is destroyed when the pod stops, but the mounted directory is not. Therefore, the metadata JSON file can be placed in the mounted directory.

[0087] In the above scheme, the metadata file records the collection progress information. After restarting, the collection can resume from the breakpoint, avoiding the repeated collection of historical data or the omission of new data, and reducing the pressure on downstream storage and processing. In addition, the collection progress information is stored in the metadata file. Even if the collector exits abnormally, the collection status can still be restored through the metadata, reducing the risk of log data loss.

[0088] Furthermore, based on the above embodiments, the log collection method provided in this application supports three modes: ds, sidecar, and eCS.

[0089] Specifically, ds mode is the daemonset mode of Kubernetes, designed for services deployed on Kubernetes. Kubernetes is a container orchestration and management service where multiple server nodes form a Kubernetes service. Each node runs many pods, and specific services run on these pods. Log collection aims to collect logs from all pods on a node. In daemonset mode, the collector is treated as a pod, with each node running a collector pod responsible for collecting logs from all other pods.

[0090] The sidecar mode is also for services deployed on Kubernetes, such as some important services or services with a large amount of logs. If these services use the DS mode, there will be a problem of resource contention because one collector collects logs from multiple pods. The sidecar mode creates an independent collector for each pod, and the collector only collects logs from that pod, ensuring that there is no resource contention.

[0091] ECS mode refers to the service being deployed directly on a server instead of on a Kubernetes cluster.

[0092] The choice between these three modes depends primarily on the deployment location of the service being collected and its importance. If the service is deployed on an ECS instance, use the ECS mode; if it's deployed on a Kubernetes instance, the DS mode is generally used; and if it's deployed on a Kubernetes instance and is critical or has a large volume of logs, the sidecar mode is used.

[0093] Furthermore, based on the above embodiments, the low-cost key-value storage database can proactively maintain a TCP connection with the monitoring and management services, thereby supporting data reporting at the second level. Compared to existing Redis monitoring, this solution provides richer monitoring metrics (slot migration data) and a supporting backend management system. It also allows for visualization of the runtime status of the low-cost key-value storage database through the backend, enabling node online / offline operations and slot data migration.

[0094] Furthermore, based on the above embodiments, the low-cost key-value storage engine can perform slotted storage of data, with each data slot storing data in the same DB file. When the cluster is scaling up or down, data is migrated by slot as the migration unit. During the migration, the slot to be migrated enters write blocking mode, obtains the local DB file, and transmits network data via streaming. Upon detecting a node failure, the slot data migration process for the failed node can be automatically initiated through configuration. For hot slots, migration to a node with low load can be specified to ensure stability.

[0095] Furthermore, based on the above embodiments, the low-cost key-value storage database supports a wider range of data formats than Redis. It provides relational data storage through a built-in Table data structure and supports index creation, improving data indexing performance. In addition, data stored on disk can support petabyte-level storage, storing more data than Redis.

[0096] In this embodiment, firstly, the log collector and log collection method provided can achieve a 1C10M log collection performance with minimal collection latency, thus exhibiting excellent collection performance. Secondly, the collection configuration in the log collector and log collection method provided in this embodiment is located on the server side, and dynamic real-time retrieval facilitates maintenance. Furthermore, the log collector and log collection method provided in this embodiment support multiple Kafka and multiple topics, thus offering flexible configuration; simultaneously, the Kafka client is destroyed within 15 minutes if not in use, thus not consuming additional resources.

[0097] Please refer to Figure 3 , Figure 3This application provides a structural block diagram of an electronic device 300, which includes at least one processor 301, at least one communication interface 302, at least one memory 303, and at least one communication bus 304. The communication bus 304 enables direct communication between these components, the communication interface 302 facilitates signaling or data communication with other node devices, and the memory 303 stores machine-readable instructions executable by the processor 301. When the electronic device 300 is running, the processor 301 communicates with the memory 303 via the communication bus 304, and the machine-readable instructions, when invoked by the processor 301, execute the aforementioned log collection method.

[0098] For example, the processor 301 in this embodiment of the application can read a computer program from the memory 303 via the communication bus 304 and execute the computer program to implement the following method: scanning the log directory according to a preset hierarchical directory structure to obtain the collection information corresponding to the application to be collected, wherein the preset hierarchical directory structure includes the application name, environment identifier, node IP and log file name; obtaining the configuration information corresponding to the application to be collected according to the collection information, wherein the configuration information includes basic configuration and extended configuration; and collecting the application logs corresponding to the application to be collected according to the configuration information.

[0099] The processor 301 may include one or more, and may be an integrated circuit chip with signal processing capabilities. The processor 301 may be a general-purpose processor, including a Central Processing Unit (CPU), a Microcontroller Unit (MCU), a Network Processor (NP), or other conventional processors; it may also be a special-purpose processor, including a Neural-network Processing Unit (NPU), a Graphics Processing Unit (GPU), a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. Furthermore, when there are multiple processors 301, some may be general-purpose processors, and others may be special-purpose processors.

[0100] The memory 303 includes one or more, which may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0101] Understandable. Figure 3 The structure shown is for illustrative purposes only; the electronic device 300 may also include components that are more advanced than those shown. Figure 3 The more or fewer components shown, or having the same Figure 3 The different configurations shown. Figure 3 The components shown can be implemented using hardware, software, or a combination thereof. In the embodiments of this application, electronic device 300 can be, but is not limited to, physical devices such as desktop computers, laptops, smartphones, smart wearable devices, and in-vehicle devices, or virtual devices such as virtual machines. Furthermore, electronic device 300 is not necessarily a single device; it can be a combination of multiple devices, such as a server cluster, etc.

[0102] This application also provides a computer program product, including a computer program stored on a computer-readable storage medium. The computer program includes computer program instructions. When the computer program instructions are executed by a computer, the computer can perform the steps of the log collection method described in the above embodiments, such as: S101: Scanning the log directory according to a preset hierarchical directory structure to obtain collection information corresponding to the application to be collected. S102: Obtaining configuration information corresponding to the application to be collected based on the collection information. S103: Collecting application logs corresponding to the application to be collected based on the configuration information.

[0103] This application also provides a computer-readable storage medium that stores computer program instructions. When the computer program instructions are executed by a computer, the computer performs the log collection method described in the foregoing method embodiments.

[0104] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0105] Furthermore, the units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0106] Furthermore, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0107] It should be noted that if the function is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0108] In this document, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, without necessarily requiring or implying any such actual relationship or order between these entities or operations.

[0109] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A log collection method, characterized in that, include: The log directory is scanned according to a preset hierarchical directory structure to obtain the collection information corresponding to the application to be collected. The preset hierarchical directory structure includes the application name, environment identifier, node IP, and log file name. Based on the collected information, the configuration information corresponding to the application to be collected is obtained, wherein the configuration information includes basic configuration and extended configuration; The application logs corresponding to the application to be collected are collected based on the configuration information.

2. The log collection method according to claim 1, characterized in that, After collecting the application logs corresponding to the application to be collected based on the configuration information, the method further includes: Receive a backtracking instruction, wherein the backtracking instruction includes a backtracking time interval and a backtracking application configuration; Historical logs within the backtracking time interval are collected according to the backtracking application configuration.

3. The log collection method according to claim 1, characterized in that, The configuration information includes the target Kafka address and the target Topic. After obtaining the configuration information corresponding to the application to be collected based on the collected information, the method further includes: Create a Kafka client based on the target Kafka address and the target Topic; After collecting the application logs corresponding to the application to be collected based on the configuration information, the method further includes: Send the application logs to the Kafka client.

4. The log collection method according to claim 1, characterized in that, The step of collecting application logs corresponding to the application to be collected according to the configuration information includes: Record the collection progress information corresponding to the application log, and write the collection progress information into the metadata file; Upon restart, the metadata file is read; The application logs are collected starting from the progress corresponding to the collection progress information, based on the metadata file.

5. A log collector, characterized in that, include: The application discovery module is used to scan the log directory according to a preset hierarchical directory structure, wherein the preset hierarchical directory structure includes the application name, environment identifier, node IP, and log file name. The configuration management module, connected to the application discovery module, is used to manage application configurations; An application management module, connected to the configuration management module, is used to obtain application configuration from the configuration management module; The log collection module is connected to the application management module and is used to collect application logs according to the application configuration.

6. The log collector according to claim 5, characterized in that, The log collector also includes: The publishing module, connected to the application management module and the log collection module, is used to create a Kafka client according to the application configuration and send the application logs to the Kafka client.

7. The log collector according to claim 5, characterized in that, The log collector also includes: The status management module, connected to the log collection module, is used to record the collection progress information of the application logs and persistently store the collection progress information.

8. A computer program product, characterized in that, It includes computer program instructions, which are read and executed by a processor to perform the log collection method as described in any one of claims 1-4.

9. An electronic device, characterized in that, include: Processor, memory, and bus; The processor and the memory communicate with each other via the bus; The memory stores computer program instructions that can be executed by the processor, and the processor can execute the log collection method as described in any one of claims 1-4 by calling the computer program instructions.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when executed by a computer, cause the computer to perform the log collection method as described in any one of claims 1-4.

Images