Cluster log collection method and device, computer equipment and storage medium

Abstract

The application relates to the technical field of log monitoring, and discloses a cluster log collection method and device, computer equipment and a storage medium, the method of which comprises the following steps: receiving configuration option information, generating a log collection rule according to the configuration option information; analyzing the log collection rule to obtain log collection configuration parameters, wherein the log collection configuration parameters comprise area parameters, cluster parameters, container parameters and file path parameters; determining a target container node according to the area parameters, the cluster parameters and the container parameters, generating a log collector configuration file according to the file path parameters; and running a log collector in the target container node according to the log collector configuration file, so as to collect target logs through the log collector. The cluster log collection method of the application saves time and effort for the configuration of the log collector, reduces the difficulty of managing cluster logs, improves the log collection efficiency, and enhances the user experience.

Description

Technical Field

[0001] This invention relates to the field of log monitoring technology, and in particular to a cluster log collection method, apparatus, computer equipment, and storage medium. Background Technology

[0002] Kubernetes clusters, or k8s clusters for short, are open-source systems for automatically deploying, scaling, and managing containerized applications. Financial institutions (such as banks) and financial services companies can leverage Kubernetes clusters to manage the underlying data of financial microservices and rapidly deploy and scale services. Digital healthcare platform companies can use Kubernetes clusters to deploy data processing applications and manage large-scale analytical data. Log service is a key component of Kubernetes cluster operations; cluster logs record detailed access requests and error information, playing a crucial role in business analysis and problem localization. As a distributed cluster system managing a large number of containers, log collection is the first step in implementing log services, involving the container logs of every container in the cluster.

[0003] Existing Kubernetes cluster systems typically deploy Filebeat on nodes to collect container logs. Filebeat is a lightweight delivery tool; by installing Filebeat as a proxy on a server, it can monitor specified log files or locations, collect log events, and forward log data. However, when using Filebeat to collect logs, the configuration process is time-consuming and laborious. Because Filebeat cannot know the relationship between container log files and containers, and because the log data is scattered and large in volume, business users need to write complex parameter configuration files to manage the logs of each container in the cluster. This negatively impacts the accuracy and efficiency of log collection. Summary of the Invention

[0004] Therefore, it is necessary to provide a cluster log collection method, device, computer equipment, and storage medium to address the aforementioned technical problems, thereby solving the issues of time-consuming and laborious configuration, poor accuracy, and low efficiency in cluster log collection.

[0005] A cluster log collection method, comprising:

[0006] Receive configuration option information and generate log collection rules based on the configuration option information;

[0007] The log collection rules are parsed to obtain log collection configuration parameters, which include region parameters, cluster parameters, container parameters, and file path parameters.

[0008] The target container node is determined based on the region parameters, cluster parameters, and container parameters, and the log collector configuration file is generated based on the file path parameters.

[0009] The log collector is run on the target container node according to the log collector configuration file to collect target logs.

[0010] A cluster log collection device, comprising:

[0011] The data collection rule generation module is used to receive configuration option information and generate log collection rules based on the configuration option information.

[0012] The configuration parameter acquisition module is used to parse the log collection rules and obtain log collection configuration parameters, which include region parameters, cluster parameters, container parameters and file path parameters.

[0013] The log collector configuration file generation module is used to determine the target container node based on the region parameters, cluster parameters, and container parameters, and to generate the log collector configuration file based on the file path parameters.

[0014] The log collector running module is used to run the log collector on the target container node according to the log collector configuration file, so as to collect target logs through the log collector.

[0015] A computer device includes a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, wherein the processor implements the above-described cluster log collection method when executing the computer-readable instructions.

[0016] A computer-readable storage medium storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the cluster log collection method described above.

[0017] In the aforementioned cluster log collection method, apparatus, computer device, and storage medium, the method receives configuration option information and generates log collection rules based on the configuration option information; it parses the log collection rules to obtain log collection configuration parameters, including region parameters, cluster parameters, container parameters, and file path parameters; it determines the target container node based on the region parameters, cluster parameters, and container parameters, and generates a log collector configuration file based on the file path parameters; it runs the log collector on the target container node according to the log collector configuration file to collect target logs. The cluster log collection method of this invention configures log collection parameters and forms log collection rules based on options on a visual page, making configuration time-saving and labor-saving, and reducing the difficulty of managing cluster logs. At the same time, based on the log collection rules, the log collector can run more efficiently and accurately, improving log collection efficiency and enhancing the user experience. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram of an application environment for a cluster log collection method according to an embodiment of the present invention;

[0020] Figure 2 This is a flowchart illustrating a cluster log collection method according to an embodiment of the present invention;

[0021] Figure 3 This is a schematic diagram of a cluster log collection device according to an embodiment of the present invention;

[0022] Figure 4 This is a schematic diagram of a computer device according to an embodiment of the present invention. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] The cluster log collection method provided in this embodiment can be applied to, for example, Figure 1In this application environment, the client communicates with the server. Clients include, but are not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be a standalone server or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0025] The cluster log collection method provided in this embodiment is applied to a Kubernetes cluster system. Kubernetes clusters provide a mechanism for deploying, planning, updating, and maintaining applications across multiple hosts on a cloud platform. A Kubernetes cluster is a distributed system that facilitates the deployment and management of microservice architecture applications by forming an abstraction layer on top of the cluster. Financial institutions (such as banks) and financial services companies can leverage Kubernetes clusters to manage the underlying data of microservices, including applications, databases, and message queues, enabling rapid deployment and scaling of services to ensure high availability and reliability of financial services. Furthermore, digital healthcare platform companies can utilize Kubernetes clusters to quickly deploy data processing applications and manage large-scale analytical results data, thereby improving the efficiency and accuracy of data analysis in digital healthcare services.

[0026] In one embodiment, such as Figure 2 As shown, a cluster log collection method is provided, which is then applied to... Figure 1 Taking the server side as an example, the explanation includes the following steps S10-S40.

[0027] S10. Receive configuration option information and generate log collection rules based on the configuration option information.

[0028] Understandably, a Kubernetes cluster is a distributed system involving logging services for multiple applications. Users will have different log collection needs in different business scenarios. Configuration options are settings related to the log collector that users input through the client according to their log collection needs. These options include multiple levels, such as region-level options, cluster-level options, container-level options, and file-level options. The client sends the configuration options to the server, which then generates log collection rules based on these options. Log collection rules are a series of rule parameters corresponding to the configuration options at each level, used to configure the log collector to meet the user's log collection requirements.

[0029] S20. Parse the log collection rules to obtain log collection configuration parameters, which include region parameters, cluster parameters, container parameters, and file path parameters.

[0030] In a Kubernetes cluster system, a region refers to the geographical location of a physical data center. A region can include multiple clusters. To achieve the highest access performance, the cluster needs to be located in the same region as the cloud server instances. For example, if the Kubernetes cluster's data center is located in Guangdong, the corresponding region is the South China region. A cluster is a collection of nodes used to deploy and manage applications. A cluster typically consists of one or more master nodes and multiple worker nodes, providing high availability and fault tolerance through the collaborative work of these nodes. In a Kubernetes cluster, a Pod is the logical host for a container, providing the network and storage environment for the container. A Pod can contain one or more containers that can share network and storage resources and communicate with each other. A container is the basic unit of execution running within a Pod, and containers are created and managed by container engines such as Docker. A node is a worker node in a Kubernetes cluster, typically a virtual machine or a physical machine. Each node runs a container engine (such as Docker) used to create and manage containers within a Pod. A file path is the location information within the file directory corresponding to the worker node, used to locate files within that directory. The server parses the log collection rules to obtain log collection configuration parameters at various levels, including region-level parameters, cluster-level parameters, container-level parameters, and file-level parameters.

[0031] S30. Determine the target container node based on the region parameters, cluster parameters, and container parameters, and generate the log collector configuration file based on the file path parameters.

[0032] In essence, a log collector is a tool used to collect and transport log files. In a Kubernetes cluster, the log collector refers to Filebeat. Filebeat is installed on the server as a log broker, monitoring log files or locations under a specified file path, collecting log events, and forwarding them to Elasticsearch or Logstash for indexing. The server determines the container node containing the corresponding container in the corresponding cluster within the corresponding region based on the parsed region parameters, cluster parameters, and container parameters. The server also determines the file location under the corresponding container node based on the parsed file path parameters, generates a Filebeat configuration file for the corresponding log path, and uploads the Filebeat configuration file to the Filebeat configuration directory on the container node. In this embodiment, the server parses the log collection rules. The parsed regional parameter is "South China", the cluster parameter is "test", and the container parameter is "nginx". The server queries the database to find the two nodes where the nginx container is located in the test cluster in the South China region, namely node A and node B. The server generates a Filebeat configuration file according to the file path parameter " / data / logs / access.log" and uploads the Filebeat configuration file to the Filebeat configuration directory " / etc / filebeat / config / " on nodes A and B.

[0033] S40. Run the log collector on the target container node according to the log collector configuration file to collect target logs through the log collector.

[0034] Understandably, when the server calls the Filebeat configuration file on the target container node, Filebeat is activated and run. Filebeat then starts one or more inputs that will search for log file data at the specified file path. For each log file found by Filebeat, Filebeat starts a collector, each collector reading a single log file and sending the log data to the data forwarding library libbeat. libbeat aggregates the log data and sends the aggregated log data to Filebeat's pre-configured output, such as a log service platform.

[0035] This embodiment receives configuration option information and generates log collection rules based on it. The log collection rules are parsed to obtain log collection configuration parameters, including region parameters, cluster parameters, container parameters, and file path parameters. The target container node is determined based on the region, cluster, and container parameters, and a log collector configuration file is generated based on the file path parameters. The log collector is then run on the target container node according to the configuration file to collect target logs. This embodiment's cluster log collection method uses a visual interface to configure log collection parameters and generate log collection rules, making configuration time-saving and labor-saving, and reducing the difficulty of managing cluster logs. Simultaneously, based on the log collection rules, the log collector can run more efficiently and accurately, improving log collection efficiency and enhancing the user experience.

[0036] In one embodiment, step S10, namely receiving configuration option information and generating log collection rules based on the configuration option information, includes:

[0037] S101. Receive a log collection configuration request and display the collection rule configuration page according to the log collection configuration request;

[0038] S102. Receive configuration selection instructions through the collection rule configuration page. The configuration selection instructions include region selection instructions, cluster selection instructions, container selection instructions, and file path selection instructions.

[0039] S103. Determine region option information according to the region selection instruction, determine cluster option information according to the cluster selection instruction, determine container option information according to the container selection instruction, and determine file path option information according to the file path selection instruction;

[0040] S104. Generate the log collection rules based on the region option information, cluster option information, container option information, and file path option information.

[0041] Understandably, a log collection configuration request is a request triggered by a user through a client-side page to configure the log collector. The server receives the log collection configuration request, interacts with the client based on the request, and displays the collection rule configuration page on the client. The collection rule configuration page displays all Kubernetes clusters in the current Kubernetes cluster system, the region of each cluster, the containers under each cluster, and the file paths of file directories in each container node, arranged at different levels. Users can select options on the client's collection rule configuration page according to their log collection needs. The client sends corresponding region selection instructions, cluster selection instructions, container selection instructions, and file path selection instructions to the server. The server determines the region option information based on the region selection instructions, the cluster option information based on the cluster selection instructions, the container option information based on the container selection instructions, and the file path option information based on the file path selection instructions. The server merges the region option information, cluster option information, container option information, and file path option information to generate log collection rules. For example, a user selects rules through the rule configuration page. The region selection command is set to "South China," the container selection command to "test," and the file path selection command to "nginx." After selecting the file path, the user clicks the "Create" button, generating a log collection rule named "spring-boot." Here, " / data / logs / access.log" indicates that access logging is enabled, specifying the log file's storage path, format, and cache size.

[0042] In this embodiment, when configuring the log collector, users do not need to fully understand the underlying mechanisms of the log collector, nor do they need to learn how to write complex log collector configuration files, nor do they need to be familiar with cluster log mounting mechanisms. Instead, a visual web page allows users to clearly see the configuration options at each level. Based on log collection needs, users select the corresponding options and generate log collection rules by clicking. This embodiment achieves more efficient and accurate log collector configuration, greatly reducing the difficulty of managing cluster logs, saving users time, and improving the user experience.

[0043] In one embodiment, the data collection rule configuration page includes a regional cluster configuration page, a container configuration page, and a file path configuration page;

[0044] In step S102, namely receiving the configuration selection instruction through the collection rule configuration page, the following is included:

[0045] S1021. Receive region selection instructions and cluster selection instructions through the region cluster configuration page, determine candidate containers according to the region selection instructions and cluster selection instructions, and generate a container configuration page corresponding to the candidate containers.

[0046] S1022. Receive a container selection instruction through the container configuration page, determine the candidate file path according to the container selection instruction, and generate a file path configuration page corresponding to the candidate file path.

[0047] S1023. Receive a file path selection instruction through the file path configuration page.

[0048] Understandably, the log collection rule configuration page includes a regional cluster configuration page, a container configuration page, and a file path configuration page, corresponding to regional and cluster-level configuration options, container-level configuration options, and file-level configuration options, respectively. Upon receiving a log collection configuration request, the first page displayed is the regional cluster configuration page. Based on the user's selection of the regional cluster configuration page, the client sends a regional selection command and a cluster selection command to the server. The server determines all candidate containers under the corresponding cluster in the corresponding region based on these commands, generating a container configuration page on the client corresponding to each candidate container, allowing the user to select a specified container as the scope of log collection. Similarly, based on the user's selection of the container configuration page, the client sends a container selection command to the server. The server determines all candidate file paths under the container node containing the corresponding container, generating a file path configuration page on the client corresponding to each candidate file path, allowing the user to select a specified file path. For example, "access.log" can specify the log file path, cache size, and log file name. The client sends a file path selection instruction to the server based on the user's selection operation on the file path configuration page, and the server receives the file path selection instruction.

[0049] This embodiment determines the candidate options for the next level by using the selection instructions of the previous level based on the architectural relationship between the levels, which simplifies the number of candidate options and avoids the phenomenon of misselection due to mismatch between the previous and next levels, and is conducive to the quick and accurate creation of log collection rules.

[0050] In one embodiment, step S30, namely determining the target container node based on the region parameters, cluster parameters, and container parameters, includes:

[0051] S301. Determine the target container query conditions based on the region parameters, cluster parameters, and container parameters;

[0052] S302. Query the database according to the target container query conditions to determine the target container node corresponding to the target container query conditions.

[0053] Understandably, the target container query conditions are used to query the database for containers corresponding to log collection requirements, including region parameters, cluster parameters, and container parameters. These parameters can be the region name, cluster name, and container name, respectively. The target container node is the node where the container corresponding to the target container query conditions resides. The database stores information about the nodes where each container in each Kubernetes cluster resides, i.e., the association between containers and nodes. Since there are multiple worker nodes in a Kubernetes cluster, and each node runs a container engine (e.g., Docker) for a container, it is necessary to query the database based on the target container query conditions to determine the target container node corresponding to those conditions. In one embodiment, the server resolves the region name to "South China," the cluster name to "test," and the container name to "nginx." Querying the database reveals that the nginx container in the "test" cluster in the South China region is distributed across two nodes; therefore, these two nodes, A and B, are the target container nodes.

[0054] This embodiment determines the target container query conditions based on region parameters, cluster parameters, and container parameters, and queries the database to determine the corresponding target container node. This can accurately determine the node where the container is located, saving time.

[0055] In one embodiment, step S10, namely generating the log collector configuration file based on the file path parameter, includes:

[0056] S303. Obtain the initial configuration file for the log collector;

[0057] S304. Update the initial configuration file of the log collector according to the file path parameter to generate the log collector configuration file;

[0058] S305. Upload the log collector configuration file to the log collector configuration directory of the target container node to run the log collector on the target container node.

[0059] Understandably, the initial log collector configuration file contains general log collector configuration information, such as basic functional information and default settings, but does not include actual application settings, such as file path information. The server retrieves the initial log collector configuration file, adds the file path parameter to it, and updates it to generate the final log collector configuration file. A log collector configuration directory is created on each container node for deploying the log collector on that container node. After the log collector configuration file is generated, the server uploads it to the target container node's log collector configuration directory based on the determined target container node.

[0060] This embodiment improves the configuration efficiency of the log collector by updating the initial configuration file of the log collector by adding the file path parameter.

[0061] In one embodiment, step S40, namely collecting the target logs through the log collector, includes:

[0062] S401. The log collector queries the file directory of the target container node to determine the target path file corresponding to the file path parameter;

[0063] S402. Obtain the target log in the target path file through the log collector, and return the target log to the log service platform.

[0064] Understandably, to facilitate file access and management, the Kubernetes cluster system establishes a file index, or file directory, which represents the mapping relationship between filenames and file physical locations. Each file directory has an entry, which must at least include the filename, file internal identifier, file type, file storage address, file length, access permissions, creation time, and access time. The target path file is the file data in the target container node located at the specified file path corresponding to the file path parameter. The target log is the log file data within the target path file. The server queries the file directory of the target container node using a log collector to determine the target path file, collects the target log from the target path file, and returns the target log to the log service platform. The log service platform assists users in solving problems in business operations, service monitoring, and log auditing scenarios through logs, providing services such as log retrieval, chart analysis, monitoring and alerting, and log delivery. In practical applications, specified log data scattered across various nodes in the cluster is collected and managed uniformly on the log service platform. Keyword searches can quickly retrieve logs of abnormal events, locate problematic nodes, and trigger alerts.

[0065] This embodiment uses a log collector to query the file directory of the target container node, obtain all current target logs in the target path file, and return the current target logs to the log service platform to ensure the accuracy and completeness of log collection.

[0066] In one embodiment, step S402, after obtaining the target log in the target path file through the log collector, further includes:

[0067] S4021. The log collector monitors the target path file according to a preset time period to determine whether there are any new target logs in the target path file;

[0068] S4022. If there is a new target log, the new target log is obtained through the log collector and returned to the log service platform.

[0069] Understandably, the preset time period is a pre-set monitoring period for the log collector. For example, the system default is 1 minute, but it can be modified to 30 seconds according to log collection needs, or it can be set to continuous. New target logs are log file data newly generated within the current time period in the target path file. In this embodiment, the server monitors and scans the target path file at a frequency of 1 minute using the log collector. Based on the scan results, it determines whether new target logs exist in the target path file. When new target logs are found, they are collected and returned to the log service platform.

[0070] This embodiment uses a log collector to continuously monitor the target path file according to a preset time period, grasp the log update dynamics, and collect newly added log file data in a timely manner to ensure the timeliness of log collection.

[0071] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0072] In one embodiment, a cluster log collection device is provided, which corresponds one-to-one with the cluster log collection method described in the above embodiments. For example... Figure 3 As shown, the cluster log collection device includes a collection rule generation module 10, a collection configuration parameter acquisition module 20, a collector configuration file generation module 30, and a log collector execution module 40. Detailed descriptions of each functional module are as follows:

[0073] The log collection rule generation module 10 is used to receive configuration option information and generate log collection rules based on the configuration option information.

[0074] The configuration parameter acquisition module 20 is used to parse the log collection rules and obtain log collection configuration parameters, which include region parameters, cluster parameters, container parameters and file path parameters.

[0075] The log collector configuration file generation module 30 is used to determine the target container node based on the region parameters, cluster parameters and container parameters, and to generate the log collector configuration file based on the file path parameters.

[0076] The log collector running module 40 is used to run the log collector on the target container node according to the log collector configuration file, so as to collect target logs through the log collector.

[0077] In one embodiment, the data collection rule generation module 10 includes:

[0078] The data collection rule configuration page display unit is used to receive log collection configuration requests and display the data collection rule configuration page according to the log collection configuration requests.

[0079] A configuration selection instruction receiving unit is used to receive configuration selection instructions through the collection rule configuration page. The configuration selection instructions include region selection instructions, cluster selection instructions, container selection instructions, and file path selection instructions.

[0080] The option information determining unit is used to determine region option information according to the region selection instruction, cluster option information according to the cluster selection instruction, container option information according to the container selection instruction, and file path option information according to the file path selection instruction;

[0081] The log collection rule generation unit is used to generate the log collection rules based on the region option information, cluster option information, container option information and file path option information.

[0082] In one embodiment, the data collection rule generation module 10 further includes:

[0083] The container configuration page generation unit is used to receive region selection instructions and cluster selection instructions through the region cluster configuration page, determine the candidate container according to the region selection instructions and cluster selection instructions, and generate a container configuration page corresponding to the candidate container.

[0084] The file path configuration page generation unit is used to receive a container selection instruction through the container configuration page, determine the candidate file path according to the container selection instruction, and generate a file path configuration page corresponding to the candidate file path.

[0085] The file path selection instruction receiving unit is used to receive file path selection instructions through the file path configuration page.

[0086] In one embodiment, the collector configuration file generation module 30 includes:

[0087] The target container query condition determination unit is used to determine the target container query conditions based on the region parameters, cluster parameters, and container parameters.

[0088] The target container node determination unit is used to query the database according to the target container query conditions and determine the target container node corresponding to the target container query conditions.

[0089] In one embodiment, the collector configuration file generation module 30 further includes:

[0090] The initial configuration file acquisition unit is used to acquire the initial configuration file of the log collector;

[0091] The log collector configuration file generation unit is used to update the initial configuration file of the log collector according to the file path parameter and generate the log collector configuration file.

[0092] The log collector configuration file upload unit is used to upload the log collector configuration file to the log collector configuration directory of the target container node, so as to run the log collector on the target container node.

[0093] In one embodiment, the log collector operation module 40 includes:

[0094] The target path file determination unit is used to query the file directory of the target container node through the log collector to determine the target path file corresponding to the file path parameter;

[0095] The target log acquisition unit is used to acquire the target log in the target path file through the log collector and return the target log to the log service platform.

[0096] In one embodiment, the log collector operation module 40 further includes:

[0097] The target path file monitoring unit is used to monitor the target path file through the log collector according to a preset time period to determine whether there are any new target logs in the target path file;

[0098] A new target log acquisition unit is added, which is used to acquire the new target log through the log collector if there is a new target log, and return the new target log to the log service platform.

[0099] Specific limitations regarding the cluster log collection device can be found in the limitations of the cluster log collection method described above, and will not be repeated here. Each module in the aforementioned cluster log collection device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0100] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, network interface, display screen, and input device connected via a system bus. The processor provides computing and control capabilities. The memory includes a readable storage medium and internal memory. The non-volatile storage medium stores the operating system and computer-readable instructions. The internal memory provides an environment for the operation of the operating system and computer-readable instructions in the readable storage medium. The network interface is used to communicate with an external server via a network connection. When the computer-readable instructions are executed by the processor, a cluster log collection method is implemented. The readable storage medium provided in this embodiment includes both non-volatile and volatile readable storage media.

[0101] In one embodiment, a computer device is provided, including a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, wherein the processor performs the following steps when executing the computer-readable instructions:

[0102] Receive configuration option information and generate log collection rules based on the configuration option information;

[0103] The log collection rules are parsed to obtain log collection configuration parameters, which include region parameters, cluster parameters, container parameters, and file path parameters.

[0104] The target container node is determined based on the region parameters, cluster parameters, and container parameters, and the log collector configuration file is generated based on the file path parameters.

[0105] The log collector is run on the target container node according to the log collector configuration file to collect target logs.

[0106] In one embodiment, one or more computer-readable storage media storing computer-readable instructions are provided. The readable storage media provided in this embodiment include non-volatile readable storage media and volatile readable storage media. The readable storage media stores computer-readable instructions, which, when executed by one or more processors, perform the following steps:

[0107] Receive configuration option information and generate log collection rules based on the configuration option information;

[0108] The log collection rules are parsed to obtain log collection configuration parameters, which include region parameters, cluster parameters, container parameters, and file path parameters.

[0109] The target container node is determined based on the region parameters, cluster parameters, and container parameters, and the log collector configuration file is generated based on the file path parameters.

[0110] The log collector is run on the target container node according to the log collector configuration file to collect target logs.

[0111] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware through computer-readable instructions. These computer-readable instructions can be stored in a non-volatile readable storage medium or a volatile readable storage medium. When executed, these computer-readable instructions can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0112] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0113] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A cluster log collection method, characterized by, include: Receive configuration option information and generate log collection rules based on the configuration option information; The log collection rules are parsed to obtain log collection configuration parameters, which include region parameters, cluster parameters, container parameters, and file path parameters. The target container node is determined based on the region parameters, cluster parameters, and container parameters, and the log collector configuration file is generated based on the file path parameters. The log collector is run on the target container node according to the log collector configuration file to collect target logs.

2. The cluster log collection method as described in claim 1, characterized in that, The process of receiving configuration option information and generating log collection rules based on the configuration option information includes: Receive a log collection configuration request and display the collection rule configuration page according to the log collection configuration request; The configuration selection instructions are received through the collection rule configuration page. These instructions include region selection instructions, cluster selection instructions, container selection instructions, and file path selection instructions. The region option information is determined according to the region selection instruction, the cluster option information is determined according to the cluster selection instruction, the container option information is determined according to the container selection instruction, and the file path option information is determined according to the file path selection instruction. The log collection rules are generated based on the region option information, cluster option information, container option information, and file path option information.

3. The cluster log collection method as described in claim 2, characterized in that, The data collection rule configuration page includes a regional cluster configuration page, a container configuration page, and a file path configuration page; Receiving configuration selection instructions through the collection rule configuration page includes: The system receives region selection instructions and cluster selection instructions through the region cluster configuration page, determines candidate containers based on the region selection instructions and cluster selection instructions, and generates a container configuration page corresponding to the candidate containers. The container configuration page receives a container selection instruction, determines the candidate file path based on the container selection instruction, and generates a file path configuration page corresponding to the candidate file path. The file path selection instruction is received through the file path configuration page.

4. The cluster log collection method as described in claim 1, characterized in that, The step of determining the target container node based on the region parameters, cluster parameters, and container parameters includes: The target container query conditions are determined based on the region parameters, cluster parameters, and container parameters. The database is queried according to the target container query conditions to determine the target container node corresponding to the target container query conditions.

5. The cluster log collection method as described in claim 1, characterized in that, The step of generating the log collector configuration file based on the file path parameter includes: Obtain the initial configuration file for the log collector; The initial configuration file of the log collector is updated according to the file path parameter to generate the log collector configuration file; Upload the log collector configuration file to the log collector configuration directory of the target container node to run the log collector on the target container node.

6. The cluster log collection method as described in claim 1, characterized in that, The process of collecting target logs through the log collector includes: The log collector queries the file directory of the target container node to determine the target path file corresponding to the file path parameter; The log collector obtains the target log from the target path file and returns the target log to the log service platform.

7. The cluster log collection method as described in claim 6, characterized in that, After obtaining the target logs from the target path file through the log collector, the process further includes: The log collector monitors the target path file according to a preset time period to determine whether there are any new target logs in the target path file; If a new target log is added, the new target log is obtained through the log collector and returned to the log service platform.

8. A cluster log collection device, characterized in that, include: The data collection rule generation module is used to receive configuration option information and generate log collection rules based on the configuration option information. The configuration parameter acquisition module is used to parse the log collection rules and obtain log collection configuration parameters, which include region parameters, cluster parameters, container parameters and file path parameters. The log collector configuration file generation module is used to determine the target container node based on the region parameters, cluster parameters, and container parameters, and to generate the log collector configuration file based on the file path parameters. The log collector running module is used to run the log collector on the target container node according to the log collector configuration file, so as to collect target logs through the log collector.

9. A computer device comprising a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, characterized in that, When the processor executes the computer-readable instructions, it implements the cluster log collection method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing computer-readable instructions, characterized in that, When the computer-readable instructions are executed by one or more processors, the one or more processors cause the cluster log collection method as described in any one of claims 1 to 7 to be performed.

Images