Dynamic log collection method and system

Abstract

The invention discloses a dynamic log collection method and system and a server. According to the method, a flexible strategy configuration management system is introduced, user-defined configuration is supported, a plurality of target servers, different log files and different collection frequencies can be configured in one log collection strategy, and flexible collection strategies can be configured for different collection objects; in addition, in the method, strategies are automatically distributed, and the complexity of deployment and management is simplified; after the change of the file is monitored, the log file in the appointed path with the change is automatically collected, the automatic log collection and storage process reduces the requirement of manual operation, reduces the possibility of error occurrence, and improves the integrity and reliability of log data; in addition, the method supports the simultaneous storage of the logs in a plurality of target storage systems, such as a local file system, a cloud storage, a database and the like, so that the flexible configuration of log storage is realized.

Description

Technical Field

[0001] The present disclosure generally relates to the field of electronic technology, and in particular to a dynamic log collection method, system and server. Background Art

[0002] Task execution logs are an integral part of the big data platform. They can not only be used to troubleshoot and optimize the performance of the big data platform, but also provide important audit and security functions to ensure the stability and security of the system.

[0003] At present, the collection, parsing and analysis of task execution logs on big data platforms are usually achieved by using some third-party tools, such as calling Fluentd (a data collector) to collect logs from each node and then sending them to Kafka (a distributed stream processing platform for real-time aggregation), calling Grok (a tool for parsing and structuring unstructured log data) for log parsing, and storing the parsed data in Elasticsearch (a distributed search and analysis engine based on the Lucene library), calling Kibana (an open source data analysis and visualization platform (build dashboards, monitor task execution status in real time), calling Spark for batch log analysis, calculating task execution time and failure rate, identifying system bottlenecks, etc.

[0004] Although the call of third-party tools can only meet basic functions and has limitations in meeting customization and special requirements, over-reliance on third-party tools may affect system stability. Once these tools have problems or stop maintenance, it will cause serious problems in system operation. Summary of the invention

[0005] In view of the above-mentioned defects or deficiencies in the prior art, it is desirable to provide a dynamic log collection method, system, server, computer-readable storage medium, and computer program product that have both flexible scalability and high stability.

[0006] In a first aspect, an embodiment of the present application provides a dynamic log collection method based on a log collection server, including:

[0007] The log collection server obtains the user's configuration information for log collection; wherein the configuration information includes a log collection strategy; the log collection strategy includes: a strategy name, one or more target servers, one or more log file paths, a collection frequency corresponding to each of the log file paths, a strategy state, and log filtering and conversion rules;

[0008] Storing the configuration information and adding it to the policy configuration table of the database;

[0009] Scan the policy configuration table in the database to determine each log collection policy to be distributed and the corresponding one or more target servers;

[0010] Each log collection strategy is distributed to a log collection client in a corresponding target server, so that the log collection client collects logs of files under each log file path and stores them in one or more designated storage systems.

[0011] In one embodiment, after distributing each log collection strategy to the log collection client in the corresponding target server, the method further includes:

[0012] Querying the log collection client for system operation data of the target server;

[0013] Analyze the returned system operation data to determine the system load status;

[0014] Dynamically adjusting the log collection strategy corresponding to the target server according to the system load status to obtain an updated log collection strategy;

[0015] The updated log collection policy is synchronously updated to the log collection client.

[0016] In one embodiment, the log collection server obtains the user's configuration information for log collection, including:

[0017] The log collection server receives the custom configuration information input by the user through the interface or API to create or modify the log collection policy;

[0018] Review whether the custom configuration information meets the preset requirements;

[0019] If satisfied, the customized configuration information is used as the configuration information.

[0020] In one embodiment, before distributing each log collection policy to the log collection client in the corresponding target server, the method further includes:

[0021] Determine the health status of the target server and judge whether it is in an available state;

[0022] If it is in an unavailable state, an alarm message indicating that the server is unavailable is generated;

[0023] If it is in an available state, execute the step of distributing each log collection policy to the log collection client in the corresponding target server.

[0024] In one embodiment, before distributing each log collection policy to the log collection client in the corresponding target server, the method further includes:

[0025] Determine the number of log collection policies to be distributed and whether the load threshold is exceeded;

[0026] If the load threshold is not exceeded, executing the step of distributing each log collection strategy to the log collection client in the corresponding target server;

[0027] If the load threshold is exceeded, the log collection strategies to be distributed are grouped to obtain a number of strategy groups;

[0028] Accordingly, the distributing of each log collection strategy to the log collection client in the corresponding target server specifically includes: sending each of the strategy groups to the log collection client in the corresponding target server in batches in sequence.

[0029] In one embodiment, after distributing each log collection strategy to the log collection client in the corresponding target server, the method further includes:

[0030] Monitor and track real-time distribution status;

[0031] The real-time distribution status is updated to the policy configuration table in real time.

[0032] In one embodiment, after the real-time distribution status is updated to the policy configuration table in real time, the method further includes:

[0033] Determine whether the log collection policy is sent successfully according to the real-time distribution status in the policy configuration table;

[0034] If the sending fails, the step of distributing each log collection strategy to the log collection client in the corresponding target server is executed after a specified interval according to the retry strategy;

[0035] Determine the number of retry attempts and whether the retry threshold has been reached;

[0036] If the retry times threshold is reached, an alarm message indicating a sending failure is generated.

[0037] In one embodiment, the configuration information further includes: policy extension information, and the policy extension information includes: policy version, policy description, policy triggering condition, and policy dependency.

[0038] In a second aspect, an embodiment of the present application provides a dynamic log collection method based on a log collection client, comprising:

[0039] The log collection client receives the log collection policy issued by the log collection server; wherein the log collection policy includes: a policy name, one or more log file paths, a collection frequency corresponding to each of the log file paths, a policy status, and log filtering and conversion rules;

[0040] Monitor the files under each of the log file paths to identify whether there are any file changes;

[0041] If a file change is detected, log collection is triggered for the changed target file;

[0042] The collected logs are stored in one or more specified storage systems according to the storage policy.

[0043] In one embodiment, after the log collection client receives the log collection policy sent by the log collection server, the method further includes: caching the log collection policy to a local file.

[0044] In one embodiment, triggering log collection for a changed target file includes:

[0045] Determine whether the space occupied by the target file reaches a large file standard;

[0046] If the large file standard is met, determining that the target file is a large file;

[0047] Slice the logs of large files;

[0048] Collect logs from each shard in parallel.

[0049] In one embodiment, after triggering log collection for the target file that has changed, the method further includes:

[0050] If an abnormality is detected in the target file, a file abnormality mark is added to the collected log.

[0051] In one embodiment, after storing the collected logs in one or more designated storage systems according to the storage policy, the method further includes:

[0052] The logs in each of the storage systems are managed for their lifecycles according to the configured management policies.

[0053] In one embodiment, storing the collected logs in one or more designated storage systems according to the storage policy includes:

[0054] Determine the types of logs collected;

[0055] Matching a corresponding storage system according to the type as the target storage system;

[0056] The collected logs are stored in a target storage system.

[0057] In a third aspect, an embodiment of the present application provides a dynamic log collection system, including:

[0058] A log collection server, used to obtain user configuration information for log collection; store the configuration information and add it to a policy configuration table in a database; scan the policy configuration table in the database to determine each log collection policy to be distributed and the corresponding one or more target servers; distribute each log collection policy to a log collection client in the corresponding target server; wherein the configuration information includes a log collection policy; the log collection policy includes: a policy name, one or more target servers, one or more log file paths, a collection frequency corresponding to each of the log file paths, a policy status, and log filtering and conversion rules;

[0059] The log collection client is used to receive the log collection policy issued by the log collection server; monitor the files under each log file path to identify whether there are file changes; if file changes are detected, trigger log collection for the changed target files; store the collected logs in one or more specified storage systems according to the storage policy.

[0060] In a fourth aspect, an embodiment of the present application provides a server, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method described in the embodiment of the present application when executing the program.

[0061] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the steps of the method described in the embodiment of the present application are implemented.

[0062] In a sixth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the method described in the embodiment of the present application.

[0063] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0065] Figure 1 A signaling diagram of a multi-model ensemble learning method provided in an embodiment of the present application is shown;

[0066] Figure 2 A strategy dynamic update signaling diagram provided by an embodiment of the present application is shown;

[0067] Figure 3An exemplary structural block diagram of a multi-model integrated learning system provided in an embodiment of the present application is shown;

[0068] Figure 4 A schematic diagram of the structure of a computer system suitable for implementing a server of an embodiment of the present application is shown. DETAILED DESCRIPTION

[0069] The present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It is also necessary to explain that, for ease of description, only the parts related to the invention are shown in the accompanying drawings.

[0070] It should be noted that, in the absence of conflict, the embodiments in the present application and the features in the embodiments can be combined with each other. The present application will be described in detail below with reference to the drawings and in combination with the embodiments. Although the embodiments of the present application provide the method operation instruction steps shown in the following embodiments or drawings, more or fewer operation instruction steps may be included in the method based on routine or no creative labor. In the steps where there is no necessary causal relationship logically, the execution order of these steps is not limited to the execution order provided by the embodiments of the present application. The method may be executed in the order of the methods shown in the embodiments or drawings or in parallel during the actual processing process or when the device is executed.

[0071] Please refer to Figure 1 , Figure 1 The signaling diagram of the dynamic log collection method provided by an embodiment of the present application is shown, and the specific implementation process of the dynamic log collection method is introduced from both the log collection server and the log collection client. Figure 1 As shown, the method mainly includes:

[0072] S101, the log collection server obtains the user's configuration information for log collection;

[0073] The configuration information of log collection mainly includes log collection strategy and other auxiliary information. The log collection strategy specifically includes the following information items: policy name, one or more target servers, one or more log file paths, the collection frequency corresponding to each log file path, policy status, log filtering and conversion rules.

[0074] The strategy name is used to uniquely identify each log collection strategy. A naming convention is usually used, such as LogCollectStrategy_<environment>_<application>_<timestamp>.

[0075] The target server refers to the server object to be collected under this policy. The target server can be one or more. This method supports the configuration of log collection of multiple target servers under one policy to improve the flexibility and applicability of log collection. The information of the target server includes not only the server host name and IP, but also the role of the server (such as application server, database server, etc.) and the geographical location of the data center, so as to better manage log collection in a multi-data center environment.

[0076] The log file path refers to the specific storage location of the log files to be collected. It supports wildcards or regular expressions, and allows the specification of multiple log files or dynamically generated log files (such as logs generated by date). This method supports the configuration of multiple log paths under one policy to achieve the simultaneous collection of files under multiple log paths under one policy.

[0077] The collection frequency can correspond to the log file path one by one, and the corresponding collection frequency can be set for each log file path, further improving the flexibility of the log collection process. The collection frequency setting supports fine-grained time expressions, such as cron expressions, allowing flexible definition of collection time and frequency.

[0078] The policy status refers to the current status of the policy, such as "running", "pause", "error", etc. By recording the current status of the policy and its historical status changes, you can better monitor the policy execution.

[0079] This method also supports adding log filtering and (data) conversion rules in the policy configuration, such as filtering log content based on keywords, or parsing and structuring log fields to achieve refined log collection and processing.

[0080] Compared with the traditional static configuration method, this method introduces a flexible policy configuration management system, which supports policy creation, policy modification or policy deletion for log collection through an interface or API (Application Programming Interface), so as to realize customized and flexible configuration of log collection policies. It can configure multiple target servers, different log files and different collection frequencies for log collection, has strong scalability, and can support the configuration of flexible collection policies for different collection objects.

[0081] S102, the log collection server stores the configuration information and adds it to the policy configuration table of the database;

[0082] The log collection server stores these configuration information and adds them to the policy configuration table of the database. Subsequent policies are mainly managed by reading the policy configuration table. The policy configuration table stores some or all of the information items of all configuration information, which can be set according to management needs and is not limited here.

[0083] S103, the log collection server scans the policy configuration table in the database to determine each log collection policy to be distributed and the corresponding one or more target servers;

[0084] By scanning the policy configuration table in the database, the log collection policies in the to-be-distributed state are determined, and one or more target servers corresponding to each log collection policy to be distributed are extracted.

[0085] S104, the log collection server distributes each log collection strategy to the log collection client in the corresponding target server;

[0086] The above realizes automated policy distribution, simplifies the complexity of deployment and management, improves the maintainability and scalability of the system, and reduces the need for manual intervention.

[0087] S201, the log collection client receives the log collection policy sent by the log collection server;

[0088] The log collection client is pre-installed on each target server. Taking the dynamic log collection for YARN Resource Manager (YetAnother Resource Negotiator, a resource manager in the Apache Hadoop ecosystem) as an example, the log collection client is pre-distributed on the YARN resource manager server of the big data engine. The log collection client is a lightweight log collection client that supports small resource usage. The client has an automatic update mechanism and can always run the latest version.

[0089] The log collection client on the resource manager of each big data cluster can communicate with the log collection server through HTTP requests to dynamically obtain the log collection policy associated with the current instance. A reliable transmission protocol and breakpoint-resume transmission mechanism are introduced between the log collection server and the log collection client to ensure the security and stability of log data during transmission and storage, which is particularly suitable for log management needs in large-scale data environments.

[0090] S202, the log collection client monitors the files under each log file path and identifies whether there are file changes;

[0091] The log collection client monitors the files under each log file path and identifies whether there are file changes. The monitoring method is not limited in this embodiment. The client can achieve this by polling the file changes under the specified log path, or by using system mechanisms such as inotify (a function of the Linux kernel for monitoring file system events) to respond to file changes in real time to reduce the performance overhead caused by polling.

[0092] S203: If the log collection client detects a file change, log collection is triggered for the target file that has changed;

[0093] After the log collection client detects file changes, it automatically collects the log files in the specified path where the changes occurred according to the log path and file name configured in the collection policy. The automated log collection and storage process reduces the need for manual operations, reduces the possibility of errors, and improves the integrity and reliability of log data.

[0094] S204: The log collection client stores the collected logs in one or more designated storage systems according to the storage policy.

[0095] This method supports storing logs in multiple target storage systems at the same time, such as local file systems, cloud storage, databases, etc., to meet different storage requirements and achieve flexible configuration of log storage.

[0096] In one embodiment, in order to further improve the storage requirements of different types of logs, this step can be implemented according to the following sub-steps: (1) determining the type of collected logs; (2) matching the corresponding storage system according to the type as the target storage system; (3) storing the collected logs in the target storage system.

[0097] This method configures different storage strategies for different log types. For example, access logs are stored in HDFS (Hadoop Distributed File System) and error logs are stored in Elasticsearch (a distributed search and analysis engine based on the Lucene library). This can meet the flexible storage requirements of different types of logs.

[0098] It should be noted that although the operations of the method of the present invention are described in a particular order in the drawings, this does not require or imply that the operations must be performed in this particular order or that all illustrated operations must be performed to achieve desired results.

[0099] Based on the above introduction, the method provided in this embodiment introduces a flexible policy configuration management system, supports user-defined configuration, and can configure multiple target servers, different log files, and different collection frequencies in one log collection policy, which can support the configuration of flexible collection strategies for different collection objects; in addition, the automatic distribution of policies in this method simplifies the complexity of deployment and management; after monitoring file changes, the log files in the specified path where the changes occur are automatically collected, and the automated log collection and storage process reduces the need for manual operation, reduces the possibility of errors, and improves the integrity and reliability of log data; in addition, this method supports the simultaneous storage of logs in multiple target storage systems, such as local file systems, cloud storage, databases, etc., thereby realizing flexible configuration of log storage.

[0100] In one embodiment, after the log collection server automatically distributes the log collection policy to the corresponding log collection client, the policy can be further dynamically adjusted according to the system operation status, so that the log collection is always at the most appropriate frequency, such as Figure 2 The figure shows a signaling diagram for dynamic policy updates. After the log collection server automatically distributes the log collection policy to the corresponding log collection client, the specific steps are as follows:

[0101] Step S105: query the log collection client for system operation data of the target server;

[0102] Step S106: Analyze the log to collect the system operation data returned by the client and determine the system load status;

[0103] Step S107: dynamically adjust the log collection strategy corresponding to the target server according to the system load status to obtain an updated log collection strategy;

[0104] In this embodiment, a dynamic adjustment mechanism of the strategy is introduced to dynamically adjust the log collection strategy (such as execution frequency and priority) according to the current status of the system (such as server load, network bandwidth, disk usage, etc.). The dynamic adjustment of the strategy improves the flexibility and adaptability of the log collection system, thereby ensuring that the system can run efficiently under different conditions.

[0105] Step S108: Synchronously update the updated log collection policy to the log collection client.

[0106] The log collection client on the resource manager of each big data cluster communicates with the log collection server through HTTP requests, dynamically obtains and updates the collection policy associated with the current instance, and implements a real-time and dynamic policy update mechanism, ensuring that the system can respond to environmental and configuration changes in real time, and improving the accuracy and efficiency of log collection.

[0107] It should be noted that Figure 2 In the example, only the load query performed by the log collection client after a log collection is taken. In this embodiment, there is no limitation on the triggering of the load query. It can be triggered at a scheduled time or under conditions. In addition, after the log collection strategy is updated, the updated log collection strategy can be used immediately after the next log collection is triggered, thereby ensuring the fastest environment adaptation.

[0108] In one embodiment, in order to ensure that the user customizes the configuration information while avoiding the impact of incorrect configuration information on the stable operation of the system, step S101 of the log collection server obtaining the user's configuration information for log collection can be performed according to the following steps:

[0109] (1) The log collection server receives the custom configuration information input by the user through the interface or API to create or modify the log collection policy;

[0110] Users can create, modify, and delete log collection policies through the interface or API to achieve flexible policy configuration and management.

[0111] To facilitate the user's custom configuration process, a series of preset policy templates can be set in the user configuration interface so that users can quickly reference and modify when creating new policies. The template can contain commonly used log paths, file name formats, and collection frequencies, which are not limited here.

[0112] (2) Review whether the custom configuration information meets the preset requirements; if so, use the custom configuration information as the configuration information.

[0113] Configure the approval process before the policy takes effect. After the policy is created or modified, it must go through the specified approval process to review whether the user-defined configuration information meets the preset requirements. The configuration information will take effect only after the review is passed, thereby improving the accuracy and security of policy configuration.

[0114] In one embodiment, since there are many nodes in the big data cluster and the operating status of each node is different, in order to avoid the impact of unsafe operation of the server on the log collection process, before executing step S104 to distribute each log collection strategy to the log collection client in the corresponding target server, the health status of the target server can be first determined to determine whether it is in an available state; if it is in an unavailable state, an alarm message of server unavailability is generated; if it is in an available state, the step of distributing each log collection strategy to the log collection client in the corresponding target server is continued.

[0115] The method provided in this embodiment performs a health check on the target server before policy distribution to ensure that the target server is available. If the target server is unavailable, the server is skipped and an alarm message is generated, thereby avoiding the impact of the unavailable server on the log collection process and improving the stability of automatic log collection.

[0116] In one embodiment, when the scale of the log collection policy data to be distributed is large, in order to avoid the large-scale distribution of log collection policies from affecting the operation of other functions of the system and to ensure the stability of the overall operation of the system, before distributing each log collection policy to the log collection client in the corresponding target server in step S104, the following steps can be further performed: determine the number of log collection policies to be distributed, and determine whether it exceeds the load threshold; if it does not exceed the load threshold, execute the step of distributing each log collection policy to the log collection client in the corresponding target server; if it exceeds the load threshold, group the log collection policies to be distributed to obtain several policy groups; accordingly, step S104 is specifically: send each policy group in batches to the log collection client in the corresponding target server in turn.

[0117] In a large-scale cluster environment, the distribution strategy can be implemented in batches to avoid excessive network and system load caused by one-time distribution, which in turn affects the operation of the system. Specifically, the distribution batches can be grouped by server and then carried out in batches in sequence, or other grouping methods can be selected according to actual application requirements, which is not limited in this embodiment.

[0118] In one embodiment, after each log collection policy is distributed to the log collection client in the corresponding target server in step S104, the real-time distribution status may be further monitored and tracked, and the real-time distribution status may be updated to the policy configuration table in real time.

[0119] During the distribution process, the distribution status of each server is tracked in real time, and the status of each distribution request is recorded in detail, such as success, failure, in progress, etc. The distribution progress is displayed in the console through the policy configuration table, and can also be persistently stored in the log to monitor the policy distribution process. Further corresponding processing can be performed according to different policy distribution statuses.

[0120] Furthermore, after the real-time distribution status is updated to the policy configuration table in real time, it is also possible to determine whether the log collection policy is sent successfully based on the real-time distribution status in the policy configuration table; if the sending fails, the step of distributing each log collection policy to the log collection client in the corresponding target server is executed after a specified interval according to the retry strategy; and the number of retries is determined after the retry to determine whether the retry threshold is reached; if the retry threshold is reached, an alarm message of sending failure is generated.

[0121] If the policy distribution fails, the system automatically retries the distribution. The number of retries and the interval time can be configured in the retry policy. If the retries fail multiple times and reach the retry threshold, the system can trigger an alarm and record detailed error information. It can also notify the operation and maintenance personnel in various ways, such as email, SMS, instant messaging, etc., to avoid the waste of resources caused by multiple retries.

[0122] In one embodiment, the configuration information also includes: policy extension information, and the policy extension information includes: policy version, policy description, policy triggering condition, and policy dependency.

[0123] Among them, the policy version number is such as v1.0, v1.1. Each time the policy is modified, a new version can be generated, which is convenient for distinguishing whether the version is updated. It can also support policy rollback and historical policy tracing.

[0124] The policy description contains a detailed policy description field, which can be used to record the purpose, background information, and special considerations of the policy, making it easier for operation and maintenance personnel to understand and manage it.

[0125] By setting policy trigger conditions, you can achieve automatic policy triggering, such as time triggering, event triggering, etc. For example, when the log file size exceeds a certain threshold, log collection is immediately triggered, realizing the automation of policy triggering.

[0126] Dependencies between policies can ensure the order and integrity of log collection. For example, the execution of policy B needs to wait for the completion of policy A.

[0127] In this embodiment, by setting the above policy extension information, not only can the management of operation and maintenance personnel be facilitated, but also the automatic sequential triggering of policies can be achieved, thereby ensuring the logic of policy execution.

[0128] In one embodiment, after the log collection client receives the log collection policy sent by the log collection server in step S201, the log collection policy may be cached to a local file.

[0129] After the client obtains the policy for the first time, it caches it in a local file so that the last policy can still be executed when the network is unavailable, ensuring the stable execution of the log collection process. In addition, the client can also be set to hot update, allowing new policy configurations to be dynamically loaded at runtime without restarting the service, thereby further improving the execution speed of the log collection process.

[0130] Regarding the specific log collection process of the log collection client, in order to improve the log collection efficiency, step S203 triggers log collection for the changed target file, which can be implemented according to the following steps: determine whether the occupied space of the target file meets the large file standard; if it meets the large file standard, determine that the target file is a large file; segment the log of the large file; and collect the logs of each segment in parallel.

[0131] Large log files are fragmented and collected in parallel after being divided into fragments according to time periods or sizes, thereby improving the collection speed and efficiency of large files. Furthermore, the collection status of each fragment can be recorded separately to ensure that subsequent collection can continue even if some fragments fail.

[0132] For non-large files, traditional collection methods can be used without limitation.

[0133] In one embodiment, after the client triggers log collection for a changed target file, the state of the target file can be determined to identify whether there is an abnormality: if an abnormality is detected in the target file, a file abnormality mark can be added to the collected log.

[0134] For log files with abnormal status, such as abnormal format or excessive size, the system can mark them separately and send a report to the administrator after collection for subsequent analysis and processing.

[0135] In an embodiment, further storage management may be performed on the logs stored in the storage system. Specifically, life cycle management may be performed on the logs in each storage system according to a configured management policy.

[0136] The method provided in this embodiment configures a lifecycle management strategy for log storage for stored logs, such as automatically archiving or deleting logs after they have been saved for a certain period of time, and compressing and storing logs and transferring them to cold storage, etc., to avoid excessive storage space occupation.

[0137] Further references Figure 3 , which shows an exemplary structural block diagram of a dynamic log collection system according to an embodiment of the present application, which mainly includes: a log collection server and a log collection client. Among them, the log collection client is installed in each target server to be log collected, and the log collection server and each log collection client can transmit data through the HTTP protocol. The log data collected by the log collection client is stored in one or more specified storage systems, each storage system can be located in each target server or in other devices. In this embodiment, there is no limitation on the installation location of the storage system, and the figure takes the installation location outside the target server as an example.

[0138] Specifically, the specific operation process of the log collection server and the log collection client is as follows:

[0139] The log collection server is used to obtain the user's configuration information for log collection; store the configuration information and add it to the policy configuration table of the database; scan the policy configuration table in the database to determine each log collection policy to be distributed and the corresponding one or more target servers; distribute each log collection policy to the log collection client in the corresponding target server; wherein the configuration information includes the log collection policy; the log collection policy includes: policy name, one or more target servers, one or more log file paths, collection frequency corresponding to each log file path, policy status, log filtering and conversion rules;

[0140] The log collection client is used to receive the log collection policy issued by the log collection server; monitor the files under each log file path to identify whether there are file changes; if file changes are detected, trigger log collection for the changed target files; store the collected logs in one or more specified storage systems according to the storage policy.

[0141] It should be understood that the units described in the above device are similar to those in the reference Figure 1 The steps in the method described above correspond to each other. Therefore, the operations and features described above for the method are also applicable to the device and the units contained therein, and will not be repeated here. The device can be pre-implemented in the browser or other security application of the server, or loaded into the browser or its security application of the server by downloading or the like. The corresponding units in the device can cooperate with the units in the server to implement the solution of the embodiment of the present application.

[0142] For the several units mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more units described above can be embodied in one unit. On the contrary, the features and functions of one unit described above can be further divided into multiple units to be embodied.

[0143] It should be noted that for details not disclosed in the dynamic log collection device of the embodiment of the present application, please refer to the details disclosed in the above embodiments of the present application, which will not be repeated here.

[0144] Reference below Figure 4 , Figure 4 A schematic diagram of the structure of a computer system suitable for implementing a server of an embodiment of the present application is shown.

[0145] like Figure 4As shown, the computer system includes a central processing unit (CPU) 401, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 402 or the program loaded from the storage part 408 into the random access memory (RAM) 403. In the RAM 403, various programs and data required for the operation instructions of the system are also stored. The CPU 401, the ROM 402, and the RAM 403 are connected to each other through a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0146] The following components are connected to the I / O interface 405: an input section 406 including a keyboard, a mouse, etc.; an output section 407 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN card, a modem, etc. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the I / O interface 405 as needed. A removable medium 411, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 410 as needed, so that a computer program read therefrom is installed into the storage section 408 as needed.

[0147] In particular, according to an embodiment of the present application, the above reference flow chart Figure 1 The described process can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer readable medium, and the computer program includes a program code for executing the method shown in the flow chart. In such an embodiment, the computer program includes a program code for executing the method shown in the flow chart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 409, and / or installed from the removable medium 411. When the computer program is executed by the central processing unit (CPU) 401, the above-mentioned functions defined in the system of the present application are executed.

[0148] It should be noted that the computer-readable medium shown in the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium such as a computer-readable storage medium that can send, propagate or transmit a program for use by or in conjunction with an instruction execution system, device or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0149] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operating instructions of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the aforementioned module, program segment or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, the boxes represented by two connections can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operating instruction, or can be implemented with a combination of dedicated hardware and computer instructions.

[0150] The units or modules involved in the embodiments described in the present application may be implemented by software or hardware. The units or modules described may also be arranged in a processor. The names of these units or modules do not, in some cases, constitute limitations on the units or modules themselves.

[0151] As another aspect, the present application further provides a computer-readable storage medium, which may be included in the server described in the above embodiment, or may exist independently without being assembled into the server. The above computer-readable storage medium stores one or more programs, and when the above programs are used by one or more processors to execute the dynamic log collection method described in the present application.

[0152] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the aforementioned disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

Claims

1. A dynamic log collection method, characterized in that: include: The log collection server obtains the user's configuration information for log collection; wherein the configuration information includes a log collection strategy; the log collection strategy includes: a strategy name, one or more target servers, one or more log file paths, a collection frequency corresponding to each of the log file paths, a strategy state, and log filtering and conversion rules; Storing the configuration information and adding it to the policy configuration table of the database; Scan the policy configuration table in the database to determine each log collection policy to be distributed and the corresponding one or more target servers; Each log collection strategy is distributed to a log collection client in a corresponding target server, so that the log collection client collects logs of files under each log file path and stores them in one or more designated storage systems.

2. The method according to claim 1, characterized in that After distributing each log collection strategy to the log collection client in the corresponding target server, the method further includes: Querying the log collection client for system operation data of the target server; Analyze the returned system operation data to determine the system load status; Dynamically adjusting the log collection strategy corresponding to the target server according to the system load status to obtain an updated log collection strategy; The updated log collection policy is synchronously updated to the log collection client.

3. The method according to claim 1, characterized in that The log collection server obtains the user's configuration information for log collection, including: The log collection server receives the custom configuration information input by the user through the interface or API to create or modify the log collection policy; Review whether the custom configuration information meets the preset requirements; If satisfied, the customized configuration information is used as the configuration information.

4. The method according to claim 1, characterized in that Before distributing each log collection strategy to the log collection client in the corresponding target server, the method further includes: Determine the health status of the target server and judge whether it is in an available state; If it is in an unavailable state, an alarm message indicating that the server is unavailable is generated; If it is in an available state, execute the step of distributing each log collection policy to the log collection client in the corresponding target server.

5. The method according to claim 1, characterized in that Before distributing each log collection strategy to the log collection client in the corresponding target server, the method further includes: Determine the number of log collection policies to be distributed and whether the load threshold is exceeded; If the load threshold is not exceeded, executing the step of distributing each log collection strategy to the log collection client in the corresponding target server; If the load threshold is exceeded, the log collection strategies to be distributed are grouped to obtain a number of strategy groups; Accordingly, the distributing of each log collection strategy to the log collection client in the corresponding target server specifically includes: sending each of the strategy groups to the log collection client in the corresponding target server in batches in sequence.

6. The method according to claim 1, characterized in that After distributing each log collection strategy to the log collection client in the corresponding target server, the method further includes: Monitor and track real-time distribution status; The real-time distribution status is updated to the policy configuration table in real time.

7. The method according to claim 6, characterized in that After the real-time distribution status is updated to the policy configuration table in real time, the method further includes: Determine whether the log collection policy is sent successfully according to the real-time distribution status in the policy configuration table; If the sending fails, the step of distributing each log collection strategy to the log collection client in the corresponding target server is executed after a specified interval according to the retry strategy; Determine the number of retry attempts and whether the retry threshold has been reached; If the retry times threshold is reached, an alarm message indicating a sending failure is generated.

8. A dynamic log collection method, characterized in that: include: The log collection client receives the log collection policy issued by the log collection server; wherein the log collection policy includes: a policy name, one or more log file paths, a collection frequency corresponding to each of the log file paths, a policy status, and log filtering and conversion rules; Monitor the files under each of the log file paths to identify whether there are any file changes; If a file change is detected, log collection is triggered for the changed target file; The collected logs are stored in one or more specified storage systems according to the storage policy.

9. A dynamic log collection system, characterized in that: include: The log collection server is used to obtain the user's configuration information for log collection; Storing the configuration information and adding it to the policy configuration table of the database; Scan the policy configuration table in the database to determine each log collection policy to be distributed and the corresponding one or more target servers; Distribute each log collection strategy to the log collection client in the corresponding target server; wherein the configuration information includes the log collection strategy; the log collection strategy includes: a strategy name, one or more target servers, one or more log file paths, a collection frequency corresponding to each of the log file paths, a strategy status, and log filtering and conversion rules; The log collection client is used to receive the log collection policy issued by the log collection server; monitor the files under each log file path to identify whether there are file changes; if file changes are detected, trigger log collection for the changed target files; store the collected logs in one or more specified storage systems according to the storage policy.

10. A server comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method according to any one of claims 1 to 7 or the steps of the method according to claim 8 are implemented.

Images