Distributed log collection and storage methods

Abstract

The present invention provides a distributed log collection, storage method and system, including the following steps: collecting each type of log through a log collection component; sending the logs collected in the log collection step to a unified log cache component for log caching; processing the logs cached in the log cache step, and providing different log processing methods for different log types; using a distributed log storage component to load balance the logs after the log processing step, so as to achieve rapid horizontal expansion and log life cycle management from writing to deletion; managing the capacity of the log processing component; and managing the capacity of the log storage component. The present invention can solve the problems of collecting, processing and storing application logs in a platform, and evaluate the capacity according to the resource usage of the collection component, and automatically expand the capacity.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a distributed log collection and storage method and system, and in particular to a distributed log collection, processing and storage method. Background Art

[0002] With the rise of industrial Internet of Things platforms and the development of various deployment methods such as distributed and containerized, logs are becoming more and more scattered, and it is becoming more and more difficult to obtain logs manually. The log storage cost of each application is also increasing. At this time, a platform-level application log centralized storage component can solve the problems of scattered application logs, difficult acquisition, and high storage costs. In the construction of an industrial Internet platform, the log component that can provide log storage PAAS services for connected applications is also an essential service component of the platform. The platform-level application log storage component can solve the problems of scattered application logs, difficult acquisition, and high storage costs, but it also brings about the problem of collection technology. Conventional log collection solutions can no longer meet the collection and storage of a large number of logs at the platform level, and cannot automatically expand capacity.

[0003] The patent document with publication number CN103209087B discloses a distributed log statistics processing method, including the following steps: obtaining the original log data on the log generation end; statistically processing the obtained original log data to obtain preliminary statistical results; sending the preliminary statistical results to the central server; merging the preliminary statistical results through the central server. However, the patent document still has the defect that the collection cannot meet the collection and storage of a large number of logs at the platform level and cannot automatically expand the capacity. Summary of the invention

[0004] In view of the defects in the prior art, the object of the present invention is to provide a distributed log collection and storage method and system.

[0005] A distributed log collection and storage method provided by the present invention includes log collection, processing, storage steps and log component capacity management steps;

[0006] The log collection, processing and storage steps include the following steps:

[0007] Log collection steps: collect each type of log through the log collection component;

[0008] Log caching step: send the logs collected in the log collection step to a unified log caching component for log caching;

[0009] Log processing step: Process the logs cached in the log caching step and provide different log processing methods for different log types;

[0010] Log storage step: After the log processing step, the logs are stored in a distributed log storage component to perform load balancing for log writing, achieve rapid horizontal expansion, and manage the log lifecycle from writing to deletion.

[0011] The log component capacity management step includes the following steps:

[0012] Log processing component capacity management steps: manage the capacity of the log processing component;

[0013] Log storage component capacity management steps: Manage the capacity of the log storage component.

[0014] Preferably, in the log collection step, the log collection component includes:

[0015] Application server log collector: monitors log files, collects newly generated logs in the files, and records the log collection location. Each line of log collection is a log. After the log is collected into the cache area, the log is sent after the number of logs reaches the set number or the cache time of the data cache area reaches the set time. Before sending the log, identify and mark the type and source of the log, identify the log type, and mark the log source;

[0016] Browser behavior log collector: Each log is sent immediately. After the log is generated, it is sent immediately after processing. Before the log is sent, the log type and source are identified and marked. The log type is identified and the log source is marked.

[0017] Mobile APP log collector: caches the logs that need to be collected. After the logs are collected into the cache area, the logs are sent after the number of logs reaches the set number or the cache time of the data cache area reaches the set time; before sending the logs, identify and mark the type and source of the logs, identify the log type, and mark the log source.

[0018] Preferably, in the log caching step, a cache queue is created for each type of log, and the logs of this type are cached.

[0019] Preferably, the log processing step includes a loading step, a processing step and a sending step;

[0020] Loading step: connect to the log cache component, load the log from the cache component to the log processing component, and then send it to the log processing step;

[0021] Processing step: The logs loaded in the log loading step are processed in an organized manner and then sent to the log sending step;

[0022] Sending steps: Divide the logs into batches, load the storage information, and then send them to the log storage component.

[0023] Preferably, in the processing steps, the processing method is one or more of the following: JSON parsing, key-value pair parsing, regular expression parsing, date parsing, field addition, deletion, modification, log addition, deletion, splitting, IP parsing, XML format parsing, URL decoding, log sampling processing or log authentication processing.

[0024] Preferably, the log processing component capacity management step includes the following steps:

[0025] Monitoring and acquisition steps: monitor the log cache component and log processing component to obtain the data required for evaluation;

[0026] Evaluation and judgment steps: Perform a log capacity evaluation on the log processing component every hour to determine whether the log processing component needs to increase or decrease nodes;

[0027] Implementation steps: Based on the capacity assessment results, increase or decrease the number of log processing nodes in the container management platform.

[0028] Preferably, determining whether the log processing component needs to increase or decrease nodes specifically includes a data acquisition step, a cache determination step, a speed determination step, and a speed node determination step;

[0029] Data acquisition steps: Get the backlog log volume of the log cache queue, the average processing speed of the log processing component within 5 minutes, and the benchmark processing speed of the log processing component;

[0030] Cache judgment step: judge whether the log cache queue is greater than M within 5 minutes, where M is a number set according to the tolerable log delay; if it is greater than M, enter the speed judgment step; if it is not greater than M, proceed to the speed node judgment step;

[0031] Speed ​​judgment step: judge whether the 5-minute average processing speed of the log processing component is greater than 80% of the benchmark processing speed; if it is greater than 80%, add a node; if it is not greater than 80%, end the evaluation;

[0032] Speed ​​node judgment step: judge whether the 5-minute average processing speed of the log processing component is less than 30% of the benchmark processing speed, and the number of log processing component nodes is greater than 2; if so, reduce one log processing component node and end the evaluation; if not, end the evaluation.

[0033] Preferably, the log storage component capacity management step includes the following steps:

[0034] Monitoring and judgment steps: Monitor the log storage component and obtain the data required for evaluation. The following evaluation data needs to be obtained;

[0035] Evaluation and judgment steps: Perform capacity evaluation on the log storage component every day to determine whether additional nodes are needed for the log storage component.

[0036] Implementation steps: Based on the capacity assessment results, increase or decrease log storage nodes in the container management platform.

[0037] Preferably, the step of judging whether the log storage component needs to add a node specifically includes a step of obtaining data, a step of determining a disk, and a step of determining the disk:

[0038] Data acquisition steps: Get the average disk usage and memory usage of the log storage component;

[0039] Disk judgment step: Check whether the average disk usage of the log storage component is greater than 75%; if it is greater than 75%, add a new node and end the evaluation; if it is not greater than 75%, proceed to the memory judgment step;

[0040] Memory judgment step: determine whether the time when the average memory usage of the log storage component is greater than 70% is greater than 12 hours; if it is greater than 12 hours, add a node and end the evaluation; if it is not greater than 12 hours, end the evaluation.

[0041] The present invention also provides a distributed log collection and storage system, including a log collection, processing, storage module and a log component capacity management module;

[0042] The log collection, processing and storage module includes the following modules:

[0043] Log collection module: collects each type of log through the log collection component;

[0044] Log cache module: sends the logs collected in the log collection step to a unified log cache component for log caching;

[0045] Log processing module: processes the logs cached in the log caching step and provides different log processing methods for different log types;

[0046] Log storage module: The logs processed by the log processing steps are stored in a distributed log storage component to perform load balancing for log writing, achieve rapid horizontal expansion, and manage the log lifecycle from writing to deletion.

[0047] The log component capacity management module includes the following modules:

[0048] Log processing component capacity management module: manages the capacity of the log processing component;

[0049] Log storage component capacity management module: manages the capacity of the log storage component.

[0050] Compared with the prior art, the present invention has the following beneficial effects:

[0051] 1. The present invention can solve the problem of log component resource capacity evaluation and automatic expansion caused by a large amount of log collection in platform mode;

[0052] 2. The present invention can solve the problems of scattered application logs, difficulty in obtaining, and high storage costs of industrial Internet of Things platforms;

[0053] 3. The present invention can solve the problems of collecting, processing and storing application logs within a platform, and can evaluate the capacity according to the resource usage of the collection components and automatically expand the capacity. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0055] Figure 1 This is an overall flow chart of the distributed log collection and storage method of the present invention;

[0056] Figure 2 It is a flow chart of log processing steps in the distributed log collection and storage method of the present invention;

[0057] Figure 3 A flow chart for determining whether a log processing component needs to increase or decrease nodes in the distributed log collection and storage method of the present invention;

[0058] Figure 4 The present invention is a flow chart for determining whether a log storage component needs to add a node in the distributed log collection and storage method of the present invention. DETAILED DESCRIPTION

[0059] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0060] Embodiment 1:

[0061] This embodiment provides a distributed log collection and storage method, including log collection, processing, storage steps and log component capacity management steps;

[0062] 1. The log collection, processing, and storage steps include the following steps:

[0063] 1.1 Log collection steps: Each type of log is collected through the log collection component; the log collection component includes: application server log collector: monitor the log file, collect the newly generated logs in the file, and record the log collection location. Each line of log collection is a log. After the log is collected into the cache area, the number of logs reaches the set number or the cache time of the data cache area reaches the set time, the log is sent; before the log is sent, the type and source of the log are identified and marked, the log type is identified, and the log source is marked; browser-side behavior log collector: each log is sent once immediately. After the log is generated, it is sent immediately after processing; before the log is sent, the type and source of the log are identified and marked, the log type is identified, and the log source is marked; mobile APP log collector: cache the logs that need to be collected. After the log is collected into the cache area, the number of logs reaches the set number or the cache time of the data cache area reaches the set time, the log is sent; before the log is sent, the type and source of the log are identified and marked, the log type is identified, and the log source is marked.

[0064] 1.2 Log caching step: The logs collected in the log collection step are sent to a unified log caching component for log caching; a cache queue is created for each type of log, and the logs of this type are cached.

[0065] 1.3 Log processing step: process the logs cached in the log caching step, and provide different log processing methods for different log types; the log processing step includes a loading step, a processing step and a sending step; the loading step: connect the log caching component, load the log from the cache component into the log processing component, and then send it to the log processing step; the processing step: perform an organized processing on the log loaded in the log loading step, and then send it to the log sending step; the sending step: batch the logs, load the storage information, and then send it to the log storage component; in the processing step, the processing method is one or more of the following: JSON parsing, key-value pair parsing, regular expression parsing, date parsing, adding, deleting, modifying fields, adding, deleting, splitting logs, IP parsing, XML format parsing, URL decoding, log sampling processing or log authentication processing.

[0066] 1.4 Log storage step: The logs processed by the log processing step are stored in a distributed log storage component to perform load balancing for log writing, achieve rapid horizontal expansion and log life cycle management from log writing to log deletion.

[0067] 2. The log component capacity management steps include the following steps:

[0068] Log processing component capacity management steps: manage the capacity of the log processing component;

[0069] Log storage component capacity management steps: Manage the capacity of the log storage component.

[0070] 2.1 The log processing component capacity management steps include the following steps:

[0071] a. Monitoring and acquisition steps: Monitor the log cache component and log processing component to obtain the data required for evaluation.

[0072] b. Evaluation and judgment steps: Perform a log capacity evaluation on the log processing component every hour to determine whether the log processing component needs to increase or decrease nodes.

[0073] Determining whether the log processing component needs to increase or decrease nodes specifically includes determining whether the log processing component needs to increase or decrease nodes specifically includes a data acquisition step, a cache determination step, a speed determination step, and a speed node determination step:

[0074] Data acquisition steps: Get the backlog log volume of the log cache queue, the average processing speed of the log processing component within 5 minutes, and the benchmark processing speed of the log processing component;

[0075] Cache judgment step: judge whether the log cache queue is greater than M within 5 minutes, where M is a number set according to the tolerable log delay; if it is greater than M, enter the speed judgment step; if it is not greater than M, proceed to the speed node judgment step;

[0076] Speed ​​judgment step: judge whether the 5-minute average processing speed of the log processing component is greater than 80% of the benchmark processing speed; if it is greater than 80%, add a node; if it is not greater than 80%, end the evaluation;

[0077] Speed ​​node judgment step: judge whether the 5-minute average processing speed of the log processing component is less than 30% of the benchmark processing speed, and the number of log processing component nodes is greater than 2; if so, reduce one log processing component node and end the evaluation; if not, end the evaluation.

[0078] c. Implementation steps: Based on the capacity assessment results, increase or decrease the log processing nodes in the container management platform.

[0079] 2.2 The log storage component capacity management steps include the following steps:

[0080] a. Monitoring and judgment steps: Monitor the log storage component and obtain the data required for evaluation. The following evaluation data needs to be obtained.

[0081] b. Evaluation and judgment steps: Perform capacity evaluation on the log storage component every day to determine whether additional nodes are needed.

[0082] Judging whether the log storage component needs to add nodes specifically includes the steps of obtaining data, disk judgment, and memory judgment:

[0083] Data acquisition steps: Get the average disk usage and memory usage of the log storage component;

[0084] Disk judgment step: Check whether the average disk usage of the log storage component is greater than 75%; if it is greater than 75%, add a new node and end the evaluation; if it is not greater than 75%, proceed to the memory judgment step;

[0085] Memory judgment step: determine whether the time when the average memory usage of the log storage component is greater than 70% is greater than 12 hours; if it is greater than 12 hours, add a node and end the evaluation; if it is not greater than 12 hours, end the evaluation.

[0086] c. Implementation steps: Based on the capacity assessment results, increase or decrease log storage nodes in the container management platform.

[0087] Embodiment 2:

[0088] This embodiment provides a distributed log collection and storage system, including a log collection, processing, storage module and a log component capacity management module;

[0089] The log collection, processing and storage module includes the following modules:

[0090] Log collection module: collects each type of log through the log collection component;

[0091] Log cache module: sends the logs collected in the log collection step to a unified log cache component for log caching;

[0092] Log processing module: processes the logs cached in the log caching step and provides different log processing methods for different log types;

[0093] Log storage module: The logs processed by the log processing steps are stored in a distributed log storage component to perform load balancing for log writing, achieve rapid horizontal expansion, and manage the log lifecycle from writing to deletion.

[0094] The log component capacity management module includes the following modules:

[0095] Log processing component capacity management module: manages the capacity of the log processing component;

[0096] Log storage component capacity management module: manages the capacity of the log storage component.

[0097] Embodiment 3:

[0098] Those skilled in the art may understand this embodiment as a more specific description of Embodiment 1 and Embodiment 2.

[0099] This embodiment provides a distributed log collection, processing, and storage method, which includes two process components: a log collection, processing, and storage process component and a log component capacity management process component.

[0100] 1. Log collection, processing, and storage process components:

[0101] This section describes the entire log collection process, which includes the following processes: the log client collects logs and then sends the logs to the log cache component. The log processing component obtains logs from the log cache component and sends the logs to the log storage component after various processing. For the overall flow chart, refer to Figure 1 .

[0102] The log collection, processing, and storage process components include the following components:

[0103] 1.1. Log collection component:

[0104] The present invention supports the collection of three types of logs: application server logs, browser behavior logs, and mobile APP logs. For the three types of logs, three types of log collection client Agents are provided respectively for the three types of log collection.

[0105] (1) The first type of application server log collector uses a log file collector to monitor log files, collect newly generated logs in the file, and record the log collection location. Each line of log collection is a log. After the log is collected into the cache area, when the log volume reaches a certain number or the data cache area is cached for a certain period of time, if any condition is met, the log is sent; before the log is sent, the log type and source are identified and marked, the log type is identified, and the log source is marked for subsequent log data processing.

[0106] (2) The second type of browser-side behavior log collector loads the log collection client in the browser. The log storage in the browser is uncontrollable. To ensure the real-time and integrity of the log, each log is sent immediately. After the log is generated, it is sent immediately after processing. Before the log is sent, the log type and source are identified and marked. The log type is identified and the log source is marked for subsequent log data processing.

[0107] (3) The third type of mobile APP log collector is installed in the mobile APP client and caches the logs that need to be collected. After the logs are collected into the cache area, when the log volume reaches a certain number or the data cache area is cached for a certain period of time, if any condition is met, the logs are sent; before the logs are sent, the logs are identified and marked, the log type is identified, and the log source type and source are marked for subsequent log data processing.

[0108] 1.2. Log cache component:

[0109] After being collected, a large number of logs are sent to a unified log cache component for log caching. A cache queue is created for each type of log to cache the logs of that type.

[0110] 1.3. Log processing component:

[0111] Process the logs and provide different log processing methods for different log types. Log processing is divided into three steps: log loading, log processing, and log sending. The log processing component process is shown in Figure 2 .

[0112] (1) Log loading: Connect to the log cache component, load the log from the cache component to the log processing component, and then send it to the log processing step.

[0113] (2) Log processing: The logs loaded in the log loading step are processed in an organized processing flow, and then sent to the log sending step. There are many methods for log processing. Each method is processed as a plug-in, and then a log processing step that can be organized is formed. The processing method is organized according to the characteristics of the log to be processed. The log processing methods provided by the present invention include the following: JSON parsing, key-value pair parsing, regular expression parsing, date parsing, adding, deleting, and modifying fields, adding, deleting, and splitting logs, IP parsing, XML format parsing, URL decoding, log sampling processing, and log authentication processing.

[0114] (3) Log sending: The processed logs are transmitted to the log sending step, which batches the logs, loads the storage information, and then sends them to the log storage component.

[0115] 1.4. Log storage component:

[0116] The log storage component uses a distributed log storage device, which can perform load balancing of log writing, can quickly expand horizontally, and can realize log life cycle management from log writing to log deletion.

[0117] 2. Log component capacity management process components:

[0118] The log component capacity management process has deployment requirements for the log processing component and log storage component in the log component: the log processing component is an independent log processing component that can independently process logs without relying on other log processing components and can be quickly deployed in a container; the log storage component needs to be a distributed storage cluster system that can add nodes without stopping the system and can be deployed in a container. Based on the above conditions, the log component capacity management component can complete the capacity management of the two log components.

[0119] 2.1. Log processing component capacity management:

[0120] Monitor the log cache component and log processing component to obtain the data required for evaluation. The following data needs to be obtained:

[0121] (1) The number of backlogged logs in the log cache component queue, which is the sum of the number of logs in all log cache queues;

[0122] (2) The average log processing speed of each log processing component node: the number of logs processed by all log processing components in the past 5 seconds / (5*number of log processing component nodes);

[0123] (3) The benchmark processing speed of the single-node log processing component, which is the maximum speed at which the single-node log processing component can process logs. The benchmark processing speed of a single node is tested before the system is deployed to obtain data.

[0124] Perform a log capacity assessment on the log processing component every hour to determine whether the log processing component needs to increase or decrease nodes. See the assessment flow chart for details. Figure 3 .

[0125] Process description:

[0126] (1) Obtaining data: the amount of backlogged logs in the log cache queue, the average processing speed of the log processing component within 5 minutes, and the benchmark processing speed of the log processing component;

[0127] (2) Determine whether the log cache queue is always greater than M (a number set according to the tolerable log delay) within 5 minutes. If so, check whether the 5-minute average processing speed of the log processing component is greater than 80% of the benchmark processing speed. If so, add a node;

[0128] (3) If the log volume in the log cache queue within 5 minutes is not greater than M, determine whether the 5-minute average processing speed of the log processing component is less than 30% of the benchmark processing speed, and the number of log processing component nodes is greater than 2. If so, reduce one log processing component node;

[0129] (4) In other cases, the number of nodes of the log processing component remains unchanged.

[0130] According to the capacity assessment results, log processing nodes are added or reduced in the container management platform.

[0131] 2.2. Log storage component capacity management:

[0132] Monitor the log storage component and obtain the data required for evaluation. The following evaluation data is required:

[0133] (1) Average disk usage of the log storage component: the sum of the disk usage of all node log storage divided by the total disk usage of all nodes;

[0134] (2) Average memory usage of the log storage component: the sum of memory usage of all nodes divided by the total memory of all nodes. The log storage component nodes are tested to obtain the optimal memory to CPU ratio, and the server is configured with a fixed ratio to reduce CPU considerations in capacity assessment.

[0135] The capacity of the log storage component is evaluated every day to determine whether the log storage component needs to add nodes; the log component capacity management component does not reduce the capacity of the log storage component, which needs to be done manually. For the evaluation process, see Figure 4 .

[0136] Process description:

[0137] (1) Obtain data: average disk usage of the log storage component and memory usage of the log storage component;

[0138] (2) Whether the average disk usage of the log storage component is greater than 75%. If so, a new node is added and the evaluation ends. If not, proceed to the next step.

[0139] (3) If the average disk usage of the log storage component is not greater than 75%, determine whether the time when the average memory usage of the log storage component is greater than 70% is greater than 12 hours. If so, add a new node and end the evaluation; if not, end the evaluation.

[0140] Based on the capacity assessment results, log storage nodes are added to the container management platform.

[0141] Embodiment 4:

[0142] Those skilled in the art may understand this embodiment as a more specific description of Embodiment 1 and Embodiment 2.

[0143] In order to better explain the present invention, the following is a description with reference to this example.

[0144] The present invention is better explained by applying the collection, processing and storage process of tomcat container logs.

[0145] You need to install the application server log collector on the application server, configure the log collector, mark the log type and source, mark the log type - tomcat log, and mark the log source - the unique identifier of the application in the platform.

[0146] The log cache component uses the open source component Kafka. A cache queue of the tomcat log type is created in the log cache component Kafka. The logs collected by the log collector are output to this queue.

[0147] In the log processing component, arrange the processing flow for the tomcat log:

[0148] (1) Regular expression matching and parsing to match the fields that need to be stored independently;

[0149] (2) Log deletion: delete invalid logs;

[0150] (3) Field deletion: delete useless and repeated fields;

[0151] (4) Add fields, add log source fields;

[0152] (5) Date parsing: parsing the Tomcat date and time into a unified storage usage time;

[0153] (6) IP resolution, resolving the corresponding geographic address through the IP address;

[0154] (7) Log authentication processing: determine whether the log has permission to be stored, and delete the log without permission to be stored.

[0155] After being processed by the log processing component, the log is sent to the log storage component. The log storage component uses the open source distributed software Elasticsearch and stores the logs in the index in Elasticsearch. The index is stored according to the application, with one index per application per day. The log storage duration for each application is then set at 6 months.

[0156] Capacity management component, set the log cache component queue backlog M = 10000, according to Figure 3 The logic evaluates the capacity of the log processing component every hour to determine whether to increase or decrease the number of log processing component nodes. Figure 4 The logic evaluates the capacity of the log storage component every day to determine whether additional log storage components are needed. Both the log processing component and the log storage component support K8S containerized deployment, which can quickly deploy nodes.

[0157] The present invention can solve the problem of collecting, processing and storing application logs in a platform, and can evaluate the capacity according to the resource usage of the collection component and automatically expand the capacity. The method of the present invention can be applied to platforms and applications, providing a method and implementation for collecting and storing logs of platforms and applications.

[0158] Those skilled in the art know that, in addition to realizing the system and its various devices, modules, and units provided by the present invention in a purely computer-readable program code, it is entirely possible to realize the same functions in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered as a hardware component, and the devices, modules, and units included therein for realizing various functions can also be regarded as structures within the hardware component; the devices, modules, and units for realizing various functions can also be regarded as both software modules for realizing the method and structures within the hardware component.

[0159] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A distributed log collection and storage method, characterized in that: Includes log collection, processing, storage steps and log component capacity management steps; The log collection, processing and storage steps include the following steps: Log collection steps: collect each type of log through the log collection component; Log caching step: send the logs collected in the log collection step to a unified log caching component for log caching; Log processing step: Process the logs cached in the log caching step and provide different log processing methods for different log types; Log storage step: After the log processing step, the logs are stored in a distributed log storage component to perform load balancing for log writing, achieve rapid horizontal expansion, and manage the log lifecycle from writing to deletion. The log component capacity management step includes the following steps: Log processing component capacity management steps: manage the capacity of the log processing component; Log storage component capacity management steps: manage the capacity of the log storage component; The log processing step includes a loading step, a processing step and a sending step; Loading step: connect the log cache component, load the log from the cache component to the log processing component, and then enter the log processing step; Processing step: perform organized processing on the logs loaded in the log loading step, and then enter the log sending step; Sending steps: After dividing the logs into batches and loading the storage information, send them to the log storage component; The log processing component capacity management step includes the following steps: Monitoring and acquisition steps: monitor the log cache component and log processing component to obtain the data required for evaluation; Evaluation and judgment steps: Perform a log capacity evaluation on the log processing component every hour to determine whether the log processing component needs to increase or decrease nodes; Implementation steps: Based on the capacity assessment results, add or reduce log processing nodes in the container management platform; The log storage component capacity management step includes the following steps: Monitoring and judgment steps: monitor the log storage component to obtain the data required for evaluation; Evaluation and judgment steps: Perform capacity evaluation on the log storage component every day to determine whether additional nodes are needed for the log storage component. Implementation steps: Based on the capacity assessment results, add log storage nodes in the container management platform; In the log collection step, the log collection component includes: Application server log collector: monitors log files, collects newly generated logs in the files, and records the log collection location. Each line of log collection is a log. After the log is collected into the cache area, the log is sent after the number of logs reaches the set number or the cache time of the data cache area reaches the set time. Before sending the log, the log is identified and marked, the log type is identified, and the log source is marked; Browser behavior log collector: Each log is sent immediately. After the log is generated, it is sent immediately after processing. Before the log is sent, it is identified and marked, the log type is identified, and the log source is marked; Mobile APP log collector: caches the logs that need to be collected. After the logs are collected into the cache area, the logs are sent after the number of logs reaches the set number or the cache time of the data cache area reaches the set time; before sending the logs, identify and mark the log types and mark the log sources; Determining whether the log processing component needs to increase or decrease nodes specifically includes a data acquisition step, a cache determination step, a speed determination step, and a speed node determination step; Data acquisition steps: Get the backlog log volume of the log cache queue, the average processing speed of the log processing component within 5 minutes, and the benchmark processing speed of the log processing component; Cache judgment step: judge whether the log cache queue is greater than M within 5 minutes, where M is a number set according to the tolerable log delay; if it is greater than M, enter the speed judgment step; if it is not greater than M, proceed to the speed node judgment step; Speed ​​judgment step: judge whether the 5-minute average processing speed of the log processing component is greater than 80% of the benchmark processing speed; if it is greater than 80%, add a node; if it is not greater than 80%, end the evaluation; Speed ​​node judgment step: judge whether the 5-minute average processing speed of the log processing component is less than 30% of the benchmark processing speed, and the number of log processing component nodes is greater than 2; if yes, reduce one log processing component node and end the evaluation; if no, end the evaluation; Determining whether the log storage component needs to add nodes specifically includes the steps of obtaining data, disk determination, and memory determination: Data acquisition steps: Get the average disk usage and memory usage of the log storage component; Disk judgment step: Check whether the average disk usage of the log storage component is greater than 75%; if it is greater than 75%, add a new node and end the evaluation; if it is not greater than 75%, proceed to the memory judgment step; Memory judgment step: Determine whether the average memory usage of the log storage component is greater than 70% for more than 12 hours. If so, add a node and end the evaluation. If not, end the evaluation.

2. The distributed log collection and storage method according to claim 1, characterized in that: In the log caching step, a cache queue is created for each type of log, and the logs of this type are cached.

3. The distributed log collection and storage method according to claim 1 is characterized in that: In the processing steps, the processing method is one or more of the following: JSON parsing, key-value pair parsing, regular expression parsing, date parsing, field addition, deletion, modification, log addition, deletion, splitting, IP parsing, XML format parsing, URL decoding, log sampling processing or log authentication processing.

Images