Distributed log collection and processing method and system

By building a distributed log acquisition and processing system on the CNP controller, using the master node to interact with the front-end and distributing task instructions to the child nodes, the problem of low efficiency in log acquisition and analysis processing in the existing technology is solved, and efficient log processing is achieved and problem positioning ability.

WO2024244404A1PCT designated stage expired Publication Date: 2024-12-05CHINA TELECOM CLOUD TECH CO LTD
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
PCT/CN2023/139640
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-06-01
Filing Date
2023-12-18
Publication Date
2024-12-05

AI Technical Summary

Technical Problem

The existing ansible distributed operation and maintenance tools cannot be directly used for log collection and analysis processing on HDFS, and cannot interact with the front-end, resulting in inflexible expansion functions and inability to provide one-click processing and problem positioning capabilities, resulting in log collection and analysis. Inefficient processing.

Method used

Design a distributed log acquisition and processing system, and use multiple CNP controllers to build a distributed system, one of which is set as the main node to interact with the front-end, and distributes analysis task instructions to collect log files and configuration files for the child nodes. Analyze and process, and analyze the results through the main node feedback to achieve efficient log processing under the distributed system.

Benefits of technology

It improves log processing efficiency, realizes one-time information feedback with the front-end, supports efficient collection and analysis of HDFS and local log files and configuration files, and provides problem positioning capabilities, improving development efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2023139640_05122024_PF_FP_ABST
    Figure CN2023139640_05122024_PF_FP_ABST
Patent Text Reader

Abstract

The present disclosure relates to the technical field of network log processing, and relates to a distributed log collection and processing method and system. The system of the embodiments of the present disclosure comprises a plurality of CNP controllers, wherein a CNP controller is set as a master node, and the remaining CNP controllers are set as sub-nodes. The master node comprises a front-end instruction acquisition module, an instruction editing module, a main collection module, a main analysis and processing module and a log packaging module. Each sub-node comprises a sub-collection module and a sub-analysis and processing module. According to an analysis task instruction sent by a front end, the master node and the sub-nodes collect, analyze and process log files and configuration files so as to obtain analysis results, which are fed back to the front end by the master node. The method of the embodiments of the present disclosure is implemented on the basis of the system. The embodiments of the present disclosure can carry out log collection and analysis and processing according to front-end requirements, and feed back analysis results of a plurality of nodes in a distributed architecture via the master node, thus achieving high log processing efficiency.
Need to check novelty before this filing date? Find Prior Art

Description

A distributed log collection and processing method and system Technical Field

[0001] The embodiments of the present disclosure relate to the technical field of network log processing, and in particular to a distributed log collection and processing method and system. Background Art

[0002] HDFS (Hadoop Distributed File System) is a distributed file system. Log collection, analysis, and processing on HDFS cannot be directly implemented using the commonly used Ansible distributed operations and maintenance tool. While Ansible can perform batch system configuration, batch program deployment, and batch command execution, it lacks direct interaction with the front-end. Furthermore, it suffers from other issues: extending functionality requires modifying the source code, resulting in poor scalability; it lacks one-click processing capabilities; and it lacks the ability to locate problems.

[0003] Most CNP (cloud network platform) controller project services are deployed in a distributed manner. Although they can access any distributed file system, viewing log files and configuration files on different hosts requires checking all relevant hosts, and it is not possible to obtain log files and configuration files from all hosts at once. When troubleshooting issues during development, developers need to frequently jump to different hosts to view logs, view configuration files, and analyze problems, which is time-consuming and inefficient in log collection, analysis, and processing.

[0004] Therefore, how to achieve efficient log processing (including viewing, extended functions, positioning, etc.) based on the CNP controller while interacting with the front end is an urgent problem to be solved in the embodiments of the present disclosure.

[0005] Summary of the Invention

[0006] The embodiments of the present disclosure provide a distributed log collection and processing method and system, aiming to solve one or more of the above-mentioned problems and other potential problems.

[0007] To achieve the above objectives, the following technical solutions are provided:

[0008] According to a first aspect of the present disclosure, a distributed log collection and processing system is provided, comprising a plurality of CNP controllers, one of which is set as a master node, and the remaining CNP controllers are set as child nodes;

[0009] The master node includes:

[0010] The front-end instruction acquisition module is used to receive the analysis task instructions sent by the front-end, and write the collection information and analysis information of the log file and configuration file into the instruction editing module according to the analysis task instructions;

[0011] The instruction editing module is used to store the collection information and analysis information of log files and configuration files, and send the collection information to the collection module of the relevant node and send the analysis information to the analysis and processing module of the relevant node;

[0012] The HDFS main collection module is used to collect log files and configuration files stored on HDFS based on the collection information of received log files and configuration files;

[0013] The local main collection module is used to collect the log files and configuration files stored locally based on the collection information of the received log files and configuration files;

[0014] The main analysis and processing module is used to analyze and process the collected log files and configuration files according to the analysis information of the written log files and configuration files and obtain the analysis results;

[0015] The log packaging module is used to package the analysis results of the master node and / or sub-nodes and feed them back to the front-end;

[0016] The sub-nodes include:

[0017] The HDFS sub-collection module is used to collect log files and configuration files stored on HDFS based on the collection information of the received log files and configuration files;

[0018] The local sub-collection module is used to collect the log files and configuration files stored locally based on the collection information of the received log files and configuration files;

[0019] The sub-analysis processing module is used to perform analysis and processing based on the collected log files and configuration files according to the analysis information of the written log files and configuration files and obtain analysis results.

[0020] The system of the disclosed embodiment utilizes several CNP controllers to construct a distributed system for log collection, analysis, and processing. By setting a CNP controller as a master node to interact with the front-end, the master node obtains the front-end's analysis task instructions, and then sends the collection information and analysis information obtained based on the analysis task instructions to the relevant child nodes, enabling each node in the distributed system to collect and analyze log files and configuration files. Ultimately, the analysis results of the master node and the analysis results of other nodes can be fed back to the front-end through the master node. This utilizes the master node to achieve one-time information feedback between the distributed system and the front-end, and can complete HDFS and local log file and configuration file collection according to the front-end's needs, resulting in high log collection and processing efficiency.

[0021] In some embodiments, the instruction editing module is an XML editing module.

[0022] In some embodiments, when the instruction editing module sends the collection information to the HDFS sub-collection module, the collection information includes the HDFS interface information; when the instruction editing module sends the collection information to the HDFS main collection module, the HDFS main collection module can directly call the HDFS interface.

[0023] In some embodiments, the local main collection module and the local sub-collection module collect log files and configuration files through shell scripts.

[0024] In some embodiments, the master node further includes a problem location module for obtaining abnormal results by identifying keywords or abnormal functions in the analysis results of the master node and / or sub-nodes, and feeding back the abnormal results to the front end.

[0025] In some embodiments, the keywords include ERROR, WARN, and NULL pointer.

[0026] In some embodiments, the front-end instruction acquisition module includes:

[0027] A receiving unit, configured to receive analysis task instructions sent by the front end;

[0028] The verification unit is used to verify the received analysis task instructions, and when the verification passes, trigger the parsing unit to work;

[0029] The parsing unit is used to parse the analysis task instruction into the collection information and analysis information of the log file and configuration file, and write the information into the instruction editing module.

[0030] In some embodiments, a B / S model is established between the system and the front end.

[0031] According to a second aspect of the present disclosure, a distributed log collection and processing method is provided, which is implemented based on the above-mentioned distributed log collection and processing system. The method includes:

[0032] The front-end instruction acquisition module of the master node acquires the analysis task instruction sent by the front-end, and writes the collection information and analysis information of the log file and configuration file into the instruction editing module of the master node according to the analysis task instruction;

[0033] The instruction editing module of the master node stores the collection information and analysis information of the log files and configuration files, and sends the collection information to the collection modules of the relevant nodes, and sends the analysis information to the analysis and processing modules of the relevant nodes;

[0034] The master node and sub-node that receive the collected information collect log files and configuration files through their collection modules;

[0035] The master node and sub-node that receive the analysis information analyze and process the collected log files and configuration files through their analysis and processing modules and obtain analysis results;

[0036] The child node sends the analysis results to the master node;

[0037] The log packaging module of the master node packages the analysis results obtained at the master node and / or the analysis results obtained from the child nodes to feed back to the front end.

[0038] In some embodiments, the main node also includes a problem location module, which obtains abnormal results by identifying keywords or abnormal functions in the analysis results of the main node and / or child nodes, and feeds back the abnormal results to the front end; the method also includes: the front-end instruction acquisition module of the main node obtains the problem location instruction sent by the front end, and triggers the problem location module to work according to the problem location instruction; the problem location module feeds back the abnormal results to the front end after obtaining the abnormal results. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The above and other objects, features and advantages of the embodiments of the present disclosure will become readily understood by reading the following detailed description with reference to the accompanying drawings, in which several embodiments of the present disclosure are shown by way of example and not limitation.

[0040] FIG1 shows a schematic diagram of a distributed log collection and processing system according to an embodiment of the present disclosure;

[0041] FIG2 shows an example diagram of an implementation of a distributed log collection and processing system according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0042] The preferred embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although preferred embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0043] As used herein, the term "including" and its variations mean open inclusion, i.e., "including but not limited to." Unless otherwise stated, the term "or" means "and / or." The term "based on" means "based, at least in part, on." The terms "one example embodiment" and "an embodiment" mean "at least one example embodiment." The term "another embodiment" means "at least one additional embodiment."

[0044] Existing operations tools can be used for log collection, but the commonly used Ansible distributed operations tool cannot be used directly to collect and analyze logs on HDFS. Furthermore, the Ansible distributed operations tool cannot interact with the front-end, making it impossible to collect and analyze logs based on front-end requirements. Its expansion capabilities are also inflexible, and it cannot collect and analyze logs from multiple nodes at once.

[0045] Although CNP controllers can collect, analyze, and process logs on HDFS, each controller typically functions as a master node to collect and analyze logs. The frontend needs to send analysis task instructions to each controller separately, and each controller must independently feed back analysis results to the frontend. This makes it impossible to collect and analyze logs from multiple nodes simultaneously. This is especially true when troubleshooting issues during development, as frequent switching to different hosts to view logs, check configuration files, and analyze the problem is time-consuming and inefficient.

[0046] Based on this, the disclosed embodiments propose a distributed log collection and processing system, comprising several CNP controllers, one of which is designated as a master node, and the remaining CNP controllers as child nodes. These CNP controllers are distributed, with the master node interacting with the front-end, receiving analysis task instructions from the front-end and issuing them to the child nodes. The master node also provides feedback to the front-end regarding analysis results from the master and child nodes. This embodiment significantly improves log processing efficiency.

[0047] FIG1 shows a distributed log collection and processing system according to an embodiment of the present disclosure. A B / S model is established between the system and the front end. The master node includes a front-end instruction acquisition module, an instruction editing module, an HDFS main collection module, a local main collection module, a main analysis and processing module, and a log packaging module. The child node includes an HDFS sub-collection module, a local sub-collection module, and a sub-analysis and processing module. The front end sends analysis task instructions to the master node, which distributes them to the child nodes. The master node and the child nodes collect, analyze, and process log files and configuration files according to the analysis task instructions sent by the front end. After obtaining the analysis results, the master node feeds them back to the front end.

[0048] The front-end instruction acquisition module is used to receive the analysis task instruction sent by the front-end, and write the collection information and analysis information of the log file and configuration file into the instruction editing module according to the analysis task instruction.

[0049] The analysis task instructions include instructions for collecting and analyzing logs for one, several, or all CNP controllers. The analysis task instructions include a collection object and collection and analysis instructions. The collection object is determined by the type of collected logs, which can include business logs, system logs, platform logs, configuration files, result files, and the like. The collection and analysis instructions are parsed into specific collection information and analysis information.

[0050] Specifically, the front-end instruction acquisition module includes a receiving unit and a parsing unit. The receiving unit is used to receive the analysis task instruction sent by the front-end. The parsing unit is used to parse the analysis task instruction into the collection information and analysis information of the log file and configuration file, and write them into the instruction editing module.

[0051] To prevent malicious code attacks caused by unsafe instructions sent by the front-end, the front-end instruction acquisition module also includes a verification unit. This unit verifies the received analysis task instructions. If the verification passes, the parsing unit is triggered to operate; otherwise, the parsing unit does not operate. This verification includes checking the legitimacy and security of the parameters passed by the front-end. This verification unit can be implemented using the checkParam function (see Figure 2).

[0052] The instruction editing module is used to store the collected and analyzed information of log files and configuration files, and send the collected information to the collection module of the relevant node, and send the analyzed information to the analysis and processing module of the relevant node. For example, if the node is a master node, the main collection module and the main analysis and processing module will perform the operation; if the node is a child node, the child collection module and the child analysis and processing module will perform the operation.

[0053] The instruction editing module is an XML editing module, which inputs log file and configuration file analysis information through XML files, and can dynamically add analysis content without modifying Java code.

[0054] The HDFS master collection module is used to collect log files and configuration files stored on HDFS based on the received log file and configuration file collection information. The HDFS master collection module in the host can directly call the HDFS interface to access logs on HDFS. The data collected by the HDFS master collection module can be fed back to the front end.

[0055] The local main acquisition module is configured to collect locally stored log files and configuration files based on the received log file and configuration file collection information. The local main acquisition module collects log files and configuration files using shell scripts. For example, the scripts may include ssh and scp commands for collecting log files and configuration files, respectively. The data collected by the local main acquisition module can be fed back to the front end.

[0056] The main analysis and processing module is used to analyze and process the collected log files and configuration files based on the analysis information written into them, and generate analysis results. Log analysis is the process of analyzing the records generated by various devices in the network environment to promptly identify network threats by analyzing various changes in the network environment. The analysis results can ultimately be fed back to the front end in the form of a result file.

[0057] The main acquisition module and main analysis and processing module can collect log information by scenario through the Get function (see Figure 2), analyze it through the Collect function (see Figure 2), and obtain analysis results. The analysis results may include log data, daily route paths, abnormal results, etc.

[0058] The log packaging module is used to package the analysis results of the main node and / or child nodes to feed back to the front end. The log packaging module can be implemented using the Compress function (see Figure 2). The log packaging module can package the analysis results of the main node alone, or the analysis results of a child node alone, or the analysis results of some nodes, or the analysis results of all nodes. The front end is provided with a download interface, which can download the packaged compressed files. In addition, the log packaging module can also package the data collected by the acquisition module, such as the collection data of the main node alone, or the collection data of a child node alone, or the collection data of some nodes, or the collection data of all nodes.

[0059] The HDFS sub-collection module is configured to collect log files and configuration files stored on HDFS based on the received collection information. When the instruction editing module sends the collection information to the HDFS sub-collection module, the collection information includes HDFS interface information. Alternatively, the HDFS main collection module of the master node sends the HDFS interface information to the HDFS sub-collection module.

[0060] The local sub-collection module is configured to collect locally stored log files and configuration files based on the received log file and configuration file collection information. The local sub-collection module collects log files and configuration files using a shell script. For example, the script may include ssh and scp commands for collecting log files and configuration files, respectively. The data collected by the local sub-collection module can be fed back to the front end.

[0061] The sub-analysis processing module is used to perform analysis and processing based on the collected log files and configuration files according to the analysis information of the written log files and configuration files and obtain analysis results.

[0062] The above-mentioned sub-collection module and sub-analysis processing module can collect log information by scenario through the Get function (see Figure 2), analyze it through the Collect function (see Figure 2), and obtain analysis results. The analysis results may include log data, daily code paths, abnormal results, etc.

[0063] The master node of the system of the embodiment of the present disclosure also includes a problem location module, which is used to obtain abnormal results by identifying keywords or abnormal functions in the analysis results of the master node and / or child nodes, and feed back the abnormal results to the front end. The keywords include ERROR, WARN, and NULL pointer. The abnormal results include log data and log code paths. The abnormal results can trigger the process code module in the node to issue an alarm. The problem location module can work after the front end initiates a problem location instruction, or it can automatically locate the problem after the analysis and processing module obtains the analysis results.

[0064] The present disclosure also provides a distributed log collection system, which is implemented based on the above-mentioned distributed log collection and processing system. The method includes:

[0065] The front-end instruction acquisition module of the master node acquires the analysis task instruction sent by the front-end, and writes the collection information and analysis information of the log file and configuration file into the instruction editing module of the master node according to the analysis task instruction;

[0066] The instruction editing module of the master node stores the collection information and analysis information of the log files and configuration files, and sends the collection information to the collection modules of the relevant nodes, and sends the analysis information to the analysis and processing modules of the relevant nodes;

[0067] The master node and sub-node that receive the collected information collect log files and configuration files through their collection modules;

[0068] The master node and sub-node that receive the analysis information analyze and process the collected log files and configuration files through their analysis and processing modules and obtain analysis results;

[0069] The child node sends the analysis results to the master node;

[0070] The log packaging module of the master node packages the analysis results obtained at the master node and / or the analysis results obtained from the child nodes to feed back to the front end.

[0071] The method of the embodiment of the present disclosure realizes the interaction between the entire distributed system and the front end based on the front end requirements through the master node, and can efficiently collect and analyze logs.

[0072] When all CNP controllers need to collect and analyze log information at once, the front-end is equipped with a one-click processing function. The front-end sends an analysis task instruction containing the one-click processing instruction to the back-end. After all nodes complete the collection and analysis work, the log packaging module of the master node feeds back the analysis results of all nodes to the front-end.

[0073] Furthermore, when the master node also includes a problem locating module, the problem locating module obtains abnormal results by identifying keywords or abnormal functions within the analysis results of the master node and / or child nodes, and feeds the abnormal results back to the front end. The method of the disclosed embodiment also includes: the front-end instruction acquisition module of the master node obtains the problem locating instruction sent by the front end, triggering the problem locating module to operate according to the problem locating instruction; after obtaining the abnormal result, the problem locating module feeds the abnormal result back to the front end.

[0074] When problem location information for all CNP controllers needs to be collected at once, the front-end is equipped with a one-click problem location function. The front-end sends an analysis task instruction containing the one-click problem location instruction to the back-end. After all nodes complete the collection and analysis work, the log packaging module of the master node feeds back the abnormal results of all problematic nodes to the front-end.

[0075] Although several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of separate embodiments may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented in multiple implementations individually or in any suitable subcombination.

[0076] Although the subject matter has been described in language specific to methodological acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

[0077] While various embodiments of the present disclosure have been described above, the foregoing description is intended to be illustrative, non-exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or technological improvements in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A distributed log collection and processing system, comprising a plurality of CNP controllers, wherein one CNP controller is set as a master node and the remaining CNP controllers are set as child nodes; characterized in that: The master node includes: The front-end instruction acquisition module is used to receive the analysis task instruction sent by the front-end, and write the collection information and analysis information of the log file and configuration file into the instruction editing module according to the analysis task instruction; The instruction editing module is used to store the collection information and analysis information of the log files and configuration files, and send the collection information to the collection module of the relevant node, and send the analysis information to the analysis processing module of the relevant node; The HDFS main collection module is used to collect log files and configuration files stored on HDFS according to the collection information of the received log files and configuration files; The local main collection module is used to collect the log files and configuration files stored locally according to the collection information of the received log files and configuration files; The main analysis and processing module is used to analyze and process the collected log files and configuration files according to the analysis information of the written log files and configuration files and obtain the analysis results; The log packaging module is used to package the analysis results of the master node and / or sub-nodes and feed them back to the front end; The sub-nodes include: The HDFS sub-collection module is used to collect the log files and configuration files stored on the HDFS according to the collection information of the received log files and configuration files; The local sub-collection module is used to collect the log files and configuration files stored locally according to the collection information of the received log files and configuration files; The sub-analysis processing module is used to perform analysis and processing based on the collected log files and configuration files according to the analysis information of the written log files and configuration files and obtain the analysis results.

2. A distributed log collection and processing system according to claim 1, characterized in that: The instruction editing module is an XML editing module.

3. A distributed log collection and processing system according to claim 1, characterized in that: When the instruction editing module sends the collection information to the HDFS sub-collection module, the collection information includes the interface information of the HDFS; when the instruction editing module sends the collection information to the HDFS main collection module, the HDFS main collection module can directly call the HDFS interface.

4. A distributed log collection and processing system according to claim 1, characterized in that: The local main collection module and the local sub-collection module collect log files and configuration files through shell scripts.

5. A distributed log collection and processing system according to claim 1, characterized in that: The main node also includes a problem locating module, which is used to obtain abnormal results by identifying keywords or abnormal functions in the analysis results of the main node and / or the sub-nodes, and feed back the abnormal results to the front end.

6. A distributed log collection and processing system according to claim 5, characterized in that: The keywords include ERROR, WARN, and NULL pointer.

7. A distributed log collection and processing system according to claim 1, characterized in that: The front-end instruction acquisition module includes: A receiving unit, used for receiving an analysis task instruction sent by the front end; The verification unit is used to verify the received analysis task instruction, and when the verification passes, the parsing unit is triggered to work; The parsing unit is used to parse the analysis task instruction into the collection information and analysis information of the log file and the configuration file, and write the information into the instruction editing module.

8. A distributed log collection and processing system according to claim 1, characterized in that: A B / S mode is established between the system and the front end.

9. A distributed log collection and processing method, implemented based on a distributed log collection and processing system as claimed in claim 1, characterized in that: Methods include: The front-end instruction acquisition module of the master node acquires the analysis task instruction sent by the front end, and writes the collection information and analysis information of the log file and the configuration file into the instruction editing module of the master node respectively according to the analysis task instruction; The instruction editing module of the master node stores the collection information and analysis information of the log files and configuration files, and sends the collection information to the collection modules of the relevant nodes, and sends the analysis information to the analysis processing modules of the relevant nodes; The master node and sub-node that receive the collected information collect log files and configuration files through their collection modules; The main node and sub-node receiving the analysis information analyze and process the collected log files and configuration files through their analysis and processing modules and obtain analysis results; The child node sends the obtained analysis results to the master node; The log packaging module of the master node packages the analysis results obtained at the master node and / or from the child nodes to feed back to the front end.

10. A distributed log collection and processing method according to claim 9, characterized in that: The main node also includes a problem locating module, which obtains abnormal results by identifying keywords or abnormal functions in the analysis results of the main node and / or child nodes, and feeds back the abnormal results to the front end; the method also includes: the front-end instruction acquisition module of the main node obtains the problem locating instruction sent by the front end, and triggers the problem locating module to work according to the problem locating instruction; the problem locating module feeds back the abnormal results to the front end after obtaining the abnormal results.

Citation Information

Patent Citations

  • HADOOP-based log analysis system

    CN108052679A

  • Cluster log collection method and apparatus, and computer-readable storage medium

    CN109542732A

  • Distributed problem tracking system and equipment

    CN110677304A

  • Monitoring method and system based on Hadoop, storage medium and equipment

    CN114328093A

  • Distributed log collecting and processing method and system

    CN116361261A