Log-based node monitoring method and device, equipment and storage medium

Abstract

The invention discloses a node monitoring method and device based on logs, equipment and a storage medium, and relates to the technical field of big data, the method comprises the following steps: pre-analyzing log data of nodes based on a pre-analysis component, determining a pre-analysis result of at least one statistical index, the log data is obtained based on the log collector and stored to the log storage node based on the cache middleware; when it is determined that the pre-analysis result of any statistical index meets the corresponding abnormal condition, performing deep analysis on the pre-analysis result and historical log data of the node based on a deep analysis component, and determining a deep analysis result of the statistical index; and determining the running state of the node according to the monitoring protocol of the node and the deep analysis result of each statistical index. According to the technical scheme, the running state of the node is determined according to the monitoring protocol which is customized in a personalized manner and corresponds to the node and the deep analysis result which is realized by decoupling log data analysis and is used for each statistical index, and personalized multi-dimensional monitoring of the node is realized.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of big data technology, and in particular to a log-based node monitoring method, device, equipment and storage medium. Background Art

[0002] Log-based node monitoring can evaluate the operating status of the node associated with the log by counting various log indicators and provide node monitoring reports.

[0003] In the prior art, a persistently stored node log is usually scanned periodically, various indicators of the node log are counted, and the running status of the node is determined according to the statistical results.

[0004] If you need to add personalized indicators, you need to write multiple step codes such as parsing logs, aggregating indicators, and analyzing results. The development cost is high and it is not conducive to later maintenance. Summary of the invention

[0005] The present invention provides a log-based node monitoring method, device, equipment and storage medium to achieve personalized monitoring of nodes.

[0006] In a first aspect, an embodiment of the present invention provides a log-based node monitoring method, comprising:

[0007] Pre-analyzing the log data of the node based on the pre-analysis component to determine the pre-analysis result of at least one statistical indicator, wherein the log data is obtained based on the log collector and stored in the log storage node based on the cache middleware;

[0008] When it is determined that the pre-analysis result of any of the statistical indicators meets the corresponding abnormal condition, a deep analysis is performed on the pre-analysis result and the historical log data of the node based on the deep analysis component to determine the deep analysis result of the statistical indicator;

[0009] The operating status of the node is determined according to the monitoring protocol of the node and the in-depth analysis results of each of the statistical indicators.

[0010] The technical solution of an embodiment of the present invention provides a log-based node monitoring method, including: pre-analyzing the log data of the node based on a pre-analysis component to determine the pre-analysis result of at least one statistical indicator, wherein the log data is acquired based on a log collector and stored in a log storage node based on a cache middleware; when it is determined that the pre-analysis result of any of the statistical indicators meets the corresponding abnormal condition, performing a deep analysis on the pre-analysis result and the historical log data of the node based on a deep analysis component to determine the deep analysis result of the statistical indicator; and determining the operating status of the node according to the monitoring protocol of the node and the deep analysis results of each of the statistical indicators. In the above technical solution, the pre-analysis component obtains the log data of the node in real time in the database and performs pre-analysis on the log data to obtain the pre-analysis results of the statistical indicators. If the pre-analysis results of the statistical indicators meet the corresponding abnormal conditions, the pre-analysis results of the statistical indicators are sent to the deep analysis component. The deep analysis component performs a deep analysis of the statistical indicators based on the pre-analysis results of the statistical indicators and the historical log data of the nodes to determine the deep analysis results of the statistical indicators. The pre-analysis and deep analysis of the statistical indicators are achieved under the premise of decoupling the log data analysis, and the amount of data for the deep analysis is reduced, which improves the analysis efficiency. Furthermore, the operation status of the node is determined according to the personalized monitoring protocol corresponding to the node and the deep analysis results of each statistical indicator, thereby realizing multi-dimensional personalized monitoring of the node.

[0011] Furthermore, before pre-analyzing the log data of the node based on the pre-analysis component, it also includes:

[0012] Acquire the log data of the node based on the log collector;

[0013] The log data is stored in a log storage node based on the cache middleware.

[0014] Further, pre-analyzing the log data of the node based on the pre-analysis component to determine the pre-analysis result of at least one statistical indicator includes:

[0015] Determine an operation statement corresponding to a pre-analysis subcomponent corresponding to each of the statistical indicators in the pre-analysis component;

[0016] By executing each of the operation statements, the log data is pre-analyzed at each statistical indicator level to obtain the pre-analysis results corresponding to each of the statistical indicators.

[0017] Further, when it is determined that the pre-analysis result of any of the statistical indicators meets the corresponding abnormal condition, it also includes:

[0018] Determine the in-depth analysis subcomponent corresponding to the statistical indicator according to the pre-analysis subcomponent corresponding to the statistical indicator;

[0019] Accordingly, the deep analysis component performs a deep analysis on the pre-analysis result and the historical log data of the node to determine the deep analysis result of the statistical indicator, including:

[0020] Based on the deep analysis subcomponent corresponding to the statistical indicator in the deep analysis component, the pre-analysis result and the historical log data are deeply analyzed to determine the deep analysis result of the statistical indicator.

[0021] Further, determining the operation status of the node according to the monitoring protocol of the node and the in-depth analysis results of each of the statistical indicators includes:

[0022] When it is determined that the in-depth analysis results of the statistical indicators of a preset proportion meet the corresponding abnormal conditions in the monitoring protocol, it is determined that the operating state of the node is abnormal operation.

[0023] Further, when it is determined that the in-depth analysis results of the statistical indicators of the preset proportion meet the corresponding abnormal conditions in the monitoring protocol, it also includes:

[0024] Execute corresponding associated instructions in the monitoring protocol to perform abnormal analysis on the node.

[0025] In a second aspect, an embodiment of the present invention further provides a log-based node monitoring device, comprising:

[0026] A pre-analysis module, used to pre-analyze the log data of the node based on the pre-analysis component, and determine the pre-analysis result of at least one statistical indicator, wherein the log data is obtained based on the log collector and stored in the log storage node based on the cache middleware;

[0027] A deep analysis module, configured to, when it is determined that the pre-analysis result of any of the statistical indicators meets the corresponding abnormal condition, perform a deep analysis on the pre-analysis result and the historical log data of the node based on the deep analysis component to determine the deep analysis result of the statistical indicator;

[0028] An execution module is used to determine the operating status of the node according to the monitoring protocol of the node and the in-depth analysis results of each of the statistical indicators.

[0029] In a third aspect, an embodiment of the present invention further provides an electronic device, the electronic device comprising:

[0030] at least one processor; and a memory communicatively coupled to the at least one processor;

[0031] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the log-based node monitoring method as described in any one of the first aspects.

[0032] In a fourth aspect, an embodiment of the present invention further provides a storage medium comprising computer executable instructions, wherein the computer executable instructions, when executed by a computer processor, are used to execute a log-based node monitoring method as described in any one of the first aspects.

[0033] In a fifth aspect, the present application provides a computer program product, which includes computer instructions. When the computer instructions are executed on a computer, the computer executes the log-based node monitoring method provided in the first aspect.

[0034] It should be noted that the above computer instructions may be stored in whole or in part on a computer-readable storage medium. The computer-readable storage medium may be packaged together with the processor of the log-based node monitoring device, or may be packaged separately from the processor of the log-based node monitoring device, and this application does not limit this.

[0035] The description of the second, third, fourth and fifth aspects in this application can refer to the detailed description of the first aspect; and the beneficial effects of the description of the second, third, fourth and fifth aspects can refer to the beneficial effect analysis of the first aspect, which will not be repeated here.

[0036] In this application, the name of the above-mentioned log-based node monitoring device does not limit the device or functional module itself. In actual implementation, these devices or functional modules may appear with other names. As long as the functions of each device or functional module are similar to those of this application, they fall within the scope of the claims of this application and their equivalent technologies.

[0037] These and other aspects of the present application will become more apparent from the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0039] Figure 1 A flow chart of a log-based node monitoring method provided by an embodiment of the present invention;

[0040] Figure 2 A flowchart of another log-based node monitoring method provided by an embodiment of the present invention;

[0041] Figure 3 A schematic diagram of a log data collection process provided by an embodiment of the present invention;

[0042] Figure 4 A schematic diagram of a log data analysis process provided by an embodiment of the present invention;

[0043] Figure 5 A schematic diagram of the structure of a log-based node monitoring device provided by an embodiment of the present invention;

[0044] Figure 6 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0045] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0046] The term "and / or" in this article is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0047] The terms "first" and "second" and the like in the specification and drawings of this application are used to distinguish different objects, or to distinguish different processing of the same object, rather than to describe a specific order of objects.

[0048] In addition, the terms "including" and "having" and any variations thereof mentioned in the description of this application are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include other steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or devices.

[0049] It should be mentioned before discussing the exemplary embodiments in more detail that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe the operations (or steps) as sequential processes, many of the operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but can also have additional steps not included in the accompanying drawings. The process can correspond to methods, functions, procedures, subroutines, subprograms, etc. In addition, the embodiments in the present invention and the features in the embodiments can be combined with each other without conflict.

[0050] It should be noted that, in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0051] In the description of the present application, unless otherwise specified, “plurality” means two or more.

[0052] At present, the conventional solution for log monitoring is to perform regular detection on the node logs in persistent storage. The specific implementation plan is as follows: collect node logs through the log collection component, store node logs persistently through the log storage node, perform regular scanning on the persistently stored node logs, count various indicators of the logs, and determine the running status of the node based on the indicators.

[0053] The existing technology has the following defects: low fault tolerance: when the node scale is large or the concurrent volume of data storage increases suddenly, the number of requests sent by the log collection component to the log storage node is large, and the log storage node will trigger the current limiting mechanism and reject some log storage requests, resulting in log loss; high cost: for each statistical indicator added, it is necessary to write code for multiple steps such as parsing logs, aggregating indicators, and analyzing results. The development cost is high and is not conducive to later maintenance.

[0054] Therefore, the present application proposes a log-based node monitoring method to achieve personalized monitoring of nodes and improve the reliability of monitoring.

[0055] The log-based node monitoring method proposed in this application will be described in detail below with reference to diagrams and embodiments.

[0056] Figure 1A flow chart of a log-based node monitoring method provided by an embodiment of the present invention is provided. This embodiment is applicable to situations where reliable and personalized monitoring of nodes is required. The method can be executed by a log-based node monitoring device, such as Figure 1 As shown, the specific steps include:

[0057] Step 110: Pre-analyze the log data of the node based on the pre-analysis component to determine a pre-analysis result of at least one statistical indicator.

[0058] In this application, in order to alleviate the pressure on the log storage node, the log collector does not directly send a persistent storage request to the log storage node, but introduces streaming computing. The log collector first sends the collected log data to the cache middleware, and then sends the log data to the log storage node based on the cache middleware.

[0059] The log storage node is composed of at least one database, and the data sending rate of the cache middleware to send log data to the log storage node does not exceed a set threshold.

[0060] In order to further alleviate the pressure on the database, the storage and analysis of log data can be separated. In addition, the analysis of log data can be decoupled into pre-analysis and in-depth analysis to improve analysis efficiency.

[0061] Specifically, the execution of a business requires the participation of multiple nodes. The nodes can be understood as microservice gateways, which have independent and logically complete functions, such as user login, order creation, and comment likes. In order to conduct more comprehensive monitoring of the nodes corresponding to the business, the statistical indicator corresponding to the log data can be at least one, that is, the pre-analysis component can pre-analyze the log data of the node and determine the pre-analysis result of at least one statistical indicator. Specifically, the pre-analysis component can obtain the log data corresponding to the node from the database in real time. The pre-analysis component can pre-analyze the log data in multiple dimensions to obtain the pre-analysis results of each statistical indicator.

[0062] In the embodiment of the present invention, the log data of the node is pre-analyzed by the pre-analysis component to determine the pre-analysis result of the statistical index, and the pre-analysis of the statistical index is achieved under the premise of decoupling the log data analysis.

[0063] Step 120: When it is determined that the pre-analysis result of any of the statistical indicators meets the corresponding abnormal condition, a deep analysis is performed on the pre-analysis result and the historical log data of the node based on a deep analysis component to determine the deep analysis result of the statistical indicator.

[0064] The pre-analysis component also has built-in abnormal conditions for each statistical indicator. After determining the pre-analysis results of each statistical indicator, it can be determined whether the pre-analysis results of each statistical indicator meet the corresponding abnormal conditions. If it is determined that the pre-analysis results of the statistical indicator meet the corresponding abnormal conditions, the statistical indicator can be further analyzed in depth.

[0065] Specifically, when it is determined that the pre-analysis result of the statistical indicator meets the corresponding abnormal condition, the pre-analysis component can send the pre-analysis result of the statistical indicator to the deep analysis component. The deep analysis component can perform a deep analysis of the statistical indicator in combination with the pre-analysis result of the statistical indicator and the historical log data of the node to determine the deep analysis result of the statistical indicator.

[0066] In an embodiment of the present invention, when it is determined that the pre-analysis results of the statistical indicators meet the corresponding abnormal conditions, an in-depth analysis is continued to be performed on them to determine the in-depth analysis results. The in-depth analysis of the statistical indicators is achieved under the premise of decoupling the log data analysis, and the amount of data for the in-depth analysis is reduced, thereby improving the analysis efficiency.

[0067] Step 130: Determine the operating status of the node according to the monitoring protocol of the node and the in-depth analysis results of each of the statistical indicators.

[0068] Among them, the node monitoring protocol is set in advance according to actual needs, and the protocol format is: loginspect: / / inspect.name=indicator name&inspect.threshold=indicator threshold&inspect.id=indicator sequence number&insect.order=statistical order&inspect.command=associated command.

[0069] Specifically, the monitoring protocol reflects the abnormal conditions of each statistical indicator through the indicator threshold, that is, if the deep analysis result of the statistical indicator exceeds the corresponding indicator threshold, it indicates that the statistical indicator is abnormal. Therefore, when the monitoring protocol corresponding to the statistical indicator is executed based on the statistical sequence, the deep analysis result of the statistical indicator is compared with the abnormal conditions of the statistical indicator in the monitoring protocol. If the deep analysis result of the statistical indicator exceeds the corresponding indicator threshold, it indicates that the statistical indicator is abnormal.

[0070] Furthermore, the operation status of the node may be determined according to the number of abnormal statistical indicators, that is, if the number of abnormal statistical indicators meets a preset requirement, the operation status of the node is determined to be abnormal.

[0071] Furthermore, when it is determined that the operation status of a node is abnormal based on the statistical indicators, the associated instructions corresponding to the statistical indicators in the monitoring protocol can be executed to obtain more operation information of the node, so as to facilitate the analysis of the cause of the node abnormality.

[0072] In the embodiment of the present invention, the operation status of the node is determined through a personalized monitoring protocol and in-depth analysis results of various statistical indicators, thereby realizing personalized monitoring of the node.

[0073] The log-based node monitoring method provided by an embodiment of the present invention includes: pre-analyzing the log data of the node based on a pre-analysis component to determine the pre-analysis result of at least one statistical indicator, wherein the log data is acquired based on a log collector and stored in a log storage node based on a cache middleware; when it is determined that the pre-analysis result of any of the statistical indicators meets the corresponding abnormal condition, performing a deep analysis on the pre-analysis result and the historical log data of the node based on a deep analysis component to determine the deep analysis result of the statistical indicator; and determining the operating status of the node according to the monitoring protocol of the node and the deep analysis results of each of the statistical indicators. In the above technical solution, the pre-analysis component obtains the log data of the node in real time in the database and pre-analyzes the log data to obtain the pre-analysis results of the statistical indicators. If the pre-analysis results of the statistical indicators meet the corresponding abnormal conditions, the pre-analysis results of the statistical indicators are sent to the deep analysis component. The deep analysis component combines the pre-analysis results of the statistical indicators and the historical log data of the nodes to perform a deep analysis of the statistical indicators to determine the deep analysis results of the statistical indicators. The pre-analysis and deep analysis of the statistical indicators are achieved under the premise of decoupling the log data analysis, and the amount of data for the deep analysis is reduced, which improves the analysis efficiency. Furthermore, the operating status of the node is determined according to the personalized monitoring protocol corresponding to the node and the deep analysis results of each statistical indicator, so as to achieve multi-dimensional monitoring of the node and personalized monitoring of the node.

[0074] Figure 2 This is a flow chart of another log-based node monitoring method provided by an embodiment of the present invention. This embodiment is specific based on the above embodiment. Figure 2 As shown, in this embodiment, the method may further include:

[0075] Step 210: acquiring the log data of the node based on the log collector; and storing the log data in a log storage node based on the cache middleware.

[0076] Figure 3 A schematic diagram of the log data collection process provided by an embodiment of the present invention is shown in FIG. Figure 3 As shown, each node corresponding to the business is provided with a log collector, which can be a general collector or a customized collector, which collects the log data of the node in real time and sends the collected log data to the log storage node through the cache middleware. In addition, the number of cache middlewares can be determined according to the amount of log data of each node.

[0077] Specifically, the log data of the node can be obtained in real time based on the log collector corresponding to the node. The log collector can send the log data to the cache middleware in real time. The cache middleware can temporarily store the log data and send the log data to the log storage node. In addition, in order to relieve the pressure of the log storage node, the data sending rate of the cache middleware to the log storage node does not exceed the set threshold.

[0078] In the embodiment of the present invention, by introducing cache middleware as a streaming computing component, safe and stable acquisition of log data is achieved, thereby avoiding the loss of log data.

[0079] Step 220: Pre-analyze the log data of the node based on the pre-analysis component to determine a pre-analysis result of at least one statistical indicator.

[0080] In one implementation, step 220 may specifically include:

[0081] Determine the operation statement corresponding to the pre-analysis subcomponent corresponding to each of the statistical indicators in the pre-analysis component; perform pre-analysis on the log data at the level of each statistical indicator by executing each of the operation statements to obtain the pre-analysis result corresponding to each of the statistical indicators.

[0082] Specifically, the pre-analysis component includes a plurality of pre-analysis sub-components, each of which has pre-edited operation statements for pre-analyzing each statistical indicator, and one pre-analysis sub-component is used to analyze one statistical indicator. After the pre-analysis component obtains the log data, each pre-analysis sub-component executes the corresponding operation statement to pre-analyze the log data at the statistical indicator level, and obtains the pre-analysis results corresponding to each statistical indicator.

[0083] For example, when the log data structure is: Code: execution result code (0-normal, non-0-abnormal)#APP: application name#Area: business execution area (Beijing, Shanghai...), and the log data is: Code:0#App:app1#Area:area1, Code:1#App:app1#Area:area2, Code:0#App:app1#Area:area1, the operation statement NOT Code:0 is used to count abnormal business, DISTINCT Area is used to count business execution areas, and count(*)Where App:app1 is used to count the usage of app1 application for the business.

[0084] In the embodiment of the present invention, the pre-analysis subcomponents constituting the pre-analysis component pre-analyze the log data of the node to determine the pre-analysis results of the statistical indicators, thereby achieving the pre-analysis of the statistical indicators under the premise of decoupling the log data analysis.

[0085] Step 230: Determine the deep analysis subcomponent corresponding to the statistical indicator according to the pre-analysis subcomponent corresponding to the statistical indicator.

[0086] Figure 4 A schematic diagram of the log data analysis process provided for an embodiment of the present invention, as shown in Figure 4, is a diagram in which the analysis of log data is implemented through a pre-analysis component and a deep analysis component. The pre-analysis component includes multiple pre-analysis sub-components, and the deep analysis component also includes multiple deep analysis sub-components. The pre-analysis sub-components are connected to the corresponding deep analysis sub-components through a pre-analysis application programming interface (Application Programming Interfac, API).

[0087] Therefore, the deep analysis subcomponent corresponding to the statistical indicator can be determined according to the pre-analysis subcomponent corresponding to the statistical indicator. Specifically, the deep analysis component connected to the pre-analysis subcomponent corresponding to the statistical indicator through the pre-analysis API can be determined as the deep analysis subcomponent corresponding to the statistical indicator.

[0088] In the embodiment of the present invention, the determination of the deep analysis subcomponent corresponding to the statistical indicator is realized.

[0089] Step 240: When it is determined that the pre-analysis result of any of the statistical indicators meets the corresponding abnormal condition, a deep analysis is performed on the pre-analysis result and the historical log data of the node based on a deep analysis component to determine the deep analysis result of the statistical indicator.

[0090] In one implementation, step 240 may specifically include:

[0091] When it is determined that the pre-analysis result of any statistical indicator meets the corresponding abnormal condition, the pre-analysis result and the historical log data are deeply analyzed based on the deep analysis subcomponent corresponding to the statistical indicator in the deep analysis component to determine the deep analysis result of the statistical indicator.

[0092] Specifically, when it is determined that the pre-analysis result of the statistical indicator meets the corresponding abnormal condition, the statistical indicator pre-analysis component can send the pre-analysis result of the statistical indicator to the deep analysis component corresponding to the statistical indicator based on the pre-analysis API. The deep analysis component can perform a deep analysis of the statistical indicator in combination with the pre-analysis result of the statistical indicator and the historical log data of the node to determine the deep analysis result of the statistical indicator. For example, the fluctuation and change rate of the statistical indicator can be determined.

[0093] For example, the deep analysis component can calculate the failure rate based on the node's call volume and business failure volume, and conduct deep analysis in combination with historical log data to monitor fluctuations in the failure rate.

[0094] In an embodiment of the present invention, when it is determined that the pre-analysis results of the statistical indicators meet the corresponding abnormal conditions, an in-depth analysis is continued to be performed on them to determine the in-depth analysis results. The in-depth analysis of the statistical indicators is achieved under the premise of decoupling the log data analysis, thereby reducing the amount of data for the in-depth analysis and improving the analysis efficiency.

[0095] Step 250: Determine the operating status of the node according to the monitoring protocol of the node and the in-depth analysis results of each of the statistical indicators.

[0096] In one implementation, step 250 may specifically include:

[0097] When it is determined that the in-depth analysis results of the preset proportion of the statistical indicators meet the corresponding abnormal conditions in the monitoring protocol, the operating status of the node is determined to be abnormal operation, and the corresponding associated instructions in the monitoring protocol are executed to perform abnormal analysis on the node.

[0098] Specifically, the monitoring protocol reflects the abnormal conditions of each statistical indicator through the indicator threshold, that is, if the deep analysis result of the statistical indicator exceeds the corresponding indicator threshold, it indicates that the statistical indicator is abnormal. Therefore, when the monitoring protocol corresponding to the statistical indicator is executed based on the statistical sequence, the deep analysis result of the statistical indicator and the abnormal conditions of the statistical indicator in the monitoring protocol are determined and compared. If the deep analysis result of the statistical indicator exceeds the corresponding indicator threshold, it indicates that the statistical indicator is abnormal.

[0099] Furthermore, the number of abnormal statistical indicators can be determined. If the number of abnormal statistical indicators meets the preset requirements, for example, the number of abnormal statistical indicators is greater than the total number of statistical indicators of a preset ratio, the operation state of the node is determined to be abnormal. At this time, the associated instructions corresponding to the statistical indicators in the monitoring protocol can be executed to obtain more operation information of the node, such as node configuration and operation environment.

[0100] The protocol corresponding to the number of null pointers in the monitoring protocol is: logininspect: / / inspect.name=Null pointer exception&inspect.threshold=0&inspect.id=1&insect.order=1&inspect.command=log, which means that the number of null pointers of the node is monitored first. When the number of null pointers is greater than 0, a null pointer exception is determined. At this time, the original log of the node can be obtained.

[0101] In the embodiment of the present invention, the operation status of the node is determined through a personalized monitoring protocol and in-depth analysis results of various statistical indicators, thereby realizing personalized monitoring of the node.

[0102] The log-based node monitoring method provided by an embodiment of the present invention includes: obtaining the log data of the node based on the log collector; storing the log data to the log storage node based on the cache middleware; pre-analyzing the log data of the node based on the pre-analysis component to determine the pre-analysis result of at least one statistical indicator; determining the deep analysis subcomponent corresponding to the statistical indicator according to the pre-analysis subcomponent corresponding to the statistical indicator; when it is determined that the pre-analysis result of any of the statistical indicators meets the corresponding abnormal condition, performing deep analysis on the pre-analysis result and the historical log data of the node based on the deep analysis component to determine the deep analysis result of the statistical indicator; and determining the operating status of the node according to the monitoring protocol of the node and the deep analysis results of each of the statistical indicators. The above technical solution can firstly introduce a cache middleware as a streaming computing component to obtain the log data of the node safely and stably to avoid the loss of log data. Secondly, the pre-analysis component can obtain the log data of the node in real time in the database, and pre-analyze the log data of the node through the pre-analysis subcomponents that constitute the pre-analysis component to determine the pre-analysis results of each statistical indicator, and obtain the pre-analysis results of each statistical indicator. If the pre-analysis results of the statistical indicators meet the corresponding abnormal conditions, the pre-analysis results of the statistical indicators are sent to the deep analysis component. The deep analysis component combines the pre-analysis results of the statistical indicators and the historical log data of the node to perform a deep analysis on the statistical indicators to determine the deep analysis results of the statistical indicators. The pre-analysis and deep analysis of the statistical indicators are realized under the premise of decoupling the log data analysis, which reduces the amount of data for the deep analysis, improves the analysis efficiency, and reduces the dimensional complexity of the log data analysis, further reducing the analysis cost. Then, the running status of the node is determined according to the personalized customized monitoring protocol corresponding to the node and the deep analysis results of each statistical indicator, and multi-dimensional personalized monitoring of the node is realized.

[0103] Figure 5 A schematic diagram of a log-based node monitoring device provided in an embodiment of the present invention is provided. The device can be applied to situations where reliable and personalized monitoring of nodes is required. The device can be implemented by software and / or hardware and is generally integrated in an electronic device, such as a computer device.

[0104] like Figure 5 As shown, the device comprises:

[0105] A pre-analysis module 510 is used to pre-analyze the log data of the node based on the pre-analysis component to determine the pre-analysis result of at least one statistical indicator, wherein the log data is obtained based on the log collector and stored in the log storage node based on the cache middleware;

[0106] A deep analysis module 520 is used to perform a deep analysis on the pre-analysis result and the historical log data of the node based on the deep analysis component to determine the deep analysis result of the statistical indicator when it is determined that the pre-analysis result of any of the statistical indicators meets the corresponding abnormal condition;

[0107] The execution module 530 is used to determine the operation status of the node according to the monitoring protocol of the node and the in-depth analysis results of each of the statistical indicators.

[0108] The log-based node monitoring device provided in this embodiment performs pre-analysis on the log data of the node based on a pre-analysis component to determine the pre-analysis result of at least one statistical indicator, wherein the log data is acquired based on a log collector and stored in a log storage node based on a cache middleware; when it is determined that the pre-analysis result of any statistical indicator meets the corresponding abnormal condition, a deep analysis is performed on the pre-analysis result and the historical log data of the node based on a deep analysis component to determine the deep analysis result of the statistical indicator; and the operating status of the node is determined according to the monitoring protocol of the node and the deep analysis results of each statistical indicator. In the above technical solution, the pre-analysis component obtains the log data of the node in real time in the database and pre-analyzes the log data to obtain the pre-analysis results of the statistical indicators. If the pre-analysis results of the statistical indicators meet the corresponding abnormal conditions, the pre-analysis results of the statistical indicators are sent to the deep analysis component. The deep analysis component combines the pre-analysis results of the statistical indicators and the historical log data of the nodes to perform a deep analysis of the statistical indicators to determine the deep analysis results of the statistical indicators. The pre-analysis and deep analysis of the statistical indicators are achieved under the premise of decoupling the log data analysis. In addition, the amount of data for the deep analysis is reduced, the analysis efficiency is improved, and the analysis cost is reduced. Furthermore, the operation status of the node is determined according to the personalized monitoring protocol corresponding to the node and the deep analysis results of each statistical indicator, thereby realizing multi-dimensional personalized monitoring of the node.

[0109] Based on the above embodiment, the device further includes:

[0110] The acquisition module is used to acquire the log data of the node based on the log collector; and store the log data in a log storage node based on the cache middleware.

[0111] Based on the above embodiment, the pre-analysis module 510 is specifically used for:

[0112] Determine the operation statement corresponding to the pre-analysis subcomponent corresponding to each of the statistical indicators in the pre-analysis component; perform pre-analysis on the log data at the level of each statistical indicator by executing each of the operation statements to obtain the pre-analysis result corresponding to each of the statistical indicators.

[0113] Based on the above embodiment, the device further includes:

[0114] A determination module, used to determine the in-depth analysis subcomponent corresponding to the statistical indicator according to the pre-analysis subcomponent corresponding to the statistical indicator;

[0115] Accordingly, the depth analysis module 520 is specifically used for:

[0116] Based on the deep analysis subcomponent corresponding to the statistical indicator in the deep analysis component, the pre-analysis result and the historical log data are deeply analyzed to determine the deep analysis result of the statistical indicator.

[0117] Based on the above embodiment, the execution module 530 is specifically used for:

[0118] When it is determined that the in-depth analysis results of the preset proportion of the statistical indicators meet the corresponding abnormal conditions in the monitoring protocol, the operating status of the node is determined to be abnormal operation, and the corresponding associated instructions in the monitoring protocol are executed to perform abnormal analysis on the node.

[0119] The log-based node monitoring device provided in the embodiment of the present invention can execute the log-based node monitoring method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the log-based node monitoring method.

[0120] It is worth noting that in the above-mentioned embodiment of the log-based node monitoring device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0121] Figure 6 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. Figure 6 A block diagram of an exemplary electronic device 6 suitable for implementing embodiments of the present invention is shown. Figure 6 The electronic device 6 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0122] like Figure 6 As shown, the electronic device 6 is in the form of a general purpose computing electronic device. The components of the electronic device 6 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 connecting different system components (including the system memory 28 and the processing unit 16).

[0123] Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. By way of example, these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0124] The electronic device 6 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device 6, including volatile and non-volatile media, removable and non-removable media.

[0125] The system memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The electronic device 6 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be used to read and write non-removable, non-volatile magnetic media ( Figure 6 not shown, usually called a "hard drive"). Although Figure 6 Not shown in the figure, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus 18 via one or more data medium interfaces. The system memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present invention.

[0126] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in system memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. Program modules 42 generally perform the functions and / or methods of the embodiments described herein.

[0127] The electronic device 6 may also communicate with one or more external devices 14 (e.g., keyboards, pointing devices, displays 24, etc.), one or more devices that enable a user to interact with the electronic device 6, and / or any device that enables the electronic device 6 to communicate with one or more other computing devices (e.g., network cards, modems, etc.). Such communication may be performed via an input / output (I / O) interface 22. Furthermore, the electronic device 6 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 20. Figure 6 As shown, the network adapter 20 communicates with other modules of the electronic device 6 via the bus 18. It should be understood that although Figure 6 Not shown, other hardware and / or software modules may be used in conjunction with the electronic device 6, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0128] The processing unit 16 executes various functional applications and page displays by running the program stored in the system memory 28, for example, implementing the log-based node monitoring method provided in the embodiment of the present invention, which includes:

[0129] Pre-analyzing the log data of the node based on the pre-analysis component to determine the pre-analysis result of at least one statistical indicator, wherein the log data is obtained based on the log collector and stored in the log storage node based on the cache middleware;

[0130] When it is determined that the pre-analysis result of any of the statistical indicators meets the corresponding abnormal condition, a deep analysis is performed on the pre-analysis result and the historical log data of the node based on the deep analysis component to determine the deep analysis result of the statistical indicator;

[0131] The operating status of the node is determined according to the monitoring protocol of the node and the in-depth analysis results of each of the statistical indicators.

[0132] Of course, those skilled in the art can understand that the processor can also implement the technical solution of the log-based node monitoring method provided by any embodiment of the present invention.

[0133] An embodiment of the present invention provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, for example, a log-based node monitoring method provided by an embodiment of the present invention is implemented. The method includes:

[0134] Pre-analyzing the log data of the node based on the pre-analysis component to determine the pre-analysis result of at least one statistical indicator, wherein the log data is obtained based on the log collector and stored in the log storage node based on the cache middleware;

[0135] When it is determined that the pre-analysis result of any of the statistical indicators meets the corresponding abnormal condition, a deep analysis is performed on the pre-analysis result and the historical log data of the node based on the deep analysis component to determine the deep analysis result of the statistical indicator;

[0136] The operating status of the node is determined according to the monitoring protocol of the node and the in-depth analysis results of each of the statistical indicators.

[0137] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to: an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0138] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0139] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0140] Computer program code for performing the operations of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0141] It should be understood by those skilled in the art that the modules or steps of the present invention described above can be implemented by a general-purpose computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, optionally, they can be implemented by a program code executable by a computer device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0142] In addition, the acquisition, storage, use, and processing of data in the technical solution of the present invention are in compliance with the relevant provisions of national laws and regulations.

[0143] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A log-based node monitoring method, characterized in that: include: Pre-analyzing the log data of the node based on the pre-analysis component to determine the pre-analysis result of at least one statistical indicator, wherein the log data is obtained based on the log collector and stored in the log storage node based on the cache middleware; When it is determined that the pre-analysis result of any of the statistical indicators meets the corresponding abnormal condition, a deep analysis is performed on the pre-analysis result and the historical log data of the node based on the deep analysis component to determine the deep analysis result of the statistical indicator; The operating status of the node is determined according to the monitoring protocol of the node and the in-depth analysis results of each of the statistical indicators.

2. The log-based node monitoring method according to claim 1, characterized in that: Before pre-analyzing the log data of the node based on the pre-analysis component, it also includes: Acquire the log data of the node based on the log collector; The log data is stored in a log storage node based on the cache middleware.

3. The log-based node monitoring method according to claim 1, characterized in that: Pre-analyzing the log data of the node based on the pre-analysis component to determine a pre-analysis result of at least one statistical indicator, including: Determine an operation statement corresponding to a pre-analysis subcomponent corresponding to each of the statistical indicators in the pre-analysis component; By executing each of the operation statements, the log data is pre-analyzed at each statistical indicator level to obtain the pre-analysis results corresponding to each of the statistical indicators.

4. The log-based node monitoring method according to claim 1, characterized in that: Before performing in-depth analysis on the pre-analysis result and the historical log data of the node based on the in-depth analysis component, the method further includes: Determine the in-depth analysis subcomponent corresponding to the statistical indicator according to the pre-analysis subcomponent corresponding to the statistical indicator; Accordingly, the deep analysis component performs a deep analysis on the pre-analysis result and the historical log data of the node to determine the deep analysis result of the statistical indicator, including: Based on the deep analysis subcomponent corresponding to the statistical indicator in the deep analysis component, the pre-analysis result and the historical log data are deeply analyzed to determine the deep analysis result of the statistical indicator.

5. The log-based node monitoring method according to claim 1, characterized in that: Determining the operation status of the node according to the monitoring protocol of the node and the in-depth analysis results of each of the statistical indicators includes: When it is determined that the in-depth analysis results of the statistical indicators of a preset proportion meet the corresponding abnormal conditions in the monitoring protocol, it is determined that the operating state of the node is abnormal operation.

6. The log-based node monitoring method according to claim 1, characterized in that: When it is determined that the in-depth analysis results of the statistical indicators of the preset proportion meet the corresponding abnormal conditions in the monitoring protocol, it also includes: Execute corresponding associated instructions in the monitoring protocol to perform abnormal analysis on the node.

7. A log-based node monitoring device, characterized in that: include: A pre-analysis module, used to pre-analyze the log data of the node based on the pre-analysis component, and determine the pre-analysis result of at least one statistical indicator, wherein the log data is obtained based on the log collector and stored in the log storage node based on the cache middleware; A deep analysis module, configured to, when it is determined that the pre-analysis result of any of the statistical indicators meets the corresponding abnormal condition, perform a deep analysis on the pre-analysis result and the historical log data of the node based on the deep analysis component to determine the deep analysis result of the statistical indicator; An execution module is used to determine the operating status of the node according to the monitoring protocol of the node and the in-depth analysis results of each of the statistical indicators.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively coupled to the at least one processor; The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the log-based node monitoring method as described in any one of claims 1-6.

9. A storage medium containing computer executable instructions, characterized in that: When the computer executable instructions are executed by a computer processor, they are used to execute the log-based node monitoring method as described in any one of claims 1-6.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the computer program implements the log-based node monitoring method according to any one of claims 1 to 6.

Images