Log data processing method and device, electronic equipment and medium

Abstract

The invention provides a log data processing method and device, electronic equipment and a medium. The problems that when log data is processed, the efficiency is too low, the monitoring and alarming capacity is lacked, the hysteresis quality is too high, and many computing resources are needed are effectively solved. The method comprises the steps that the cleaning module cleans log data in the message queue to obtain a cleaning result; the cleaning module sends the cleaning result to a monitoring module, so that the monitoring module obtains a field extraction result; the monitoring module performs different processing on the field extraction result based on processing modes corresponding to different dimensions to obtain a corresponding dimension processing result, and determines an alarm judgment condition of the corresponding dimension based on the dimension corresponding to the dimension processing result; and the monitoring module judges the dimension processing result by adopting the alarm judgment condition of the corresponding dimension to obtain a judgment result, and alarms based on the judgment result.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method, device, electronic device and medium for processing log data. Background Art

[0002] Currently, when operation and maintenance personnel process log data, they use command lines to process and analyze logs. However, the efficiency of log data processing is too low and the problem cannot be reflected intuitively. Then, Shell scripts are used to count and analyze the domain name status codes in the log data, but they do not have the ability to monitor and alarm the status codes, and there is a long lag in discovering potential risks or abnormal situations. On the management machine, commands are used to obtain logs from all proxy nodes and clean the logs, and then they are transmitted back to the management machine for secondary processing. Finally, various status codes of the domain names are counted. Correspondingly, more computing resources are required, which will put a greater test on the resources on the management terminal. Summary of the invention

[0003] In view of this, the purpose of the present application is to provide a method, device, electronic device and medium for processing log data, which effectively solve the problems of low efficiency, lack of monitoring and alarm capabilities, strong lag and large number of computing resources required when log data is processed.

[0004] In a first aspect, an embodiment of the present application provides a method for processing log data, which is applied to a processing system, wherein the processing system includes a collection module, a cleaning module, and a monitoring module, and the method includes:

[0005] The collection module collects the log data generated by nginx and pushes the log data to the message queue, so that the cleaning module cleans the log data in the message queue to obtain a cleaning result;

[0006] The cleaning module sends the cleaning result to the monitoring module, so that the monitoring module performs field extraction based on the cleaning result to obtain a field extraction result;

[0007] The monitoring module performs different processing on the field extraction results based on processing methods corresponding to different dimensions to obtain corresponding dimension processing results, so as to determine the alarm judgment condition of the corresponding dimension based on the dimension corresponding to the dimension processing result;

[0008] The monitoring module uses the alarm judgment condition of the corresponding dimension to judge the dimension processing result to obtain a judgment result, so as to issue an alarm based on the judgment result.

[0009] In combination with the first aspect, an embodiment of the present application provides a first possible implementation of the first aspect, wherein the monitoring module performs different processing on the field extraction result based on processing methods corresponding to different dimensions to obtain corresponding dimension processing results, including:

[0010] The monitoring module confirms the dimension of the field extraction result processing based on the log data;

[0011] The monitoring module processes the field extraction result based on the processing method corresponding to the dimension to obtain a dimension processing result.

[0012] In combination with the first aspect, the embodiment of the present application provides a second possible implementation of the first aspect, wherein the dimension includes TPS calculation;

[0013] The monitoring module processes the field extraction result based on the processing method corresponding to the dimension to obtain a dimension processing result, including:

[0014] The monitoring module obtains a TPS indicator by matching multiple fields in the field extraction result with a preset field combination;

[0015] The monitoring module calculates target derived data of the TPS indicator to obtain a dimensional processing result based on the target derived data.

[0016] In combination with the first aspect, the embodiment of the present application provides a third possible implementation of the first aspect, wherein the dimension includes time-consuming distribution calculation;

[0017] The monitoring module processes the field extraction result based on the processing method corresponding to the dimension to obtain a dimension processing result, including:

[0018] The monitoring module extracts the target field in the log data based on the processing method of the corresponding dimension, and determines the time consumption index of the corresponding dimension;

[0019] The monitoring module counts the frequency of the target field corresponding to the time-consuming indicator to obtain the dimension processing result based on the frequency.

[0020] In combination with the first aspect, the embodiment of the present application provides a fourth possible implementation of the first aspect, wherein the dimension includes flow calculation;

[0021] The monitoring module processes the field extraction result based on the processing method corresponding to the dimension to obtain a dimension processing result, including:

[0022] The monitoring module extracts corresponding byte quantities based on multiple fields in the field extraction result and determines corresponding traffic indicators;

[0023] The monitoring module calculates the total number of bytes corresponding to the traffic indicator to obtain a corresponding dimensional processing result based on the total number of bytes.

[0024] In combination with the first aspect, an embodiment of the present application provides a fifth possible implementation of the first aspect, wherein the monitoring module calculates the target derived data of the TPS indicator to obtain a dimension processing result based on the target derived data, including:

[0025] The monitoring module generates a function call instruction based on the TPS indicator to call a calculation function based on the function call instruction;

[0026] The calculation function calculates the TPS indicator based on the function call instruction to obtain target derived data.

[0027] In combination with the first aspect, an embodiment of the present application provides a sixth possible implementation of the first aspect, wherein the monitoring module uses the alarm judgment condition of the corresponding dimension to judge the dimension processing result to obtain a judgment result, and issues an alarm based on the judgment result, including:

[0028] The monitoring module determines whether the dimension processing result meets the alarm judgment condition of the corresponding dimension;

[0029] If it is consistent, the monitoring module fuses the judgment results to obtain a fusion result, and issues an alarm based on the fusion result.

[0030] In a second aspect, an embodiment of the present application provides a log data processing device, which is applied to a processing system, wherein the processing system includes a collection module, a cleaning module, and a monitoring module, and the device includes:

[0031] A collection module is used for collecting log data generated by nginx and pushing the log data to a message queue, so that the cleaning module cleans the log data in the message queue to obtain a cleaning result;

[0032] A sending module, used for the cleaning module to send the cleaning result to the monitoring module, so that the monitoring module performs field extraction based on the cleaning result to obtain a field extraction result;

[0033] A processing module, used for the monitoring module to perform different processing on the field extraction results based on processing methods corresponding to different dimensions to obtain corresponding dimension processing results, so as to determine the alarm judgment condition of the corresponding dimension based on the dimension corresponding to the dimension processing result;

[0034] The alarm module is used for the monitoring module to use the alarm judgment condition of the corresponding dimension to judge the dimension processing result to obtain a judgment result, so as to issue an alarm based on the judgment result.

[0035] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, the steps of any one of the log data processing methods are performed.

[0036] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of any one of the log data processing methods are executed.

[0037] The embodiment of the present application provides an application processing system, the processing system includes a collection module, a cleaning module, and a monitoring module. The method first collects log data generated by nginx through the collection module, and pushes the log data to a message queue, so that the cleaning module cleans the log data in the message queue to obtain a cleaning result; secondly, the cleaning module sends the cleaning result to the monitoring module, so that the monitoring module extracts fields based on the cleaning result to obtain a field extraction result; then the monitoring module performs different processing on the field extraction result based on the processing methods corresponding to different dimensions to obtain corresponding dimension processing results, so as to determine the alarm judgment condition of the corresponding dimension based on the dimension corresponding to the dimension processing result; the monitoring module uses the alarm judgment condition of the corresponding dimension to judge the dimension processing result to obtain a judgment result, so as to issue an alarm based on the judgment result, and based on the cleaning module and the monitoring module, the efficiency of log data when being processed is improved, the effect of having monitoring and alarm capabilities is achieved, and the timely processing of log data is achieved, avoiding too strong lag and avoiding the occurrence of problems such as a large number of computing resources of the management terminal required to be used. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 A schematic diagram showing a flow chart of a first log data processing method provided in an embodiment of the present application;

[0040] Figure 2 A schematic diagram showing the effect of TPS calculation dimension provided by an embodiment of the present application is shown;

[0041] Figure 3 A schematic diagram showing the effect of flow calculation dimensions provided by an embodiment of the present application is shown;

[0042] Figure 4 A schematic diagram showing the structure of a first log data processing device provided in an embodiment of the present application is shown;

[0043] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0044] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of explanation and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in this application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can be implemented out of sequence, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart under the guidance of the content of the present application, or remove one or more operations from the flowchart.

[0045] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0046] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.

[0047] At present, the command line is used for log processing and analysis, which results in low efficiency in log data processing and cannot directly reflect the problem. Then, Shell scripts are used to count and analyze the domain name status codes in the log data, but they do not have the ability to monitor and alarm the status codes, and there is a long lag in discovering potential risks or abnormal situations. On the management machine, commands are used to obtain logs from all proxy nodes and clean the logs, and then they are transmitted back to the management machine for secondary processing. Finally, various status codes of the domain names are counted. Correspondingly, more computing resources are required, which will put a greater test on the resources on the management terminal.

[0048] Based on this, the embodiments of the present application provide a method, device, electronic device and medium for processing log data, which are described below through embodiments.

[0049] Example 1

[0050] To facilitate understanding of this embodiment, a method for processing log data disclosed in an embodiment of this application is first described in detail. Figure 1 A flow chart of a method for processing log data is shown in the figure. A method for processing log data provided by the present application is applied to a processing system. The processing system includes a collection module, a cleaning module, and a monitoring module. The method includes:

[0051] S101, the collection module collects the log data generated by nginx, and pushes the log data to the message queue, so that the cleaning module cleans the log data in the message queue to obtain a cleaning result;

[0052] S102, the cleaning module sends the cleaning result to the monitoring module, so that the monitoring module performs field extraction based on the cleaning result to obtain a field extraction result;

[0053] S103, the monitoring module performs different processing on the field extraction result based on the processing methods corresponding to different dimensions to obtain corresponding dimension processing results, so as to determine the alarm judgment condition of the corresponding dimension based on the dimension corresponding to the dimension processing result;

[0054] S104: The monitoring module uses the alarm judgment condition of the corresponding dimension to judge the dimension processing result to obtain a judgment result, and issues an alarm based on the judgment result.

[0055] In step S101, the collection module collects the log data generated in nginx based on the collector, wherein the collector is a relatively common device of nginx, and its specific model is not limited. The collector and nginx have a one-to-one relationship, and the collection module can collect log data generated by multiple nginx at the same time. After the log data is collected, the log data is pushed to the message queue kafka according to a predefined log format. The message queue kafka plays a buffering role to avoid too much log data collected at the same time, causing the cleaner to be stuck. At this time, the cleaning module cleans the log data in the message queue kafka according to a preset cleaning method to obtain a cleaning result. The preset cleaning method is specifically set based on the actual situation and is not limited here. The cleaning module is implemented based on the exporter program, and the preset cleaning method of the present application extracts multiple preset types of fields from the log data. The cleaning result is a set of multiple preset types of fields, and the remaining log data is processed according to a preset processing method, which can be deleted, discarded or re-stored, etc. The preset type is specifically set based on the actual situation and is not limited here.

[0056] In step S102, the cleaning module and the monitoring module have a communication function, and the field extraction result can be sent to the monitoring module. The communication function ensures that the monitoring module can obtain the cleaning result corresponding to the latest log data in real time, so as to perform effective monitoring and management. The monitoring module can understand the effect of data cleaning and field extraction in real time, so as to ensure the accuracy and efficiency of the entire processing flow. The monitoring module is implemented based on promethues, that is, the field extraction result is written into the monitoring module, and the monitoring module stores the field extraction result according to the type set by the monitoring module for the field extraction result, and processes the field extraction result after the storage, such as performing field extraction, to obtain a field extraction result, and the field extraction result includes multiple extracted fields.

[0057] In step S103, the monitoring module applies different processing methods according to different dimensions. These processing methods may include extraction, calculation, combination, etc., aiming to extract information useful for monitoring and analysis from the field extraction results. After applying different processing methods, the monitoring module will generate dimension processing results. These results represent a specific view or summary of the data on each dimension, providing a basis for subsequent alarm judgment. Based on the dimension processing results, the monitoring module will determine the corresponding alarm judgment conditions on each dimension. These conditions define when an alarm should be triggered, for example, when the data value of a certain dimension exceeds or falls below a certain threshold. Once the alarm judgment conditions are determined, the monitoring module can monitor the data on each dimension according to these conditions. If the data meets the alarm conditions of a certain dimension, the monitoring module will trigger the corresponding alarm. The whole process is automated and usually real-time. The monitoring module can continuously receive the field extraction results, process them, and update the alarm judgment conditions based on the latest data.

[0058] In the specific implementation process of step S103, there is an embodiment in which the monitoring module performs different processing on the field extraction result based on the processing methods corresponding to different dimensions to obtain the corresponding dimension processing results, including:

[0059] S1031, the monitoring module confirms the dimension of the field extraction result processing based on the log data;

[0060] S1032. The monitoring module processes the field extraction result based on the processing method corresponding to the dimension to obtain a dimension processing result.

[0061] In steps S1031-S1032, the monitoring module confirms the dimensions of the field extraction result processing based on the log data, that is, in order to ensure the stability of the source of the log data, the source of the log data includes various software or web pages running on computers, computers, servers and other smart devices, so it is necessary to perform multi-faceted and multi-dimensional processing on the field extraction results corresponding to the log data. Therefore, the dimensions that need to be processed are confirmed from the perspective of affecting the stability of the source of the log data. In the present application, the dimensions to be processed include TPS calculation, time distribution calculation and traffic calculation, respectively. Different dimensions correspond to different processing methods, that is, the field extraction results are processed in the corresponding dimensional processing method to achieve different dimensional processing, thereby obtaining the corresponding dimensional processing results.

[0062] In the specific implementation process of step S1032, there is an embodiment in which: the dimension includes TPS calculation;

[0063] The monitoring module processes the field extraction result based on the processing method corresponding to the dimension to obtain a dimension processing result, including:

[0064] S103211, the monitoring module matches the multiple fields in the field extraction result with the preset field combination to obtain the TPS indicator;

[0065] S103212. The monitoring module calculates the target derived data of the TPS indicator to obtain a dimensional processing result based on the target derived data.

[0066] In steps S103211-S103212, the TPS calculation is an error calculation. To calculate the tps of a domain name or all domain names, that is, error statistics, separate fields are required, such as the domain name and status code to form a sample data. Since the pull mode is adopted, the monitoring module will store the TPS indicator in the memory, so all data cannot be written into prometheus, otherwise a huge Cartesian set will be formed. Therefore, the defined TPS indicator can indicate what happened to a certain domain name in a certain computer room at a certain point in time. To calculate the TPS indicator, we can use the counter type to record transactions, and then calculate the average rate. The TPS indicator composition can be composed of at least the following tags: domain name + computer room + status code + other components, among which the timestamp does not need to be added, and the monitoring module will generate the time itself according to the write time.

[0067] Usually in the http protocol, if the status code is 200, it means normal, the status code 499 means timeout, and 5xx and above means that the interface returns an error. The error status codes are mainly 499, 502, 504, etc., among which 502 means web page exception and 504 means another timeout. Therefore, TPS calculation is performed based on 499, 502, 504, etc. When processing the TPS calculation dimension, the monitoring module calls the preset field combination during the TPS calculation to obtain the TPS indicator based on the field combination. In the TPS indicator, the status code is recorded in kv in ng_status. The field The combination is obtained by arranging and combining multiple fields, and the number is multiple. The TPS indicator includes domain name + computer room + status code + others. After the cleaning module cleans the log data each time, the TPS indicator will be updated. The updating method is that after each log data cleaning, if multiple fields in the field extraction result match the preset field combination, the TPS indicator + 1, otherwise, a new field combination is obtained, and the new field combination is stored, thereby completing the update task of the TPS indicator, and finally forming a TPS indicator named nginx_http_request counter type, the field combination is host+status+IDC+tag+env+request_method, the corresponding fields are extracted from the field extraction result, the fields are arranged and combined or the TPS indicator is captured to prometheus for storage, and the monitoring module further calculates the target derivative data of the TPS indicator. The TPS indicator has multiple derivative data, and the value corresponding to the average growth rate is the target derivative data. The target derivative data calculated at this time, that is, the average growth rate, is the dimension processing result.

[0068] In the specific implementation process of step S103212, there is an embodiment in which the monitoring module calculates the target derived data of the TPS indicator to obtain a dimension processing result based on the target derived data, including:

[0069] S1032121. The monitoring module generates a function call instruction based on the TPS indicator to call a calculation function based on the function call instruction;

[0070] S1032122. The calculation function calculates the TPS indicator based on the function call instruction to obtain target derivative data.

[0071] In steps S1032121-S1032122, the monitoring module generates a function call instruction based on the TPS indicator according to a preset rule or algorithm. The function call instruction contains the parameters and call information required for the calculation function, so that the subsequent calculation process can be correctly executed to call the calculation function based on the function call instruction. The calculation function is an average growth rate calculation function. The calculation function is commonly used in the prior art and is not supplemented here. The calculation function calculates the TPS indicator according to a specific algorithm or logic based on the function call instruction. The calculation process includes data conversion, statistical analysis, trend prediction and other steps to finally generate target derivative data, such as obtaining the average growth rate of tps of a certain type of ng_tag, or the average growth rate of tps of all ng_tags. It can also be filtered according to a single domain name. Just enter the ng_host field to obtain the corresponding average growth rate of the TPS indicator. The average growth rate obtained can also be summed using sum, such as Figure 2 Shown are display diagrams of 499, 502, and 504, which can be specifically set according to actual conditions.

[0072] In the specific implementation process of step S1032, there is an embodiment in which: the dimension includes time consumption distribution calculation;

[0073] The monitoring module processes the field extraction result based on the processing method corresponding to the dimension to obtain a dimension processing result, including:

[0074] S103221. The monitoring module extracts the target field in the log data based on the processing method of the corresponding dimension, and determines the time consumption index of the corresponding dimension;

[0075] S103222. The monitoring module counts the frequency of the target field corresponding to the time-consuming indicator to obtain the dimension processing result based on the frequency.

[0076] In steps S103221-S103222, the time consumption is specifically the request time consumption. The time consumption is a very important indicator, which represents the quality of user access. It is necessary to calculate the average time consumption and the quantile distribution of each time consumption so that developers and operation and maintenance personnel can better understand the performance bottleneck of the system and optimize the factors that cause increased delays. When the computing dimension is the time consumption distribution calculation, the monitoring module extracts the target field in the log data, the target field is request_time, and writes it into the value in the histogram type. The time consumption indicator is composed of domain name + environment + computer room, where the frequency is the number of times the target field corresponding to the time consumption indicator appears. The time consumption indicator is calculated using TP50, TP90, and TP99 as examples. This solution uses the histogram type for calculation, which uses a fixed number of buckets, each of which has an upper limit value. When an observation is recorded, it is assigned to the first bucket that is greater than or equal to the observation. We define the maximum request time of nginx as 10 seconds, which is the upper limit of the bucket. Generally, the time consumption of less than 0.1 seconds is the lower limit of the bucket. The rest of the buckets can be customized. For example, the bucket range is defined as: 0.1, 0.3, 0.5, 1, 3, 5, 7, 10. This can better cover the request time range from 0.1 seconds to 10 seconds. For example, the cumulative counts when calculating tp90 are:

[0077] [0,0.1): 100; [0.1,0.3): 150; [0.3,0.5): 180; [0.5,1): 200

[0078] [1,3): 210; [3,5): 215; [5,7): 217; [7,10): 219; [10,+∞): 220;

[0079] Target position: The frequency of the total number of requests is 220, and the target position of tp90 is 220*0.90=198. The target position falls in the [0.5,1) bucket, so the value of tp90 is approximately between 0.5 and 1. The specific value can be calculated by linear interpolation, that is, the dimension processing result at this time is the value of tp90.

[0080] In the specific implementation process of step S1032, there is an embodiment in which: the dimension includes flow calculation;

[0081] The monitoring module processes the field extraction result based on the processing method corresponding to the dimension to obtain a dimension processing result, including:

[0082] S103231, the monitoring module extracts the corresponding number of bytes based on the multiple fields in the field extraction result and determines the corresponding traffic index;

[0083] S103232. The monitoring module calculates the total number of bytes corresponding to the traffic indicator to obtain the corresponding dimensional processing result based on the total number of bytes.

[0084] In steps S103231-S103232, network traffic is often an important indicator, so traffic calculation is required. The nginx log records the sent and received bytes of each request, so the key fields are extracted to form indicators to calculate the second-level traffic. When a problem occurs, it is also possible to promptly discover which domain name occupies a large amount of traffic. In the log data, upstream_bytes_sent represents the sent traffic, and upstream_bytes_received represents the received traffic. Therefore, the traffic index is based on the sent traffic upstream_bytes_sent and the received traffic upstream_bytes_received. The traffic index is used to calculate the inbound and outbound traffic, and the monitoring module uses the counter type to calculate the traffic index. For each log data of the cleaning module, the number of bytes corresponding to the traffic index is taken out, and the value is accumulated to obtain the total number of bytes of the traffic index. The total number of bytes is the corresponding dimension processing result, where the kv composition of the traffic index can be composed of computer room + environment + domain name, and the traffic display is specifically realized through sql, such as Figure 3 shown.

[0085] In step S104, the monitoring module implements alarms through the open source component altermanag, sets different alarm judgment conditions for different dimensions, and uses the alarm judgment conditions of the corresponding dimensions to judge the dimension processing results of the corresponding dimensions to obtain judgment results. In the present application, the dimensions include TPS calculation, time distribution calculation and traffic calculation, and the corresponding dimension processing results also include three dimensions of TPS calculation, time distribution calculation and traffic calculation. The corresponding judgment results also include three dimensions of TPS calculation, time distribution calculation and traffic calculation. The alarm judgment condition for the TPS calculation dimension is that the dimension processing result exceeds the preset error threshold or the error of the status code exceeds the preset number threshold. The alarm judgment condition for the traffic calculation dimension is that the dimension processing result exceeds the preset traffic threshold. The alarm judgment condition for the time-consuming calculation dimension is that the dimension processing result exceeds the preset time threshold, that is, if the status code is successful but the time is long, an alarm will also be issued, wherein the preset error threshold, preset number threshold, preset traffic threshold, and preset time threshold are specifically set according to the actual situation and are not limited here. The judgment result includes two types: satisfied or not satisfied, and an alarm is issued based on the judgment result.

[0086] In the specific implementation process of step S104, there is an embodiment in which the monitoring module uses the alarm judgment condition of the corresponding dimension to judge the dimension processing result to obtain a judgment result, and issues an alarm based on the judgment result, including:

[0087] S1041, the monitoring module determines whether the dimension processing result meets the alarm judgment condition of the corresponding dimension;

[0088] S1042: If it is in compliance, the monitoring module performs fusion based on the judgment result to obtain a fusion result, and issues an alarm based on the fusion result.

[0089] In steps S1041-S1042, the monitoring module determines whether the dimension processing result meets the alarm judgment condition of the corresponding dimension. If not, the judgment result is that the alarm judgment condition is not met. If it is met, the judgment result is that the alarm judgment condition is met. There are various ways to fuse the judgment results. In the present application, the fusion method adopted is that if the dimension processing results corresponding to the three dimensions have one dimension that meets the dimension processing, that is, the judgment result meets the alarm judgment condition, then the fusion result is that an alarm needs to be issued, that is, at least one judgment result meets the alarm judgment condition, then the fusion result is to issue an alarm. The specific alarm method can be a table or a graphical representation such as a line chart to issue an alarm. When the monitoring module issues an alarm, it also displays the specific dimensions of the alarm to facilitate the operation and maintenance personnel to handle it. If there are two or more dimensions that meet the alarm condition at the same time, the monitoring module is based on the priority of the dimension, the severity of the alarm, or merges multiple related alarms into a more comprehensive alarm. Automatic corresponding measures can also be set. When the monitoring module issues an alarm based on the fusion result, the corresponding measures are automatically triggered for processing to handle the alarm as soon as possible.

[0090] Example 2

[0091] The present application also provides a log data processing device, such as Figure 4 The block diagram of a log data processing device is shown. The functions implemented by the log data processing device correspond to the steps of executing a log data processing method on a terminal device. The device can be understood as a component of a server including a processor, which is applied to a processing system. The processing system includes a collection module, a cleaning module, and a monitoring module. The device includes:

[0092] The collection module 401 is used for collecting the log data generated by nginx and pushing the log data to the message queue, so that the cleaning module cleans the log data in the message queue to obtain a cleaning result;

[0093] A sending module 402, configured for the cleaning module to send the cleaning result to the monitoring module, so that the monitoring module performs field extraction based on the cleaning result to obtain a field extraction result;

[0094] Processing module 403, used for the monitoring module to perform different processing on the field extraction result based on the processing methods corresponding to different dimensions to obtain corresponding dimension processing results, so as to determine the alarm judgment condition of the corresponding dimension based on the dimension corresponding to the dimension processing result;

[0095] The alarm module 404 is used for the monitoring module to use the alarm judgment condition of the corresponding dimension to judge the dimension processing result to obtain a judgment result, so as to issue an alarm based on the judgment result.

[0096] In a feasible implementation manner, the processing module includes:

[0097] A confirmation module, used for the monitoring module to confirm the dimension of the field extraction result processing based on the log data;

[0098] The first processing module is used for the monitoring module to process the field extraction result based on the processing method corresponding to the dimension to obtain a dimension processing result.

[0099] In a feasible implementation manner, the processing module further includes:

[0100] A matching module, used for the monitoring module to match multiple fields in the field extraction result with a preset field combination to obtain a TPS indicator;

[0101] The first calculation module is used for the monitoring module to calculate the target derived data of the TPS indicator to obtain a dimension processing result based on the target derived data.

[0102] In a feasible implementation manner, the processing module also includes:

[0103] A first extraction module, used for the monitoring module to extract the target field in the log data based on the processing method of the corresponding dimension, and determine the time consumption index of the corresponding dimension;

[0104] The statistical module is used by the monitoring module to count the frequency of the target field corresponding to the time-consuming indicator, so as to obtain the dimension processing result based on the frequency.

[0105] In a feasible implementation manner, the processing module further includes:

[0106] A second extraction module, used for the monitoring module to extract the corresponding number of bytes based on the multiple fields in the field extraction result and determine the corresponding traffic index;

[0107] The second calculation module is used by the monitoring module to calculate the total amount of bytes corresponding to the traffic indicator, so as to obtain the corresponding dimensional processing result based on the total amount of bytes.

[0108] In a feasible implementation manner, the processing module further includes:

[0109] A generating module, used for the monitoring module to generate a function calling instruction based on the TPS indicator, so as to call a computing function based on the function calling instruction;

[0110] The third calculation module is used for the calculation function to calculate the TPS indicator based on the function call instruction to obtain target derivative data.

[0111] In a feasible implementation manner, the alarm module further includes:

[0112] A judgment module, used by the monitoring module to judge whether the dimension processing result meets the alarm judgment condition of the corresponding dimension;

[0113] The fusion module is used to, if it meets the requirements, enable the monitoring module to fuse the judgment results to obtain a fusion result, so as to issue an alarm based on the fusion result.

[0114] Example 3

[0115] The present application also provides an electronic device, such as Figure 5 As shown, it includes: a processor 501, a memory 502 and a bus 503, the memory 502 stores machine-readable instructions executable by the processor 501, when the electronic device is running, the processor 501 and the memory 502 communicate through the bus 503, and when the machine-readable instructions are executed by the processor 501, the steps of any one of the log data processing methods are performed.

[0116] Example 4

[0117] The present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of any one of the log data processing methods are executed.

[0118] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the method embodiment, and will not be repeated in this application. In the several embodiments provided in this application, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0119] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0120] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0121] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a platform server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks, or optical disks.

[0122] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for processing log data, characterized in that: Applied to a processing system, the processing system includes a collection module, a cleaning module, and a monitoring module, and the method includes: The collection module collects the log data generated by nginx and pushes the log data to the message queue, so that the cleaning module cleans the log data in the message queue to obtain a cleaning result; The cleaning module sends the cleaning result to the monitoring module, so that the monitoring module performs field extraction based on the cleaning result to obtain a field extraction result; The monitoring module performs different processing on the field extraction results based on processing methods corresponding to different dimensions to obtain corresponding dimension processing results, so as to determine the alarm judgment condition of the corresponding dimension based on the dimension corresponding to the dimension processing result; The monitoring module uses the alarm judgment condition of the corresponding dimension to judge the dimension processing result to obtain a judgment result, so as to issue an alarm based on the judgment result.

2. The method according to claim 1, characterized in that The monitoring module performs different processing on the field extraction results based on the processing methods corresponding to different dimensions to obtain corresponding dimension processing results, including: The monitoring module confirms the dimension of the field extraction result processing based on the log data; The monitoring module processes the field extraction result based on the processing method corresponding to the dimension to obtain a dimension processing result.

3. The method according to claim 2, characterized in that The dimensions include TPS calculation; The monitoring module processes the field extraction result based on the processing method corresponding to the dimension to obtain a dimension processing result, including: The monitoring module obtains a TPS indicator by matching multiple fields in the field extraction result with a preset field combination; The monitoring module calculates target derived data of the TPS indicator to obtain a dimensional processing result based on the target derived data.

4. The method according to claim 2, characterized in that: The dimensions include time-consuming distribution calculations; The monitoring module processes the field extraction result based on the processing method corresponding to the dimension to obtain a dimension processing result, including: The monitoring module extracts the target field in the log data based on the processing method of the corresponding dimension, and determines the time consumption index of the corresponding dimension; The monitoring module counts the frequency of the target field corresponding to the time-consuming indicator to obtain the dimension processing result based on the frequency.

5. The method according to claim 2, characterized in that: Said dimensions include flow calculation; The monitoring module processes the field extraction result based on the processing method corresponding to the dimension to obtain a dimension processing result, including: The monitoring module extracts corresponding byte quantities based on multiple fields in the field extraction result and determines corresponding traffic indicators; The monitoring module calculates the total number of bytes corresponding to the traffic indicator to obtain a corresponding dimensional processing result based on the total number of bytes.

6. The method according to claim 3, characterized in that: The monitoring module calculates the target derived data of the TPS indicator to obtain a dimension processing result based on the target derived data, including: The monitoring module generates a function call instruction based on the TPS indicator to call a calculation function based on the function call instruction; The calculation function calculates the TPS indicator based on the function call instruction to obtain target derived data.

7. The method according to claim 1, characterized in that The monitoring module uses the alarm judgment condition of the corresponding dimension to judge the dimension processing result to obtain a judgment result, and issues an alarm based on the judgment result, including: The monitoring module determines whether the dimension processing result meets the alarm judgment condition of the corresponding dimension; If it is consistent, the monitoring module fuses the judgment results to obtain a fusion result, and issues an alarm based on the fusion result.

8. A log data processing device, characterized in that: Applied to a processing system, the processing system includes a collection module, a cleaning module, and a monitoring module, and the device includes: A collection module is used for collecting log data generated by nginx and pushing the log data to a message queue, so that the cleaning module cleans the log data in the message queue to obtain a cleaning result; A sending module, used for the cleaning module to send the cleaning result to the monitoring module, so that the monitoring module performs field extraction based on the cleaning result to obtain a field extraction result; A processing module, used for the monitoring module to perform different processing on the field extraction results based on processing methods corresponding to different dimensions to obtain corresponding dimension processing results, so as to determine the alarm judgment condition of the corresponding dimension based on the dimension corresponding to the dimension processing result; The alarm module is used for the monitoring module to use the alarm judgment condition of the corresponding dimension to judge the dimension processing result to obtain a judgment result, so as to issue an alarm based on the judgment result.

9. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of a log data processing method as described in any one of claims 1 to 7 are performed.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the log data processing method according to any one of claims 1 to 7 are executed.

Images