Business log identification and evaluation method and system based on big data and medium

Abstract

The embodiment of the invention provides a business log identification and evaluation method and system based on big data and a medium. The method comprises the steps of monitoring a service system in real time, collecting an abnormal log data set in a preset time period, performing data preprocessing on the abnormal log data set, performing analysis processing through a preset language behavior processing model according to the processed abnormal log data set, obtaining abnormal log behavior information, and sending the abnormal log behavior information to a server; extracting feature data according to the abnormal log behavior information, inputting the feature data into a preset business abnormal log recognition model, processing to obtain a log abnormal recognition index, and performing threshold comparison according to the log abnormal recognition index and a preset log abnormal threshold. Obtaining a corresponding preset adjustment scheme according to a threshold comparison result to carry out data correction, information adjustment or monitoring alarm on the service log; therefore, business log identification and evaluation are carried out through big data, and an adjustment scheme and a monitoring alarm are obtained.

Description

Technical Field

[0001] The present application relates to the technical field of big data and business log monitoring, and in particular to a business log identification and evaluation method, system and medium based on big data. Background Art

[0002] With the rapid development of information technology and the in-depth advancement of enterprise digital transformation, business systems have become an indispensable part of enterprise management and operations. Exception logs are an important carrier for recording abnormal events and error information in business systems, and contain key information and status changes during system operation. When an abnormality occurs in the system, identifying and evaluating the exception logs can help operation and maintenance personnel quickly identify the root cause of the problem, locate the cause of the fault, and take timely measures to repair it. However, since the text content of the exception log is usually unstructured and diverse, traditional manual processing methods can no longer meet the needs of efficient processing of exception logs. Therefore, how to design a set of technologies that can improve the efficiency of exception log identification and processing and the accuracy of evaluation, and realize the evaluation and presentation of the operation status of power grid business, has become a current vacant technology.

[0003] In view of the above problems, effective technical solutions are urgently needed. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a business log identification and evaluation method, system and medium based on big data, which can use big data to identify, evaluate and monitor and alarm business log anomalies within a preset time period, realize business log identification and evaluation through big data and obtain adjustment plans and monitoring alarms, as well as graphically present the evaluation results of business operation status.

[0005] The present application also provides a method for identifying and evaluating business logs based on big data, including the following steps:

[0006] Monitor the business system in real time and collect abnormal log data sets within a preset time period, and pre-process the abnormal log data sets to obtain processed abnormal log data sets;

[0007] The abnormal log data set is parsed and processed by a preset language behavior processing model to obtain abnormal log behavior information;

[0008] Extracting feature data according to the abnormal log behavior information, and inputting the feature data into a preset business abnormal log recognition model to obtain a log abnormality recognition index;

[0009] A threshold comparison is performed based on the log anomaly identification index and a preset log anomaly threshold, and a corresponding preset adjustment plan is obtained based on the threshold comparison result to perform data correction, information adjustment or monitoring alarm on the business log.

[0010] Optionally, in the big data-based business log identification and evaluation method described in the embodiment of the present application, the real-time monitoring of the business system and the collection of abnormal log data sets within a preset time period, and the data preprocessing of the abnormal log data sets to obtain the processed abnormal log data sets include:

[0011] Monitor business systems in real time and collect abnormal log data sets within a preset time period, including abnormal log timestamp data, log abnormality level data, abnormality type data, stack trace data, business abnormality identification data, and error handling data;

[0012] The abnormal log data set is preprocessed to obtain a cleaned and optimized abnormal log data set.

[0013] Optionally, in the big data-based business log identification and evaluation method described in the embodiment of the present application, the abnormal log data set is parsed and processed by a preset language behavior processing model to obtain abnormal log behavior information, including:

[0014] Obtain a trained preset language behavior processing model;

[0015] The abnormal log data set is parsed and processed by the preset language behavior processing model to obtain abnormal log behavior information;

[0016] The abnormal log behavior information includes: business process interruption information, business data abnormal change information, business rule violation information, transaction abnormality information, order status abnormality information, customer behavior abnormality information, business indicator abnormal fluctuation information and business response abnormality information.

[0017] Optionally, in the big data-based business log identification and evaluation method described in the embodiment of the present application, extracting feature data according to the abnormal log behavior information, and inputting the feature data into a preset business abnormality log identification model to obtain a log abnormality identification index, including:

[0018] Extract feature data based on the abnormal log behavior information, the feature data including abnormal behavior discovery frequency data, abnormal duration data, abnormal behavior degree classification data, abnormal behavior time distribution feature data, behavior deviation data, and key indicator impact value data corresponding to the log behavior information;

[0019] Input the abnormal behavior discovery frequency data, abnormal duration data, abnormal behavior degree classification data, abnormal behavior time distribution characteristic data, behavior deviation data and key indicator impact value data into a preset business abnormality log recognition model, and process to obtain a log abnormality recognition index;

[0020] The calculation formula of the log anomaly identification index is:

[0021]

[0022] Among them, A C is the log anomaly identification index, q w 、e r ,t y 、u i , o p 、s d The abnormal behavior discovery frequency data, abnormal duration data, abnormal behavior degree classification data, abnormal behavior time distribution characteristic data, behavior deviation data and key indicator impact value data are respectively p is the business system operation compensation factor, ε, η and λ are preset characteristic coefficients.

[0023] Optionally, in the big data-based business log identification and evaluation method described in the embodiment of the present application, the threshold comparison is performed according to the log anomaly identification index and the preset log anomaly threshold, and the corresponding preset adjustment plan is obtained according to the threshold comparison result to perform data correction, information adjustment or monitoring alarm on the business log, including:

[0024] Performing a threshold comparison between the log anomaly identification index and a preset log anomaly threshold level to obtain a corresponding threshold comparison result threshold level;

[0025] Obtaining the corresponding log abnormality level and business log operation adjustment plan according to the threshold comparison result threshold level;

[0026] According to the business log operation adjustment plan, the business log is corrected, the information is adjusted or monitored and alarmed to achieve the identification and evaluation of the log data of the business system

[0027] Optionally, the big data-based business log identification and evaluation method described in the embodiment of the present application further includes:

[0028] The log abnormality level and the business log operation adjustment plan are recorded and displayed through a preset business log monitoring table, and different preset font colors are used to display the corresponding log abnormality level and the business log operation adjustment plan;

[0029] Collect a set of log anomaly levels within a preset time period, including log anomaly levels corresponding to multiple preset time periods;

[0030] Processing is performed according to the log abnormality level set to obtain the business log abnormality fluctuation index within the preset time period;

[0031] Input the log abnormality level set in combination with the business log abnormal fluctuation index into the preset business log monitoring table for recording, and obtain a business log operation status monitoring chart, thereby obtaining a chart presentation and display identification of the business operation status;

[0032] The calculation formula of the business log abnormal fluctuation index is:

[0033]

[0034] Among them, K x is the abnormal fluctuation index of business log, A ci is the log anomaly identification index of the i-th sampling period node in the log anomaly level set, n is the number of log anomaly identification indexes in the log anomaly level set, ψ i is the preset characteristic coefficient of the log anomaly level of the node in the i-th sampling period.

[0035] In a second aspect, an embodiment of the present application provides a business log identification and evaluation system based on big data, the system comprising: a memory and a processor, the memory comprising a program of a business log identification and evaluation method based on big data, and the program of the business log identification and evaluation method based on big data is executed by the processor to implement the following steps:

[0036] Monitor the business system in real time and collect abnormal log data sets within a preset time period, and pre-process the abnormal log data sets to obtain processed abnormal log data sets;

[0037] The abnormal log data set is parsed and processed by a preset language behavior processing model to obtain abnormal log behavior information;

[0038] Extracting feature data according to the abnormal log behavior information, and inputting the feature data into a preset business abnormal log recognition model to obtain a log abnormality recognition index;

[0039] A threshold comparison is performed based on the log anomaly identification index and a preset log anomaly threshold, and a corresponding preset adjustment plan is obtained based on the threshold comparison result to perform data correction, information adjustment or monitoring alarm on the business log.

[0040] Optionally, in the big data-based business log identification and evaluation system described in the embodiment of the present application, the real-time monitoring business system and collecting abnormal log data sets within a preset time period, and performing data preprocessing on the abnormal log data sets to obtain the processed abnormal log data sets include:

[0041] Monitor business systems in real time and collect abnormal log data sets within a preset time period, including abnormal log timestamp data, log abnormality level data, abnormality type data, stack trace data, business abnormality identification data, and error handling data;

[0042] The abnormal log data set is preprocessed to obtain a cleaned and optimized abnormal log data set.

[0043] Optionally, in the big data-based business log identification and evaluation system described in the embodiment of the present application, the abnormal log data set after processing is parsed by a preset language behavior processing model to obtain abnormal log behavior information, including:

[0044] Obtain a trained preset language behavior processing model;

[0045] The abnormal log data set is parsed and processed by the preset language behavior processing model to obtain abnormal log behavior information;

[0046] The abnormal log behavior information includes: business process interruption information, business data abnormal change information, business rule violation information, transaction abnormality information, order status abnormality information, customer behavior abnormality information, business indicator abnormal fluctuation information and business response abnormality information.

[0047] In a third aspect, an embodiment of the present application further provides a computer-readable storage medium, which includes a business log identification and evaluation method program based on big data. When the business log identification and evaluation method program based on big data is executed by a processor, the steps of the business log identification and evaluation method based on big data as described in any one of the above items are implemented.

[0048] As can be seen from the above, the business log identification and evaluation method, system and medium based on big data provided by the embodiments of the present application monitor the business system in real time and collect abnormal log data sets within a preset time period, and pre-process the abnormal log data sets, parse and process the processed abnormal log data sets through a preset language behavior processing model to obtain abnormal log behavior information, extract feature data based on the abnormal log behavior information, and input the feature data into a preset business abnormal log identification model to obtain a log abnormality identification index, perform a threshold comparison between the log abnormality identification index and the preset log abnormality threshold, and obtain the corresponding preset adjustment plan based on the threshold comparison result to perform data correction, information adjustment or monitoring alarm on the business log; thereby realizing business log identification and evaluation through big data and obtaining adjustment plans and monitoring alarms, as well as graphical presentation of the evaluation results of the business operation status.

[0049] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by practicing the embodiments of the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0051] Figure 1 A flowchart of a business log identification and evaluation method based on big data provided in an embodiment of the present application;

[0052] Figure 2 A flowchart of obtaining a processed abnormal log data set for a business log identification and evaluation method based on big data provided in an embodiment of the present application;

[0053] Figure 3 A flowchart of obtaining abnormal log behavior information of a business log identification and evaluation method based on big data provided in an embodiment of the present application;

[0054] Figure 4 A flowchart of obtaining the real-time operation status of a business log and the corresponding log anomaly identification index of a business log identification and evaluation method based on big data provided in an embodiment of the present application. DETAILED DESCRIPTION

[0055] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0056] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0057] Please refer to Figure 1 , Figure 1 1 is a flowchart of a business log identification and evaluation method based on big data in some embodiments of the present application. The business log identification and evaluation method based on big data is used in terminal devices, such as computers, mobile terminals, etc. The business log identification and evaluation method based on big data includes the following steps:

[0058] S11, real-time monitoring of the business system and collecting abnormal log data sets within a preset time period, and performing data preprocessing on the abnormal log data sets to obtain a processed abnormal log data set;

[0059] S12, parsing the processed abnormal log data set through a preset language behavior processing model to obtain abnormal log behavior information;

[0060] S13, extracting feature data according to the abnormal log behavior information, and inputting the feature data into a preset business abnormal log recognition model to obtain a log abnormality recognition index;

[0061] S14: performing a threshold comparison between the log anomaly identification index and a preset log anomaly threshold, and obtaining a corresponding preset adjustment plan according to the threshold comparison result to perform data correction, information adjustment or monitoring alarm on the business log.

[0062] It should be noted that, since the text content of the exception log is usually unstructured and diverse, the traditional manual processing method can no longer meet the demand for efficient processing of the exception log, and the exception log is an important carrier for recording abnormal events and error information in the business system. Therefore, the real-time monitoring of the business system within a preset time period and the collection of the exception log data set data collection, processing and evaluation are the main design layout of the technology for implementing the business log evaluation and operation. By preprocessing, parsing and extracting information from the real-time monitored and collected exception log data set, multiple feature data that can reflect the abnormal log behavior can be obtained. The feature data is input into the preset business exception log recognition model to obtain the log anomaly recognition index that can feedback the real-time operation status of the business log. Then, by comparing the threshold with the preset log anomaly threshold, the corresponding preset adjustment plan is obtained according to the threshold level where the threshold comparison result falls, and the business log of the business system is corrected according to the specific adjustment plan. Information adjustment or monitoring alarm is performed to ensure that the abnormal situation of the business log is discovered and responded to in the first time.

[0063] Please refer to Figure 2 , Figure 2 The flowchart of the business log identification and evaluation method based on big data in some embodiments of the present application is to obtain the processed abnormal log data set. According to the embodiment of the present invention, the real-time monitoring business system and collecting the abnormal log data set within a preset time period, and performing data preprocessing on the abnormal log data set to obtain the processed abnormal log data set, specifically:

[0064] S21, real-time monitoring of the business system and collection of abnormal log data sets within a preset time period, including abnormal log timestamp data, log abnormality level data, abnormality type data, stack trace data, business abnormality identification data, and error handling data;

[0065] S22: performing data preprocessing on the abnormal log data set to obtain a cleaned and optimized abnormal log data set.

[0066] It should be noted that first, it is necessary to collect data from the real-time monitoring database to obtain abnormal log data sets. These abnormal log data sets include a variety of abnormal data, which are crucial for subsequent evaluation and analysis, and preprocess these abnormal log data sets, including data cleaning, eliminating duplicate data, missing data and irrelevant data, retaining key abnormal data, and annotating the cleaned abnormal logs to obtain accurate and complete abnormal log data sets after processing.

[0067] Please refer to Figure 3 , Figure 3The flowchart of the business log identification and evaluation method based on big data in some embodiments of the present application is to obtain abnormal log behavior information. According to the embodiment of the present invention, the abnormal log data set after processing is parsed through a preset language behavior processing model to obtain abnormal log behavior information, specifically:

[0068] S31, obtaining a trained preset language behavior processing model;

[0069] S32, performing parsing processing on the processed abnormal log data set through the preset language behavior processing model to obtain abnormal log behavior information;

[0070] S33. The abnormal log behavior information includes: business process interruption information, business data abnormal change information, business rule violation information, transaction abnormality information, order status abnormality information, customer behavior abnormality information, business indicator abnormal fluctuation information and business response abnormality information.

[0071] It should be noted that the preprocessed exception log data set needs to be parsed and processed to identify the abnormal behaviors contained therein. Therefore, it is necessary to obtain the language behavior processing model that has been trained through historical log data sets and standard log data on the platform, and then input the preprocessed accurate and complete exception log data set into the language behavior processing model for parsing. After processing, different types of abnormal log behavior information are identified, including: business process interruption information, abnormal business data change information, business rule violation information, transaction abnormality information, order status abnormality information, customer behavior abnormality information, business indicator abnormal fluctuation information and business response abnormality information.

[0072] Please refer to Figure 4 , Figure 4 The flowchart of the business log recognition and evaluation method based on big data in some embodiments of the present application is to obtain the real-time operation status of the business log and the corresponding log anomaly recognition index. According to the embodiment of the present invention, the feature data is extracted according to the abnormal log behavior information, and the feature data is input into the preset business abnormal log recognition model to obtain the log anomaly recognition index, specifically:

[0073] S41, extracting feature data according to the abnormal log behavior information, the feature data including abnormal behavior discovery frequency data, abnormal duration data, abnormal behavior degree classification data, abnormal behavior time distribution feature data, behavior deviation data and key indicator impact value data corresponding to the log behavior information;

[0074] S42, inputting the abnormal behavior discovery frequency data, abnormal duration data, abnormal behavior degree classification data, abnormal behavior time distribution characteristic data, behavior deviation data and key indicator impact value data into a preset business abnormality log recognition model, and processing to obtain a log abnormality recognition index;

[0075] The calculation formula of the log anomaly identification index is:

[0076]

[0077] Among them, A C is the log anomaly identification index, q w 、e r ,t y 、u i , o p 、s d The abnormal behavior discovery frequency data, abnormal duration data, abnormal behavior degree classification data, abnormal behavior time distribution characteristic data, behavior deviation data and key indicator impact value data are respectively p is the business system operation compensation factor, ε, η, λ are preset characteristic coefficients. (The business system operation compensation factor and characteristic coefficient are obtained by querying the preset business log platform database).

[0078] It should be noted that in order to evaluate the abnormal behavior of business logs within a preset time period, it is necessary to extract feature data from the abnormal log behavior information, where the extracted feature data include the frequency of common behavior detection, abnormal duration, abnormal behavior degree classification, abnormal behavior time distribution characteristics, behavior deviation, and key indicator impact value data. The above data is input into the calculation formula of the business abnormal log recognition model for calculation and processing to obtain the real-time operation status of the business log and the corresponding log anomaly recognition index.

[0079] According to an embodiment of the present invention, the log anomaly identification index is compared with a preset log anomaly threshold, and a corresponding preset adjustment scheme is obtained according to the threshold comparison result to correct data, adjust information or monitor alarms for the business log, specifically:

[0080] Performing a threshold comparison between the log anomaly identification index and a preset log anomaly threshold level to obtain a corresponding threshold comparison result threshold level;

[0081] Obtaining the corresponding log abnormality level and business log operation adjustment plan according to the threshold comparison result threshold level;

[0082] According to the business log operation adjustment plan, the business log is corrected, the information is adjusted or monitored and alarmed to achieve the identification and evaluation of the log data of the business system

[0083] It should be noted that after completing the evaluation of the log anomaly of the business log system within the preset time period and obtaining the log anomaly identification index for identifying the real-time operation status of the business, a threshold comparison is performed based on the log anomaly identification index of the business log and the preset log anomaly threshold level, and the corresponding log anomaly level and the business log operation adjustment plan are obtained according to the threshold level range in which the threshold comparison result falls, that is, the operation abnormality state of the business log system within the time period is rated and the next adjustment plan corresponding to the rating result is obtained, and the business log is set and adjusted according to the plan to make the business log system adapt to the changing abnormal dynamic situation, and make real-time dynamic contingency adjustments to meet the normal operation requirements of the business log. In this embodiment, the preset threshold level is divided into one to three levels, and the corresponding threshold ranges correspond to levels I to III, respectively, and level I corresponds to (0.67, 1], level II corresponds to (0.33, 0.67], and level III corresponds to (0, 0.33], such as the log anomaly identification index of a business log A and the threshold comparison result of the corresponding log anomaly threshold level is 0.52, which falls into the level II range, then its corresponding threshold level is level II, and its log anomaly level is level II.

[0084] According to an embodiment of the present invention, it also includes:

[0085] The log abnormality level and the business log operation adjustment plan are recorded and displayed through a preset business log monitoring table, and different preset font colors are used to display the corresponding log abnormality level and the business log operation adjustment plan;

[0086] Collect a set of log anomaly levels within a preset time period, including log anomaly levels corresponding to multiple preset time periods;

[0087] Processing is performed according to the log abnormality level set to obtain the business log abnormality fluctuation index within the preset time period;

[0088] Input the log abnormality level set in combination with the business log abnormal fluctuation index into the preset business log monitoring table for recording, and obtain a business log operation status monitoring chart, thereby obtaining a chart presentation and display identification of the business operation status;

[0089] The calculation formula of the business log abnormal fluctuation index is:

[0090]

[0091] Among them, K x is the abnormal fluctuation index of business log, A ci is the log anomaly identification index of the i-th sampling period node in the log anomaly level set, n is the number of log anomaly identification indexes in the log anomaly level set, ψ i is the preset characteristic coefficient of the log anomaly level of the node in the i-th sampling period (the characteristic coefficient is obtained by querying the preset business log platform database).

[0092] It should be noted that after obtaining the log exception level and the business log operation adjustment plan, in order to more intuitively understand the business log operation status, a log monitoring table is used to record and display, and different preset font colors are used to display the corresponding log exception level and business log operation adjustment plan. In this embodiment of the scheme, the log exception and business log operation adjustment plan with a level of one are displayed in yellow font, the log exception and business log operation adjustment plan with a level of two are displayed in orange font, and the log exception and business log operation adjustment plan with a level of three are displayed in red font. In addition, in order to further monitor the operation status of the business log, a set of log exception levels for a longer time period is collected. The time period can be set to one month, three months or half a year, and the collected log exception level set is analyzed and processed to obtain the business log exception fluctuation index within the preset time period, and then the obtained business log exception fluctuation index is entered into the business log monitoring table. The monitoring table can display a graphical presentation of the evaluation of the business operation status within the preset time period.

[0093] The second aspect of the present invention further discloses a business log identification and evaluation system based on big data, comprising a memory and a processor, wherein the memory comprises a business log identification and evaluation method program based on big data, and when the business log identification and evaluation method program based on big data is executed by the processor, the following steps are implemented:

[0094] Monitor the business system in real time and collect abnormal log data sets within a preset time period, and pre-process the abnormal log data sets to obtain processed abnormal log data sets;

[0095] The abnormal log data set is parsed and processed by a preset language behavior processing model to obtain abnormal log behavior information;

[0096] Extracting feature data according to the abnormal log behavior information, and inputting the feature data into a preset business abnormal log recognition model to obtain a log abnormality recognition index;

[0097] A threshold comparison is performed based on the log anomaly identification index and a preset log anomaly threshold, and a corresponding preset adjustment plan is obtained based on the threshold comparison result to perform data correction, information adjustment or monitoring alarm on the business log.

[0098] It should be noted that, since the text content of the exception log is usually unstructured and diverse, the traditional manual processing method can no longer meet the demand for efficient processing of the exception log, and the exception log is an important carrier for recording abnormal events and error information in the business system. Therefore, the real-time monitoring of the business system within a preset time period and the collection of the exception log data set data collection, processing and evaluation are the main design layout of the technology for implementing the business log evaluation and operation. By preprocessing, parsing and extracting information from the real-time monitored and collected exception log data set, multiple feature data that can reflect the abnormal log behavior can be obtained. The feature data is input into the preset business exception log recognition model to obtain the log anomaly recognition index that can feedback the real-time operation status of the business log. Then, by comparing the threshold with the preset log anomaly threshold, the corresponding preset adjustment plan is obtained according to the threshold level where the threshold comparison result falls, and the business log of the business system is corrected according to the specific adjustment plan. Information adjustment or monitoring alarm is performed to ensure that the abnormal situation of the business log is discovered and responded to in the first time.

[0099] According to an embodiment of the present invention, the real-time monitoring service system collects an abnormal log data set within a preset time period, and performs data preprocessing on the abnormal log data set to obtain a processed abnormal log data set, specifically:

[0100] Monitor business systems in real time and collect abnormal log data sets within a preset time period, including abnormal log timestamp data, log abnormality level data, abnormality type data, stack trace data, business abnormality identification data, and error handling data;

[0101] The abnormal log data set is preprocessed to obtain a cleaned and optimized abnormal log data set.

[0102] It should be noted that first, it is necessary to collect data from the real-time monitoring database to obtain abnormal log data sets. These abnormal log data sets include a variety of abnormal data, which are crucial for subsequent evaluation and analysis, and preprocess these abnormal log data sets, including data cleaning, eliminating duplicate data, missing data and irrelevant data, retaining key abnormal data, and annotating the cleaned abnormal logs to obtain accurate and complete abnormal log data sets after processing.

[0103] According to an embodiment of the present invention, the abnormal log data set after processing is parsed through a preset language behavior processing model to obtain abnormal log behavior information, specifically:

[0104] Obtain a trained preset language behavior processing model;

[0105] The abnormal log data set is parsed and processed by the preset language behavior processing model to obtain abnormal log behavior information;

[0106] The abnormal log behavior information includes: business process interruption information, business data abnormal change information, business rule violation information, transaction abnormality information, order status abnormality information, customer behavior abnormality information, business indicator abnormal fluctuation information and business response abnormality information.

[0107] It should be noted that the preprocessed exception log data set needs to be parsed and processed to identify the abnormal behaviors contained therein. Therefore, it is necessary to obtain the language behavior processing model that has been trained through historical log data sets and standard log data on the platform, and then input the preprocessed accurate and complete exception log data set into the language behavior processing model for parsing. After processing, different types of abnormal log behavior information are identified, including: business process interruption information, abnormal business data change information, business rule violation information, transaction abnormality information, order status abnormality information, customer behavior abnormality information, business indicator abnormal fluctuation information and business response abnormality information.

[0108] According to an embodiment of the present invention, the feature data is extracted according to the abnormal log behavior information, and the feature data is input into a preset business abnormal log recognition model to obtain a log abnormality recognition index, specifically:

[0109] Extract feature data based on the abnormal log behavior information, the feature data including abnormal behavior discovery frequency data, abnormal duration data, abnormal behavior degree classification data, abnormal behavior time distribution feature data, behavior deviation data, and key indicator impact value data corresponding to the log behavior information;

[0110] Input the abnormal behavior discovery frequency data, abnormal duration data, abnormal behavior degree classification data, abnormal behavior time distribution characteristic data, behavior deviation data and key indicator impact value data into a preset business abnormality log recognition model, and process to obtain a log abnormality recognition index;

[0111] The calculation formula of the log anomaly identification index is:

[0112]

[0113] Among them, A C is the log anomaly identification index, q w 、e r ,t y 、u i , o p 、s dThe abnormal behavior discovery frequency data, abnormal duration data, abnormal behavior degree classification data, abnormal behavior time distribution characteristic data, behavior deviation data and key indicator impact value data are respectively p is the business system operation compensation factor, ε, η, λ are preset characteristic coefficients. (The business system operation compensation factor and characteristic coefficient are obtained by querying the preset business log platform database).

[0114] It should be noted that in order to evaluate the abnormal behavior of business logs within a preset time period, it is necessary to extract feature data from the abnormal log behavior information, where the extracted feature data include the frequency of common behavior detection, abnormal duration, abnormal behavior degree classification, abnormal behavior time distribution characteristics, behavior deviation, and key indicator impact value data. The above data are input into the trained business abnormality log recognition model obtained from the business platform for processing to obtain the real-time operation status of the business log and the corresponding log anomaly recognition index.

[0115] According to an embodiment of the present invention, the log anomaly identification index is compared with a preset log anomaly threshold, and a corresponding preset adjustment scheme is obtained according to the threshold comparison result to correct data, adjust information or monitor alarms for the business log, specifically:

[0116] Performing a threshold comparison between the log anomaly identification index and a preset log anomaly threshold level to obtain a corresponding threshold comparison result threshold level;

[0117] Obtaining the corresponding log abnormality level and business log operation adjustment plan according to the threshold comparison result threshold level;

[0118] According to the business log operation adjustment plan, the business log is corrected, the information is adjusted or monitored and alarmed to achieve the identification and evaluation of the log data of the business system

[0119] It should be noted that after completing the evaluation of the log anomaly of the business log system within the preset time period and obtaining the log anomaly identification index for identifying the real-time operation status of the business, a threshold comparison is performed based on the log anomaly identification index of the business log and the preset log anomaly threshold level, and the corresponding log anomaly level and the business log operation adjustment plan are obtained according to the threshold level range in which the threshold comparison result falls, that is, the operation abnormality state of the business log system within the time period is rated and the next adjustment plan corresponding to the rating result is obtained, and the business log is set and adjusted according to the plan to make the business log system adapt to the changing abnormal dynamic situation, and make real-time dynamic contingency adjustments to meet the normal operation requirements of the business log. In this embodiment, the preset threshold level is divided into one to three levels, and the corresponding threshold ranges correspond to levels I to III, respectively, and level I corresponds to (0.67, 1], level II corresponds to (0.33, 0.67], and level III corresponds to (0, 0.33], such as the log anomaly identification index of a business log A and the threshold comparison result of the corresponding log anomaly threshold level is 0.52, which falls into the level II range, then its corresponding threshold level is level II, and its log anomaly level is level II.

[0120] According to an embodiment of the present invention, it also includes:

[0121] The log abnormality level and the business log operation adjustment plan are recorded and displayed through a preset business log monitoring table, and different preset font colors are used to display the corresponding log abnormality level and the business log operation adjustment plan;

[0122] Collect a set of log anomaly levels within a preset time period, including log anomaly levels corresponding to multiple preset time periods;

[0123] Processing is performed according to the log abnormality level set to obtain the business log abnormality fluctuation index within the preset time period;

[0124] Input the log abnormality level set in combination with the business log abnormal fluctuation index into the preset business log monitoring table for recording, and obtain a business log operation status monitoring chart, thereby obtaining a chart presentation and display identification of the business operation status;

[0125] The calculation formula of the business log abnormal fluctuation index is:

[0126]

[0127] Among them, K x is the abnormal fluctuation index of business log, A ci is the log anomaly identification index of the i-th sampling period node in the log anomaly level set, n is the number of log anomaly identification indexes in the log anomaly level set, ψ iis the preset characteristic coefficient of the log anomaly level of the node in the i-th sampling period (the characteristic coefficient is obtained by querying the preset business log platform database).

[0128] It should be noted that after obtaining the log exception level and the business log operation adjustment plan, in order to more intuitively understand the business log operation status, a log monitoring table is used to record and display, and different preset font colors are used to display the corresponding log exception level and business log operation adjustment plan. In this embodiment of the scheme, the log exception and business log operation adjustment plan with a level of one are displayed in yellow font, the log exception and business log operation adjustment plan with a level of two are displayed in orange font, and the log exception and business log operation adjustment plan with a level of three are displayed in red font. In addition, in order to further monitor the operation status of the business log, a set of log exception levels for a longer time period is collected. The time period can be set to one month, three months or half a year, and the collected log exception level set is analyzed and processed to obtain the business log exception fluctuation index within the preset time period, and then the obtained business log exception fluctuation index is entered into the business log monitoring table. The monitoring table can display a graphical presentation of the evaluation of the business operation status within the preset time period.

[0129] The third aspect of the present invention provides a computer-readable storage medium, which includes a business log identification and evaluation method program based on big data. When the business log identification and evaluation method program based on big data is executed by a processor, the steps of the business log identification and evaluation method based on big data as described in any one of the above items are implemented.

[0130] The present invention discloses a business log identification and evaluation method, system and medium based on big data, which monitors the business system in real time and collects abnormal log data sets within a preset time period, and performs data preprocessing on the abnormal log data sets, performs parsing processing on the processed abnormal log data sets through a preset language behavior processing model to obtain abnormal log behavior information, extracts feature data based on the abnormal log behavior information, and inputs the feature data into a preset business abnormal log identification model, processes to obtain a log abnormality identification index, performs a threshold comparison between the log abnormality identification index and a preset log abnormality threshold, and obtains a corresponding preset adjustment plan based on the threshold comparison result to perform data correction, information adjustment or monitoring alarm on the business log; thereby realizing business log identification and evaluation through big data and obtaining adjustment plans and monitoring alarms, as well as graphical presentation of the evaluation results of the business operation status.

[0131] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0132] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0133] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0134] Those skilled in the art can understand that all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions, and the aforementioned program can be stored in a readable storage medium. When the program is executed, it executes the steps of the above method embodiments; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), disks or optical disks, and other media that can store program codes.

[0135] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product that contributes to the prior art. The software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

Claims

1. A business log identification and evaluation method based on big data, characterized in that: The following steps are involved: Monitor the business system in real time and collect abnormal log data sets within a preset time period, and pre-process the abnormal log data sets to obtain processed abnormal log data sets; The abnormal log data set is parsed and processed by a preset language behavior processing model to obtain abnormal log behavior information; Extracting feature data according to the abnormal log behavior information, and inputting the feature data into a preset business abnormal log recognition model to obtain a log abnormality recognition index; A threshold comparison is performed based on the log anomaly identification index and a preset log anomaly threshold, and a corresponding preset adjustment plan is obtained based on the threshold comparison result to perform data correction, information adjustment or monitoring alarm on the business log.

2. The business log identification and evaluation method based on big data according to claim 1 is characterized in that: The real-time monitoring of the business system and collecting the abnormal log data set within a preset time period, and performing data preprocessing on the abnormal log data set to obtain the processed abnormal log data set specifically includes: Monitor business systems in real time and collect abnormal log data sets within a preset time period, including abnormal log timestamp data, log abnormality level data, abnormality type data, stack trace data, business abnormality identification data, and error handling data; The abnormal log data set is preprocessed to obtain a cleaned and optimized abnormal log data set.

3. The business log identification and evaluation method based on big data according to claim 2 is characterized in that: The processing of the processed abnormal log data set by using a preset language behavior processing model to obtain abnormal log behavior information specifically includes: Obtain a trained preset language behavior processing model; The abnormal log data set is parsed and processed by the preset language behavior processing model to obtain abnormal log behavior information; The abnormal log behavior information includes: business process interruption information, business data abnormal change information, business rule violation information, transaction abnormality information, order status abnormality information, customer behavior abnormality information, business indicator abnormal fluctuation information and business response abnormality information.

4. The business log identification and evaluation method based on big data according to claim 3 is characterized in that: The extracting feature data according to the abnormal log behavior information, and inputting the feature data into a preset business abnormal log recognition model, and processing to obtain a log abnormality recognition index specifically includes: Extract feature data based on the abnormal log behavior information, the feature data including abnormal behavior discovery frequency data, abnormal duration data, abnormal behavior degree classification data, abnormal behavior time distribution feature data, behavior deviation data, and key indicator impact value data corresponding to the log behavior information; Input the abnormal behavior discovery frequency data, abnormal duration data, abnormal behavior degree classification data, abnormal behavior time distribution characteristic data, behavior deviation data and key indicator impact value data into a preset business abnormality log recognition model, and process to obtain a log abnormality recognition index; The calculation formula of the log anomaly identification index is: Among them, A C is the log anomaly identification index, q w 、e r ,t y 、u i , o p 、s d The abnormal behavior discovery frequency data, abnormal duration data, abnormal behavior degree classification data, abnormal behavior time distribution characteristic data, behavior deviation data and key indicator impact value data are respectively p is the business system operation compensation factor, ε, η and λ are preset characteristic coefficients.

5. The business log identification and evaluation method based on big data according to claim 4 is characterized in that: The step of comparing the log anomaly identification index with a preset log anomaly threshold and obtaining a corresponding preset adjustment plan to correct data, adjust information or monitor and alarm the business log according to the threshold comparison result specifically includes: Performing a threshold comparison between the log anomaly identification index and a preset log anomaly threshold level to obtain a corresponding threshold comparison result threshold level; Obtaining the corresponding log abnormality level and business log operation adjustment plan according to the threshold comparison result threshold level; According to the business log operation adjustment plan, data correction, information adjustment or monitoring alarm is performed on the business log to achieve identification and evaluation of the log data of the business system.

6. The business log identification and evaluation method based on big data according to claim 5 is characterized in that: Also includes: The log abnormality level and the business log operation adjustment plan are recorded and displayed through a preset business log monitoring table, and different preset font colors are used to display the corresponding log abnormality level and the business log operation adjustment plan; Collect a set of log anomaly levels within a preset time period, including log anomaly levels corresponding to multiple preset time periods; Processing is performed according to the log abnormality level set to obtain the business log abnormality fluctuation index within the preset time period; Input the log abnormality level set in combination with the business log abnormal fluctuation index into the preset business log monitoring table for recording, and obtain a business log operation status monitoring chart, thereby obtaining a chart presentation and display identification of the business operation status; The calculation formula of the business log abnormal fluctuation index is: Among them, K x is the abnormal fluctuation index of business log, A ci is the log anomaly identification index of the i-th sampling period node in the log anomaly level set, n is the number of log anomaly identification indexes in the log anomaly level set, ψ i is the preset characteristic coefficient of the log anomaly level of the node in the i-th sampling period.

7. A business log identification and evaluation system based on big data, characterized in that: The system includes: a memory and a processor, wherein the memory includes a program of a business log identification and evaluation method based on big data, and when the program of the business log identification and evaluation method based on big data is executed by the processor, the following steps are implemented: Monitor the business system in real time and collect abnormal log data sets within a preset time period, and pre-process the abnormal log data sets to obtain processed abnormal log data sets; The abnormal log data set is parsed and processed by a preset language behavior processing model to obtain abnormal log behavior information; Extracting feature data according to the abnormal log behavior information, and inputting the feature data into a preset business abnormal log recognition model to obtain a log abnormality recognition index; A threshold comparison is performed based on the log anomaly identification index and a preset log anomaly threshold, and a corresponding preset adjustment plan is obtained based on the threshold comparison result to perform data correction, information adjustment or monitoring alarm on the business log.

8. The business log identification and evaluation system based on big data according to claim 7 is characterized in that: The real-time monitoring of the business system and collecting the abnormal log data set within a preset time period, and performing data preprocessing on the abnormal log data set to obtain the processed abnormal log data set specifically includes: Monitor business systems in real time and collect abnormal log data sets within a preset time period, including abnormal log timestamp data, log abnormality level data, abnormality type data, stack trace data, business abnormality identification data, and error handling data; The abnormal log data set is preprocessed to obtain a cleaned and optimized abnormal log data set.

9. The business log identification and evaluation system based on big data according to claim 8 is characterized in that: The processing of the processed abnormal log data set by using a preset language behavior processing model to obtain abnormal log behavior information specifically includes: Obtain a trained preset language behavior processing model; The abnormal log data set is parsed and processed by the preset language behavior processing model to obtain abnormal log behavior information; The abnormal log behavior information includes: business process interruption information, business data abnormal change information, business rule violation information, transaction abnormality information, order status abnormality information, customer behavior abnormality information, business indicator abnormal fluctuation information and business response abnormality information.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a business log identification and evaluation method program based on big data. When the business log identification and evaluation method program based on big data is executed by a processor, the steps of the business log identification and evaluation method based on big data as described in any one of claims 1 to 6 are implemented.

Images