Log data quality monitoring method and device, electronic equipment, medium and program

Abstract

The embodiment of the invention discloses a log data quality monitoring method and device, electronic equipment, a medium and a program, and the method comprises the steps: obtaining target monitoring log data; calculating a fuzzy monitoring level of each log quality index of the target monitoring log data according to a fuzzy level calculation rule; wherein the fuzzy monitoring level is identified through different priority levels; and determining the business influence degree of the target monitoring log data according to the fuzzy monitoring level of the log quality index. According to the technical scheme, the false alarm rate of the log data can be reduced, and the accuracy of log data monitoring is improved.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of data processing technology, and in particular to a log data quality monitoring method, device, electronic device, storage medium and program. Background Art

[0002] With the popularity of large-scale distributed systems, the amount of log data has increased dramatically. Traditional data quality monitoring methods are difficult to cope with massive data and complex scenarios. In particular, the quality of log data has an important impact on system stability and business operations. How to accurately evaluate the quality of log data and predict its impact on the business has become a major challenge in operation and maintenance and data governance.

[0003] In the prior art, most log quality monitoring systems use simple rules or thresholds to determine the quality level of log data, which makes it difficult to fully consider the relationship between various quality indicators and is unable to dynamically adapt to complex business environments. Summary of the invention

[0004] The embodiments of the present invention provide a log data quality monitoring method, device, electronic device, storage medium and program, which can reduce the false alarm rate of log data and improve the accuracy of log data monitoring.

[0005] According to one aspect of the present invention, a log data quality monitoring method is provided, comprising:

[0006] Get target monitoring log data;

[0007] Calculating the fuzzy monitoring level of each log quality indicator of the target monitoring log data according to the fuzzy level calculation rule; wherein the fuzzy monitoring level is identified by different priority levels;

[0008] The business impact degree of the target monitoring log data is determined according to the fuzzy monitoring level of the log quality indicator.

[0009] According to another aspect of the present invention, there is provided a log data quality monitoring device, comprising:

[0010] Target monitoring log data module acquisition, used to obtain target monitoring log data;

[0011] A fuzzy monitoring level calculation module, used to calculate the fuzzy monitoring level of each log quality indicator of the target monitoring log data according to a fuzzy level calculation rule; wherein the fuzzy monitoring level is identified by different priority levels;

[0012] The business impact degree determination module is used to determine the business impact degree of the target monitoring log data according to the fuzzy monitoring level of the log quality indicator.

[0013] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0014] at least one processor; and

[0015] a memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the log data quality monitoring method described in any embodiment of the present invention.

[0017] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the log data quality monitoring method described in any embodiment of the present invention when executed.

[0018] According to another aspect of the present invention, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the method for monitoring log data quality according to any embodiment of the present invention is implemented.

[0019] After acquiring the target monitoring log data, the embodiment of the present invention calculates the fuzzy monitoring level of each log quality indicator of the target monitoring log data according to the fuzzy level calculation rule, and then determines the business impact of the target monitoring log data according to the fuzzy monitoring level of the log quality indicator. Since the fuzzy monitoring level is identified by different priority levels, it is possible to achieve flexible processing of the log monitoring results, thereby avoiding false alarms of the log data monitoring results, solving the problems of high false alarm rate and low accuracy of existing log data monitoring, and being able to reduce the false alarm rate of log data and improve the accuracy of log data monitoring.

[0020] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 is a flow chart of a log data quality monitoring method provided by Embodiment 1 of the present invention;

[0023] Figure 2 is a flow chart of a log data quality monitoring method provided by Embodiment 2 of the present invention;

[0024] Figure 3 is a schematic diagram of a log data quality monitoring device provided by Embodiment 3 of the present invention;

[0025] Figure 4 A schematic diagram of the structure of an electronic device provided in Embodiment 4 of the present invention. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0027] It should be noted that the terms "include", "have" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products or apparatus.

[0028] Embodiment 1

[0029] Figure 1 This is a flow chart of a log data quality monitoring method provided by the first embodiment of the present invention. This embodiment is applicable to the case where the log data quality is monitored by fuzzy evaluation. The method can be performed by a log data quality monitoring device, which can be implemented by software and / or hardware and can generally be integrated in an electronic device. The electronic device can be a terminal device or a server device. As long as it can perform log data quality monitoring, the embodiment of the present invention does not limit the specific device type of the electronic device. Accordingly, Figure 1 As shown, the method includes the following operations:

[0030] S110, obtaining target monitoring log data.

[0031] The target monitoring log data may be associated log data collected from the target business application. The target business application may be a business application type that needs to determine the business impact through log monitoring. The embodiment of the present invention does not limit the target monitoring log data and the type of target business application generated.

[0032] When it is necessary to evaluate the quality of log data and further predict the business impact of the target business application based on the log data quality evaluation results, the target monitoring log data can be collected for the target business application. In an embodiment of the present invention, the target monitoring log data can be obtained by the log monitoring system. Optionally, the log monitoring system can be configured with a data acquisition module, and the original log data is collected in real time from the log source associated with the target business application through the data acquisition module. Optionally, the log source may include but is not limited to application logs, operating system logs, and network logs. Accordingly, after collecting and acquiring the original log data, the log monitoring system can clean, parse and standardize the original log data through the configured data preprocessing module to generate structured log records as the target monitoring log data.

[0033] S120. Calculate the fuzzy monitoring level of each log quality indicator of the target monitoring log data according to a fuzzy level calculation rule; wherein the fuzzy monitoring level is identified by different priority levels.

[0034] Among them, the fuzzy level calculation rules may be relevant calculation rules for calculating the fuzzy monitoring level of each log quality indicator of the target monitoring log data. Log quality indicators may be indicators used to evaluate the quality of log data, for example, may include but are not limited to completeness, accuracy, consistency, timeliness, uniqueness and effectiveness, etc. Optionally, the log quality indicator may also be a custom indicator type, and the embodiment of the present invention does not limit the specific indicator type and quantity of the log quality indicator. The fuzzy monitoring level may be a monitoring level calculated by a certain flexible processing method. The so-called flexible processing method is to set a certain tolerance for the data, that is, to allow some insignificant errors in the data, or to lower the criteria for judging certain error types in the data.

[0035] In an embodiment of the present invention, the quality of the target monitoring log data can be evaluated in a fuzzy monitoring level manner. Specifically, the types of each log quality indicator of the target monitoring log data can be determined first. Optionally, a variety of common log quality indicators and customized log quality indicators can be used to evaluate the quality of the log data. Exemplarily, common indicators of log quality indicators include the following: Completeness: Check whether the log record contains all necessary fields, such as timestamp, log level and message content; Accuracy: Whether the data of the log record accurately reflects the system status; Consistency: Whether the same event records in different log sources are consistent; Timeliness: Whether the time interval of log generation meets expectations; Uniqueness: Whether the log record is repeated; Validity: Whether the log field value conforms to the predefined format, and the evaluation value of validity can be obtained by weighted calculation based on the results of indicators such as completeness, accuracy and consistency.

[0036] Accordingly, after determining the log quality index applicable to the evaluation of the log data quality, the fuzzy monitoring level of each log quality index of the target monitoring log data can be calculated according to the pre-configured fuzzy level calculation rules. The fuzzy monitoring level can be understood as the quality evaluation result of the fuzzy calculation of the target monitoring log data. Optionally, the fuzzy monitoring level can include multiple different priority levels, that is, identified by different priority levels. For example, the fuzzy monitoring level of integrity may include but is not limited to three priority levels of high, medium and low, and the fuzzy monitoring level of accuracy may also include but is not limited to three priority levels of high, medium and low. The configuration of the fuzzy monitoring level of different types of log quality indicators may be the same or different, and the embodiment of the present invention does not limit this. For example, the fuzzy monitoring level of integrity may include but is not limited to three priority levels of high, medium and low, and the fuzzy monitoring level of accuracy may include but is not limited to two priority levels of high and low.

[0037] The fuzzy monitoring level is divided into different priority levels, and the fuzzy monitoring level of each priority level can correspond to the quality assessment results covering a variety of flexible processing, which can avoid the log data that should not be warned from being misjudged as low-quality logs, thereby reducing the false alarm rate of log data and improving the accuracy of log data monitoring.

[0038] S130: Determine the business impact degree of the target monitoring log data according to the fuzzy monitoring level of the log quality indicator.

[0039] The business impact level reflects the severity of the problem in the target business application.

[0040] After evaluating and obtaining the fuzzy monitoring level of each log quality indicator of the target monitoring log data, the fuzzy monitoring level can be used as a basis to further determine the business impact of the target monitoring log data. Optionally, the higher the level of the fuzzy monitoring level, the less ideal the log quality indicator is, and the more serious the problem in the target monitoring log data is, and it can be further determined that the business impact of the target monitoring log data is more serious. Exemplarily, when the fuzzy monitoring levels of each log quality indicator of the target monitoring log data calculated according to the fuzzy level calculation rules are all low, it indicates that the quality of the target monitoring log data is high, and it can be further determined that the business impact of the target monitoring log data is low.

[0041] After acquiring the target monitoring log data, the embodiment of the present invention calculates the fuzzy monitoring level of each log quality indicator of the target monitoring log data according to the fuzzy level calculation rule, and then determines the business impact of the target monitoring log data according to the fuzzy monitoring level of the log quality indicator. Since the fuzzy monitoring level is identified by different priority levels, it is possible to achieve flexible processing of the log monitoring results, thereby avoiding false alarms of the log data monitoring results, solving the problems of high false alarm rate and low accuracy of existing log data monitoring, and being able to reduce the false alarm rate of log data and improve the accuracy of log data monitoring.

[0042] Embodiment 2

[0043] Figure 2 is a flow chart of a log data quality monitoring method provided by the second embodiment of the present invention. This embodiment is specific based on the above embodiment. In this embodiment, multiple specific optional implementation methods for calculating the fuzzy monitoring level of each log quality indicator of the target monitoring log data and determining the business impact of the target monitoring log data are provided. Figure 2 As shown, the method of this embodiment may include:

[0044] S210: Obtain target monitoring log data.

[0045] In an embodiment of the present invention, in order to calculate the fuzzy monitoring level of each log quality indicator of the target monitoring log data, the collected target monitoring log data can be processed by fuzzy logic rules, cleaned and parsed into a structured format, and fuzzy tolerance processing can be performed on field deviations.

[0046] Specifically, the target monitoring log data collected and acquired through fuzzy logic rule processing may include the following steps:

[0047] (1) Fuzzy data cleaning.

[0048] The target monitoring log data that is missing some fields is marked instead of being directly filtered and discarded. For example, if log entry 4 in the target monitoring log data is missing a timestamp, the log monitoring system can infer the missing timestamp through context analysis and mark the log entry as "partially complete".

[0049] (2) Fuzzy data analysis.

[0050] Optionally, a regular expression may be used to parse the log structure of the target monitoring log data and convert it into a field table, and missing fields may be marked as empty, as shown in Table 1, where "N / A" in Table 1 and subsequent tables represents an empty value.

[0051] Table 1 Structured field table

[0052]

[0053] In an optional embodiment of the present invention, the log quality indicators may include but are not limited to completeness, accuracy, consistency and effectiveness.

[0054] Accordingly, the above method may further include the following steps:

[0055] S220. Calculate the integrity fuzzy monitoring level of the target monitoring log data according to the integrity calculation rule in the fuzzy level calculation rule.

[0056] The integrity calculation rule may be a data processing rule for calculating the integrity fuzzy monitoring level. The integrity fuzzy monitoring level may be a fuzzy monitoring level of a certain dimension calculated according to a log quality index of the integrity of the target monitoring log data.

[0057] S230. Calculate the accuracy fuzzy monitoring level of the target monitoring log data according to the accuracy calculation rule in the fuzzy level calculation rule.

[0058] The accuracy calculation rule may be a data processing rule for calculating the accuracy fuzzy monitoring level. The accuracy fuzzy monitoring level may be a fuzzy monitoring level of a certain dimension calculated according to a log quality index of the accuracy of the target monitoring log data.

[0059] S240. Calculate the consistency fuzzy monitoring level of the target monitoring log data according to the consistency calculation rule in the fuzzy level calculation rule.

[0060] The consistency calculation rule may be a data processing rule for calculating the consistency fuzzy monitoring level. The consistency fuzzy monitoring level may be a fuzzy monitoring level of a certain dimension calculated according to the log quality index of the consistency of the target monitoring log data.

[0061] S250. Calculate the validity fuzzy monitoring level of the target monitoring log data according to the integrity fuzzy monitoring level, the accuracy fuzzy monitoring level and the consistency fuzzy monitoring level.

[0062] The effectiveness fuzzy monitoring level may be a fuzzy monitoring level of a certain dimension calculated based on relevant log quality indicators such as integrity, accuracy and consistency of target monitoring log data.

[0063] It is understandable that different log quality indicators can be configured to correspond to different fuzzy level calculation rules. The fuzzy level calculation rules specify the fuzzy monitoring level calculation method for different deviation fields in each target monitoring log data. Specifically, the integrity fuzzy monitoring level of the target monitoring log data can be calculated according to the integrity calculation rules in the fuzzy level calculation rules, the accuracy fuzzy monitoring level of the target monitoring log data can be calculated according to the accuracy calculation rules in the fuzzy level calculation rules, and the consistency fuzzy monitoring level of the target monitoring log data can be calculated according to the consistency calculation rules in the fuzzy level calculation rules. Finally, the validity fuzzy monitoring level of the target monitoring log data can be calculated according to the integrity fuzzy monitoring level, the accuracy fuzzy monitoring level and the consistency fuzzy monitoring level. Alternatively, the validity fuzzy monitoring level of the target monitoring log data can also be calculated according to the validity calculation rules in the fuzzy level calculation rules, and the embodiments of the present invention do not limit the calculation method of the validity fuzzy monitoring level.

[0064] Exemplarily, when a log entry is missing a timestamp, the integrity fuzzy monitoring level of the log entry may be medium. When the IP address format of a log entry is correct, the validity fuzzy monitoring level of the log entry may be high. When a message field of a log entry is missing, the accuracy fuzzy monitoring level of the log entry may be medium.

[0065] The fuzzy level calculation rules corresponding to different log quality indicators can further distinguish the detailed level types of calculated log quality indicators, such as timestamp integrity, log level validity, IP address validity, host name integrity, message integrity and process ID consistency, as shown in the fuzzy monitoring level refinement list in Table 2.

[0066] Table 2 Fuzzy monitoring level refinement list

[0067] Timestamp integrity Log level validity IP Address Validity Hostname integrity Message Integrity Process ID consistency high high high Low high middle high high N / A high high N / A high high N / A N / A high high middle high high N / A high N / A

[0068] In an optional embodiment of the present invention, the calculation of the integrity fuzzy monitoring level of the target monitoring log data according to the integrity calculation rules in the fuzzy level calculation rules may include: calculating the integrity fuzzy monitoring level according to the full field integrity, secondary field integrity and main field integrity of the target monitoring log data; the calculation of the accuracy fuzzy monitoring level of the target monitoring log data according to the accuracy calculation rules in the fuzzy level calculation rules may include: calculating the accuracy fuzzy monitoring level according to the data deviation degree of the target monitoring log data; the calculation of the consistency fuzzy monitoring level of the target monitoring log data according to the consistency calculation rules in the fuzzy level calculation rules may include: calculating the consistency fuzzy monitoring level according to the consistency of different log sources of the same event of the target monitoring log data, the consistency of secondary fields and the consistency of key fields; the calculation of the validity fuzzy monitoring level of the target monitoring log data according to the integrity fuzzy monitoring level, the accuracy fuzzy monitoring level and the consistency fuzzy monitoring level may include: calculating the weighted value of the integrity fuzzy monitoring level, the accuracy fuzzy monitoring level, the consistency fuzzy monitoring level and the corresponding weights to obtain the validity fuzzy monitoring level.

[0069] Among them, full field integrity refers to the integrity of all fields in the log entry. Secondary field integrity can represent the missing of secondary fields. Primary field integrity can represent the missing of primary fields. The consistency of different log sources for the same event can represent the consistency of the same event recorded in different log sources.

[0070] Specifically, the integrity calculation rules can calculate the integrity fuzzy monitoring level of the target monitoring log data based on the integrity of all fields, the integrity of secondary fields and the integrity of primary fields. The accuracy calculation rules can calculate the accuracy fuzzy monitoring level of the target monitoring log data based on the degree of data deviation. The consistency calculation rules can calculate the consistency fuzzy monitoring level of the target monitoring log data based on the consistency of different log sources for the same event, the consistency of secondary fields and the consistency of key fields. Finally, the weighted values ​​of the integrity fuzzy monitoring level, the accuracy fuzzy monitoring level, the consistency fuzzy monitoring level and the corresponding weights can be calculated to obtain the effectiveness fuzzy monitoring level. Optionally, each type of fuzzy monitoring level is configured with a fuzzy evaluation score. The fuzzy evaluation score can be used to determine the evaluation result of the corresponding log quality indicator in the form of a score value.

[0071] In a specific example, when calculating the integrity fuzzy monitoring level according to the full field integrity, secondary field integrity and primary field integrity of the target monitoring log data, the following rules can be referred to: when all fields are complete, the integrity fuzzy monitoring level is high; when the secondary field is missing, the integrity fuzzy monitoring level is medium; when the primary field is missing, the integrity fuzzy monitoring level is low, and the fuzzy evaluation score is 0.2. When calculating the accuracy fuzzy monitoring level according to the data deviation degree of the target monitoring log data, the following rules can be referred to: when the data deviation degree is less than or equal to 5%, the accuracy fuzzy monitoring level is high; when the data deviation degree is greater than 5% and less than or equal to 20%, the accuracy fuzzy monitoring level is medium; when the data deviation degree is greater than 20%, the accuracy fuzzy monitoring level is low. When calculating the consistency fuzzy monitoring level according to the consistency of different log sources of the same event of the target monitoring log data, the consistency of the secondary fields and the consistency of the key fields, the following rules can be referred to: when the same event record is completely consistent in different log sources, the consistency fuzzy monitoring level is high; when the secondary fields are inconsistent, the consistency fuzzy monitoring level is medium; when the key fields are inconsistent, the consistency fuzzy monitoring level is low. At the same time, each type of fuzzy monitoring level can be configured with a corresponding fuzzy evaluation score. Exemplarily, when the integrity fuzzy monitoring level is high, the fuzzy evaluation score is 0.9; when the integrity fuzzy monitoring level is medium, the fuzzy evaluation score is 0.5; when the integrity fuzzy monitoring level is low, the fuzzy evaluation score is 0.2. Optionally, the higher the priority level of the fuzzy monitoring level, the larger the corresponding fuzzy evaluation score, indicating that the quality of the target monitoring log data is higher and the risk factor is smaller. On the contrary, the lower the priority level of the fuzzy monitoring level, the smaller the corresponding fuzzy evaluation score, indicating that the quality of the target monitoring log data is worse and the risk factor is greater.

[0072] Furthermore, corresponding weights can be set for the integrity fuzzy monitoring level, the accuracy fuzzy monitoring level, and the consistency fuzzy monitoring level. For example, if the weight of the integrity fuzzy monitoring level is 30%, the accuracy fuzzy monitoring level is 30%, and the consistency fuzzy monitoring level is 40%, then the evaluation result of the effectiveness fuzzy monitoring level can be: the final score calculated based on the expression "the fuzzy evaluation score corresponding to the integrity fuzzy monitoring level * 30% + the fuzzy evaluation score corresponding to the accuracy fuzzy monitoring level * 30% + the fuzzy evaluation score corresponding to the consistency fuzzy monitoring level * 40%".

[0073] Alternatively, the above-mentioned type of fuzzy level calculation rule may be used to calculate the effectiveness fuzzy monitoring level including different priority levels without referring to other log quality indicators, as shown in Table 3.

[0074] Table 3 Fuzzy monitoring level evaluation list of log quality indicators

[0075] Log entries Completeness accuracy consistency Effectiveness 1 high high high high 2 high high N / A high 3 high middle middle high 4 middle middle N / A high

[0076] For each priority level of the log quality indicator, a fuzzy evaluation score can be configured accordingly.

[0077] Optionally, the log monitoring system can also dynamically adjust the fuzzy level calculation rules according to business needs. Exemplarily, if the target business application is an application for high-risk scenarios, such as a payment system, etc., the fuzzy tolerance can be reduced to improve the accuracy of abnormal log capture. If the target business application is an application for low-risk scenarios, such as an auxiliary service application, the fuzzy tolerance can be increased to reduce false alarms. Exemplarily, lowering the fuzzy tolerance can be: if the original integrity calculation rule stipulates that the integrity fuzzy monitoring level of missing secondary fields is medium, the integrity calculation rule can be modified when the fuzzy tolerance is lowered, such as the integrity fuzzy monitoring level of missing secondary fields is low. Exemplarily, increasing the fuzzy tolerance can be: if the original integrity calculation rule stipulates that the integrity fuzzy monitoring level of missing secondary fields is medium, the integrity calculation rule can be modified when the fuzzy tolerance is increased, such as the integrity fuzzy monitoring level of missing secondary fields is high.

[0078] S260: Calculate the indicator fuzzy evaluation score of the target monitoring log data according to the fuzzy evaluation score of the fuzzy monitoring level of the log quality indicator.

[0079] The indicator fuzzy evaluation score may be evaluation data obtained by calculating the target monitoring log data using the fuzzy evaluation score.

[0080] Since each fuzzy monitoring level can be configured with a corresponding fuzzy evaluation score, after calculating the corresponding fuzzy monitoring level for each log quality indicator of the target monitoring log data, the sum of the fuzzy evaluation scores of each fuzzy monitoring level can be calculated to obtain the indicator fuzzy evaluation score of the target monitoring log data.

[0081] S270: Determine the business impact of the target monitoring log data according to the indicator fuzzy evaluation score of the target monitoring log data.

[0082] Optionally, the smaller the indicator fuzzy evaluation score is, the worse the quality of the target monitoring log data is, and the more serious the business impact of the target monitoring log data is. Different indicator fuzzy evaluation scores can determine different business impacts. Generally, the higher the indicator fuzzy evaluation score is, the lower the business impact is.

[0083] Exemplarily, it is assumed that the integrity fuzzy monitoring level of the target monitoring log data is high, the accuracy fuzzy monitoring level is high, the consistency fuzzy monitoring level is low, and the effectiveness fuzzy monitoring level is high. If the fuzzy evaluation score of the indicator with a high integrity fuzzy monitoring level is 0.9, the fuzzy evaluation score of the indicator with a high accuracy fuzzy monitoring level is 0.8, the fuzzy evaluation score of the indicator with a low consistency fuzzy monitoring level is 0.1, and the fuzzy evaluation score of the indicator with a high effectiveness fuzzy monitoring level is 0.8, then the fuzzy evaluation score of the indicator of the target monitoring log data is 0.9+0.8+0.1+0.8=2.6. Assuming that the value of the business impact degree determined according to the fuzzy evaluation score of the indicator is 0.2, and the assessment threshold of the business impact degree is 0.5, when the fuzzy evaluation score of the indicator of the target monitoring log data is 2.6, it indicates that the quality of the target monitoring log data is high, its business impact degree is small, and the log quality problem has no obvious impact on the business.

[0084] In an optional embodiment of the present invention, determining the business impact degree of the target monitoring log data based on the indicator fuzzy evaluation score of the target monitoring log data may include: calculating the indicator precise evaluation score of each log quality indicator of the target monitoring log data according to the precise score calculation rule; weighting the indicator precise evaluation score of each log quality indicator and the indicator fuzzy evaluation score; determining the business impact degree of the target monitoring log data based on the target evaluation score and business indicators of the target monitoring log data.

[0085] Among them, the precise score calculation rule can be a calculation rule for calculating the precise evaluation score of the target monitoring log data. The indicator precise evaluation score can be a score obtained by specifically calculating a certain indicator of the target monitoring log data. For example, the indicator precise evaluation score can be a completeness score of 96 points. The target evaluation score can be the final score calculated after combining the precise evaluation score of the indicator and the fuzzy evaluation score of the indicator.

[0086] In a specific example, when calculating the accuracy evaluation score of the completeness indicator, the log monitoring system can check whether each log contains necessary fields (such as timestamp, log level, message content, etc.). Assume that among one million logs, 950,000 are complete, and the score is: S1 = 950,000 / 1,000,000 × 100 = 95. When calculating the accuracy evaluation score of the indicator, the log monitoring system can compare the key fields in the log with the actual system status. For example, whether the user ID recorded in the log exists in the user database. Assume that after comparison, it is found that 900,000 records are accurate, and the score is: S2 = 900,000 / 1,000,000 × 100 = 90. When calculating the consistency evaluation score of the indicator, the log monitoring system can check whether the records of the same transaction ID in different log sources are consistent. Assume that the consistency of 850,000 transaction IDs is verified, and the score is: S3 = 850,000 / 1,000,000 × 100 = 85. When calculating the precise evaluation score of the timeliness indicator, the log monitoring system can detect whether the generation and collection of logs meet the expected time interval. For example, it is expected to collect 10,000 logs per minute, and whether the actual collection volume meets the requirements. Assuming the timeliness rate is 92%, the score is: S4 = 92. When calculating the precise evaluation score of the uniqueness indicator, the log monitoring system can check whether there are duplicate log entries. Assuming that 50,000 duplicates are found, the score is: S5 = (1-50000 / 1000000) × 100 = 95. When calculating the precise evaluation score of the effectiveness indicator, the log monitoring system can verify whether the log field meets the format requirements. For example, the timestamp format, the value range of the log level, etc. The verification results of each field are synthesized according to a certain weight to calculate the overall effectiveness score. Assuming that the weight of each field verification is equal, the calculation formula of S6 can be expressed as:

[0087]

[0088] Where n is the number of fields to be verified, for example, the number of timestamp and log level fields. i For the lower i fields, Valid(f i ) is the field verification result. If f i The result is 1 if the format requirement is met. i The format requirement is not met, and the result is 0.

[0089] For example, assume that the log record has three fields: timestamp, log level, and IP address. If the timestamp format is correct: score 1, the log level is invalid: score 0, and the IP address format is correct: score 1, then S6 is calculated as: S6 = 1 / 3 (1 + 0 + 1) = 23 ≈ 0.67, and this value is multiplied by 100, and the final validity score is 67.

[0090] Optionally, the weight of the indicator's precise evaluation score and the weight of the indicator's fuzzy evaluation score can be configured as needed. For example, the weight of the indicator's precise evaluation score can be 0.5, and the weight of the indicator's fuzzy evaluation score can be 0.5. In actual operation, the weight of the indicator's precise evaluation score and the weight of the indicator's fuzzy evaluation score can be dynamically adjusted according to the type of target business application and specific log monitoring requirements. As long as the sum of the weight of the indicator's precise evaluation score and the weight of the indicator's fuzzy evaluation score is 1, the embodiment of the present invention does not limit the specific values ​​of the weights of the indicator's precise evaluation score and the indicator's fuzzy evaluation score.

[0091] Before determining the business impact of the target monitoring log data according to the target evaluation score of the target monitoring log data, a model that can determine the business impact of the target monitoring log data according to the target evaluation score of the target monitoring log data can be first trained. Since the relationship between the log quality evaluation score and the business impact is often nonlinear, a gradient boosting decision tree (GBDT) model can be optionally used to automatically predict the relationship between the log quality score and the business impact. The GBDT model is a powerful ensemble learning algorithm that trains multiple decision trees through multiple iterations to gradually optimize the model prediction error. GBDT is particularly good at dealing with regression problems and has good modeling capabilities for nonlinear relationships between features. GBDT can handle the complex relationship between the log quality evaluation score and the business impact, and can automatically select the features that have the greatest impact on the results during the training process, thereby improving the efficiency of the model. At the same time, by controlling the depth of the tree and using regularization, GBDT can effectively prevent overfitting when processing high-dimensional data.

[0092] Before using GBDT to determine the business impact of the target monitoring log data according to the target evaluation score of the target monitoring log data, training data can be first collected to train GBDT. Optionally, the evaluation score of the log sample data can be combined with the business indicator through feature engineering to obtain a sample data set for model training. The log monitoring system can collect business-related indicators in a variety of ways. Exemplarily, the business indicators may include but are not limited to: system failure rate: obtain the failure rate through the monitoring system, such as the number of failures that occur per hour in the system; number of alarms: obtain the number of log alarms within a certain period of time from the alarm management system, such as the number of alarms triggered per minute; number of user complaints: collect user complaint data through the customer service system, such as the total number of complaints recorded per week. Further, the association relationship between the evaluation score of the log sample data and the business indicator can be established. Optionally, the evaluation score of the log sample data can be an indicator fuzzy evaluation score, or it can also be an evaluation score that is weighted and calculated by combining the indicator precise evaluation score and the indicator fuzzy evaluation score. The embodiment of the present invention does not limit this. Exemplarily, the association relationship between the evaluation score of the log sample data and the business indicator is shown in Table 4. Accordingly, the sample data set used for training the model may include the correlation between the evaluation score and the business indicator, and the corresponding business impact degree. For example, the sample data set may be as shown in Table 5.

[0093] Table 4 Correlation between evaluation scores of log sample data and business indicators

[0094] Completeness accuracy consistency Effectiveness Failure rate Number of alarms Number of user complaints 0.8 0.8 0.9 0.9 0.01 5 2 0.9 0.5 0.8 0.5 0.03 15 6 0.5 0.6 0.5 0.8 0.05 30 10 0.2 0.1 0.4 0.4 0.10 50 20

[0095] Table 5 Sample dataset example list

[0096] Completeness accuracy consistency Effectiveness Failure rate Number of alarms Number of user complaints Business impact 0.8 0.8 0.9 0.9 0.01 5 2 0.1 0.9 0.5 0.8 0.5 0.03 15 6 0.4 0.5 0.6 0.5 0.8 0.05 30 10 0.6 0.2 0.1 0.4 0.4 0.10 50 20 0.85

[0097] During the GBDT model training process, the sample data set can be split. For example, the sample data set can be divided into a training set and a test set in an 80% / 20% ratio. The training set is used to train the model, and the test set is used to evaluate the performance of the model. During the model training process, the distributed gradient boosting library can be used to implement the gradient boosting tree algorithm and perform model training on the training set. During the training process, the model parameters are continuously optimized by minimizing loss functions such as the mean square error. After the model training is completed, the model can be evaluated using the test set. Evaluation indicators can generally include but are not limited to: Mean Squared Error (MSE): measures the difference between the model prediction and the actual value; Coefficient of Determination (R 2 ): Measures the model's ability to explain data variation. At the same time, the hyperparameters of GBDT, such as learning rate and tree depth, can be adjusted through methods such as cross-validation and grid search to improve the accuracy and generalization ability of the model.

[0098] After the model is trained and tested, the model can be used to predict the business impact of the target monitoring log data based on the target evaluation score of the target monitoring log data. Through this complex data processing process, the log monitoring system realizes efficient log quality monitoring based on fuzzy logic, and dynamically predicts the impact of logs on the business through machine learning, realizing flexible early warning and precise operation and maintenance.

[0099] In a specific example, the business impact can be calculated based on the following formula: Business impact = α1*completeness score + α2*accuracy score + α3*consistency score + α4*timeliness score + α5*uniqueness score + α6*effectiveness score + β1*failure rate + β2*number of alarms + β3*number of user complaints. Among them, αi and βi are weight coefficients automatically learned during model training. These weights reflect the impact of different log quality indicators and business indicators on the final business impact prediction results. Through gradient boosting tree training, these coefficients are continuously optimized to achieve more accurate predictions.

[0100] In an optional embodiment of the present invention, after determining the business impact of the target monitoring log data according to the fuzzy monitoring level of the log quality indicator, it may also include: determining a division threshold of a multi-level impact level according to historical business impact data; optimizing and updating the division threshold of the multi-level impact level according to threshold adjustment associated factors; wherein the threshold adjustment associated factors include business demand factors, business feedback factors, adaptive adjustment factors and model output impact factors; and determining the early warning trigger processing flow of the target monitoring log data according to the optimized and updated multi-level impact levels and the business impact of the target monitoring log data.

[0101] Among them, historical business impact data can be relevant data on business impact obtained by historical calculations. Impact levels can be levels divided according to different priorities, used to distinguish different degrees of impact. Threshold adjustment association factors can be reference factors for optimizing and updating the division thresholds for dividing multiple levels of impact. Business demand factors can be threshold adjustment factors composed of specific needs of target business applications. Business feedback factors can be threshold adjustment factors composed of business feedback data. Adaptive adjustment factors can be factors for automated prediction of threshold adjustment. Model output impact factors can be factors for adjusting thresholds based on the impact of model output.

[0102] In an embodiment of the present invention, after determining the business impact of the target monitoring log data, corresponding warning measures can be executed in the form of flexible business warning. Exemplarily, if the business impact is low, such as less than 0.2, no alarm is triggered and only monitoring is performed. If the business impact is moderate, such as between 0.4-0.6, attention can be prompted. If the business impact is high, such as greater than 0.8, a high priority alarm is triggered. In a specific example, assuming that the business impact of log 1 is 0.1, no alarm is triggered; the business impact of log 3 is 0.6, requiring operation and maintenance intervention; the business impact of log 4 is 0.85, triggering an emergency alarm.

[0103] It is understandable that when designing an assessment model for the impact of log data on business, defining the threshold of business impact is a key step. The selection of thresholds is usually based on factors such as historical data analysis, business needs, risk assessment, and business tolerance. Specifically, the following steps can be used to define the thresholds for dividing multiple levels of impact. The log monitoring system also supports users to manually set the thresholds for dividing multiple levels of impact.

[0104] Specifically, the thresholds for dividing the multi-level impact levels can be determined first based on the historical business impact data. The relationship between the target evaluation scores of the historical log data and actual business problems such as system downtime, user complaints, business interruptions, etc. can be analyzed to grade the business impact and obtain multi-level impact levels. For example: Impact level of 0.0 to 0.2: The impact is minimal, and log quality issues have no obvious impact on the business. Impact level of 0.2 to 0.4: The impact is small, and may have a slight impact on some low-priority businesses or non-critical functions. Impact level of 0.4 to 0.6: The impact is medium, and may have a medium impact on key business processes or larger-scale businesses, which requires attention. Impact level of 0.6 to 0.8: The impact is large, and may have a serious impact on business stability and user experience. Impact level of 0.8 to 1.0: The impact is huge, usually indicating a failure in a business-critical system, which may result in business interruption or significant losses.

[0105] After determining the division thresholds of the multi-level impact levels, the division thresholds of the multi-level impact levels can be optimized and updated according to the threshold adjustment associated factors. Exemplarily, the division thresholds can be adjusted by business demand factors of business needs and risk assessment. For example, for high-risk business scenarios involving critical businesses, such as payment systems and order processing systems, the thresholds of the multi-level impact levels are usually lower. For example, if the business impact exceeds 0.3, it can be regarded as a serious impact and immediate measures need to be taken. For low-risk business scenarios involving non-core businesses or less important ancillary services, higher thresholds can be set. For example, an alarm will only be triggered if the impact exceeds 0.6.

[0106] Optionally, the threshold can be adjusted dynamically based on business feedback factors or adaptive adjustment factors. The log monitoring system can dynamically adjust the multi-level impact level division thresholds based on the log data and business feedback data accumulated during operation to better adapt to business changes. The following are several optional methods for dynamically adjusting the threshold:

[0107] (1) Adjust thresholds based on business feedback data: Through cooperation with the business team and the operation and maintenance team, actual business feedback data is collected to determine whether the thresholds for dividing the multi-level impact levels are reasonable. Unreasonable thresholds can be adjusted regularly.

[0108] (2) Adaptive adjustment of thresholds: Using machine learning algorithms, the thresholds for dividing multiple levels of impact are automatically adjusted based on the difference between the model's predicted results on the degree of business impact and the actual business situation. For example, if the business impact predicted by the model is high during a certain period of time, but it does not actually cause major problems, the thresholds for dividing multiple levels of impact can be adjusted to make the model more conservative.

[0109] Optionally, the threshold value may be dynamically adjusted according to the model output influence factor, that is, the division threshold value of multiple levels of influence degree may be defined based on the influence degree of the model output.

[0110] Assume that the default setting method for dividing the threshold of multi-level impact levels based on the output value of the GBDT model is as follows:

[0111] Business impact degree 0.0 to 0.2: The impact on the business is very small and can usually be ignored;

[0112] Business impact level 0.2 to 0.4: The impact on the business is small and may require the attention of operation and maintenance personnel, but does not need to be handled immediately;

[0113] Business impact level 0.4 to 0.6: The impact on business is moderate, which may cause system performance degradation or inconvenience to some users. It is recommended to take appropriate measures, such as checking logs and optimizing the system.

[0114] Business impact level 0.6 to 0.8: The business is greatly impacted and some business functions may have been interrupted. Measures should be taken as soon as possible to resolve the problem.

[0115] Business impact level 0.8 to 1.0: The impact on the business is huge, usually indicating that a serious failure or accident has occurred and emergency measures must be taken immediately to restore the business.

[0116] In a specific example, the log monitoring system can determine the warning trigger processing flow of the target monitoring log data according to the division thresholds of the above-mentioned multi-level impact levels. For example, assuming that the model determines that the business impact of the first data is 0.25 based on the target evaluation score and business indicators of the target monitoring log data, it indicates that the impact on the business is small, and the log monitoring system will continue to monitor without triggering an alarm. When the model determines that the business impact of the first data is 0.45 based on the target evaluation score and business indicators of the target monitoring log data, it indicates that the impact on the business is medium, and the operation and maintenance personnel need to pay attention and make appropriate adjustments, and may need to optimize the log processing flow or check the alarm configuration. When the model determines that the business impact of the first data is 0.65 based on the target evaluation score and business indicators of the target monitoring log data, it indicates that the impact on the business is large and the operation and maintenance personnel need to deal with it urgently, such as finding the root cause of log generation or adding monitoring indicators. When the model determines that the business impact of the first data is 0.85 based on the target evaluation score and business indicators of the target monitoring log data, it indicates that the impact on the business is extremely large and immediate measures need to be taken to solve the problem, such as repairing system failures or suspending affected services.

[0117] Optionally, the log monitoring system can also display the target assessment score and business impact of each log data through a visual interface, so that operation and maintenance personnel can quickly identify potential risk logs. Optionally, the visual interface can display the target assessment score trend chart, business impact prediction chart and warning log list, etc.

[0118] The above technical solution uses fuzzy level calculation rules to calculate the fuzzy monitoring level of each log quality indicator of the target monitoring log data, which can handle uncertainty and improve fault tolerance to tolerate slight deviations, such as missing fields and format errors, and reduce log discards and false alarms. Fuzzy rules can also be dynamically updated to meet new log sources and business needs. At the same time, the division thresholds of the multi-level impact levels are optimized and updated according to the threshold adjustment association factors, and then the warning trigger processing flow of the target monitoring log data is determined according to the optimized and updated multi-level impact levels and the business impact of the target monitoring log data. It can support flexible and detailed evaluation, dynamically adjust the rules of dividing thresholds to adapt to different scenario requirements, and refine the source of log quality problems. The introduction of a model to predict the business impact of the target monitoring log data based on the target evaluation score and business indicators of the target monitoring log data can enhance the prediction and decision-making capabilities, provide fine-grained input for machine learning, improve the prediction accuracy of business impact, and support a more flexible alarm mechanism.

[0119] It should be noted that the relevant information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, data for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data comply with relevant laws, regulations and standards in relevant regions.

[0120] It should be noted that any arrangement and combination of the technical features in the above embodiments also falls within the protection scope of the present invention.

[0121] Embodiment 3

[0122] Figure 3 is a schematic diagram of a log data quality monitoring device provided by Embodiment 3 of the present invention, such as Figure 3 As shown, the device includes: a target monitoring log data module acquisition module 310, a fuzzy monitoring level calculation module 320 and a business impact degree determination module 330, wherein:

[0123] The target monitoring log data module acquires 310, which is used to acquire the target monitoring log data;

[0124] A fuzzy monitoring level calculation module 320, used to calculate the fuzzy monitoring level of each log quality indicator of the target monitoring log data according to a fuzzy level calculation rule; wherein the fuzzy monitoring level is identified by different priority levels;

[0125] The business impact degree determination module 330 is used to determine the business impact degree of the target monitoring log data according to the fuzzy monitoring level of the log quality indicator.

[0126] After acquiring the target monitoring log data, the embodiment of the present invention calculates the fuzzy monitoring level of each log quality indicator of the target monitoring log data according to the fuzzy level calculation rule, and then determines the business impact of the target monitoring log data according to the fuzzy monitoring level of the log quality indicator. Since the fuzzy monitoring level is identified by different priority levels, it is possible to achieve flexible processing of the log monitoring results, thereby avoiding false alarms of the log data monitoring results, solving the problems of high false alarm rate and low accuracy of existing log data monitoring, and being able to reduce the false alarm rate of log data and improve the accuracy of log data monitoring.

[0127] Optionally, the log quality indicators include integrity, accuracy, consistency and validity; the fuzzy monitoring level calculation module 320 is also used to: calculate the integrity fuzzy monitoring level of the target monitoring log data according to the integrity calculation rules in the fuzzy level calculation rules; calculate the accuracy fuzzy monitoring level of the target monitoring log data according to the accuracy calculation rules in the fuzzy level calculation rules; calculate the consistency fuzzy monitoring level of the target monitoring log data according to the consistency calculation rules in the fuzzy level calculation rules; calculate the validity fuzzy monitoring level of the target monitoring log data according to the integrity fuzzy monitoring level, the accuracy fuzzy monitoring level and the consistency fuzzy monitoring level.

[0128] Optionally, the fuzzy monitoring level calculation module 320 is also used to: calculate the integrity fuzzy monitoring level according to the full field integrity, secondary field integrity and main field integrity of the target monitoring log data; calculate the accuracy fuzzy monitoring level according to the degree of data deviation of the target monitoring log data; calculate the consistency fuzzy monitoring level according to the consistency of different log sources of the same event of the target monitoring log data, the consistency of secondary fields and the consistency of key fields; calculate the weighted values ​​of the integrity fuzzy monitoring level, the accuracy fuzzy monitoring level, the consistency fuzzy monitoring level and the corresponding weights to obtain the effectiveness fuzzy monitoring level.

[0129] Optionally, the fuzzy monitoring level is configured with a fuzzy evaluation score, and the business impact degree determination module 330 is also used to: calculate the indicator fuzzy evaluation score of the target monitoring log data according to the fuzzy evaluation score of the fuzzy monitoring level of the log quality indicator; determine the business impact degree of the target monitoring log data according to the indicator fuzzy evaluation score of the target monitoring log data.

[0130] Optionally, the business impact degree determination module 330 is also used to: calculate the precise evaluation score of each log quality indicator of the target monitoring log data according to the precise score calculation rules; weight the precise evaluation score of each log quality indicator and the fuzzy evaluation score of the indicator to calculate the target evaluation score of each log quality indicator; determine the business impact degree of the target monitoring log data according to the target evaluation score of the target monitoring log data.

[0131] Optionally, the above-mentioned device also includes an early warning trigger processing module, which is used to: determine the division threshold of multi-level impact levels based on historical business impact data; optimize and update the division threshold of the multi-level impact levels based on threshold adjustment associated factors; wherein the threshold adjustment associated factors include business demand factors, business feedback factors, adaptive adjustment factors and model output influence factors; determine the early warning trigger processing flow of the target monitoring log data based on the optimized and updated multi-level impact levels and the business impact of the target monitoring log data.

[0132] The above-mentioned log data quality monitoring device can execute the log data quality monitoring method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method. For technical details not described in detail in this embodiment, please refer to the log data quality monitoring method provided by any embodiment of the present invention.

[0133] Since the log data quality monitoring device introduced above is a device that can execute the log data quality monitoring method in the embodiment of the present invention, based on the log data quality monitoring method introduced in the embodiment of the present invention, the technical personnel of the field can understand the specific implementation of the log data quality monitoring device of the present embodiment and its various variations, so how the log data quality monitoring device implements the log data quality monitoring method in the embodiment of the present invention is not introduced in detail here. As long as the technical personnel of the field implement the device adopted by the log data quality monitoring method in the embodiment of the present invention, it belongs to the scope of protection of this application.

[0134] Embodiment 4

[0135] Figure 4 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0136] like Figure 4As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0137] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0138] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 11 executes the various methods and processes described above, such as the log data quality monitoring method.

[0139] Optionally, the log data quality monitoring method may include: obtaining target monitoring log data; calculating the fuzzy monitoring level of each log quality indicator of the target monitoring log data according to a fuzzy level calculation rule; wherein the fuzzy monitoring level is identified by different priority levels; and determining the business impact degree of the target monitoring log data according to the fuzzy monitoring level of the log quality indicator.

[0140] In some embodiments, the log data quality monitoring method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the log data quality monitoring method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the log data quality monitoring method in any other appropriate manner (e.g., by means of firmware).

[0141] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0142] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0143] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0144] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0145] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0146] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0147] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0148] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A log data quality monitoring method, characterized in that: include: Get target monitoring log data; Calculating the fuzzy monitoring level of each log quality indicator of the target monitoring log data according to the fuzzy level calculation rule; wherein the fuzzy monitoring level is identified by different priority levels; The business impact degree of the target monitoring log data is determined according to the fuzzy monitoring level of the log quality indicator.

2. The method according to claim 1, characterized in that The log quality indicators include completeness, accuracy, consistency and effectiveness; The step of calculating the fuzzy monitoring level of each log quality indicator of the target monitoring log data according to the fuzzy level calculation rule includes: Calculate the integrity fuzzy monitoring level of the target monitoring log data according to the integrity calculation rule in the fuzzy level calculation rule; Calculating the accuracy fuzzy monitoring level of the target monitoring log data according to the accuracy calculation rule in the fuzzy level calculation rule; Calculating the consistency fuzzy monitoring level of the target monitoring log data according to the consistency calculation rule in the fuzzy level calculation rule; The validity fuzzy monitoring level of the target monitoring log data is calculated according to the integrity fuzzy monitoring level, the accuracy fuzzy monitoring level and the consistency fuzzy monitoring level.

3. The method according to claim 2, characterized in that The step of calculating the integrity fuzzy monitoring level of the target monitoring log data according to the integrity calculation rule in the fuzzy level calculation rule includes: Calculating the integrity fuzzy monitoring level according to the full field integrity, secondary field integrity and primary field integrity of the target monitoring log data; The step of calculating the accuracy fuzzy monitoring level of the target monitoring log data according to the accuracy calculation rule in the fuzzy level calculation rule includes: Calculating the accuracy fuzzy monitoring level according to the data deviation degree of the target monitoring log data; The step of calculating the consistency fuzzy monitoring level of the target monitoring log data according to the consistency calculation rule in the fuzzy level calculation rule includes: Calculate the consistency fuzzy monitoring level according to the consistency of different log sources of the same event of the target monitoring log data, the consistency of the secondary fields and the consistency of the key fields; The calculating the validity fuzzy monitoring level of the target monitoring log data according to the integrity fuzzy monitoring level, the accuracy fuzzy monitoring level and the consistency fuzzy monitoring level comprises: The weighted value of the integrity fuzzy monitoring level, the accuracy fuzzy monitoring level, the consistency fuzzy monitoring level and the corresponding weights is calculated to obtain the validity fuzzy monitoring level.

4. The method according to claim 1, characterized in that The fuzzy monitoring level is configured with a fuzzy evaluation score, and determining the business impact of the target monitoring log data according to the fuzzy monitoring level of the log quality indicator includes: Calculate the fuzzy evaluation score of the target monitoring log data according to the fuzzy evaluation score of the fuzzy monitoring level of the log quality indicator; The business impact degree of the target monitoring log data is determined according to the indicator fuzzy evaluation score of the target monitoring log data.

5. The method according to claim 4, characterized in that The determining the business impact degree of the target monitoring log data according to the indicator fuzzy evaluation score of the target monitoring log data includes: Calculate the index precision evaluation score of each log quality index of the target monitoring log data according to the precision score calculation rule; Calculate the target evaluation score of each log quality indicator by weighting the precise evaluation score of each log quality indicator and the fuzzy evaluation score of each indicator; The business impact degree of the target monitoring log data is determined according to the target evaluation score of the target monitoring log data.

6. The method according to any one of claims 1 to 5, characterized in that: After determining the service impact degree of the target monitoring log data according to the fuzzy monitoring level of the log quality indicator, the method further includes: Determine the thresholds for dividing multiple levels of impact based on historical business impact data; Optimizing and updating the division thresholds of the multi-level impact levels according to threshold adjustment associated factors; wherein the threshold adjustment associated factors include business demand factors, business feedback factors, adaptive adjustment factors and model output impact factors; According to the optimized and updated multi-level impact levels and the business impact degree of the target monitoring log data, the early warning triggering processing flow of the target monitoring log data is determined.

7. A log data quality monitoring device, characterized in that: include: Target monitoring log data module acquisition, used to obtain target monitoring log data; A fuzzy monitoring level calculation module, used to calculate the fuzzy monitoring level of each log quality indicator of the target monitoring log data according to a fuzzy level calculation rule; wherein the fuzzy monitoring level is identified by different priority levels; The business impact degree determination module is used to determine the business impact degree of the target monitoring log data according to the fuzzy monitoring level of the log quality indicator.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the log data quality monitoring method described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the log data quality monitoring method described in any one of claims 1-6 when executed.

10. A computer program product comprising a computer program / instructions, wherein: When the computer program / instructions are executed by a processor, the log data quality monitoring method described in any one of claims 1 to 6 is implemented.

Images