Abnormal operation determination method, device, equipment and storage medium

Abstract

The embodiment of the present application discloses a method, device, equipment and storage medium for determining abnormal operations. The method includes: obtaining the number of message processing failures and the average message processing delay of real-time streaming operations within a preset time period; determining the probability of abnormal operations based on the preset fuzzy logic reasoning rules according to the number of message processing failures and the average message processing delay; judging whether the probability of abnormal operations is equal to or greater than the preset abnormal operation probability threshold, and if so, determining that the real-time streaming operation is an abnormal operation. The embodiment of the present application realizes the automated determination of abnormal operations, and improves the accuracy and efficiency of abnormal operation determination.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of data processing technology, and in particular, to a method, device, equipment and storage medium for determining abnormal operations. Background Art

[0002] With the development of big data technology, enterprises have higher and higher requirements for the timeliness of data processing. More and more scenarios use real-time stream computing to improve the timeliness of data processing. The scale of real-time stream computing continues to grow. Correspondingly, the abnormal judgment of the running status of real-time stream jobs becomes increasingly important.

[0003] In the prior art, the following processing methods are usually used to determine whether the running status of a real-time streaming job is abnormal: if a message processing failure occurs during the running of a real-time streaming job, the real-time streaming job is considered abnormal and requires manual intervention; or, information such as the number of message processing failures and message delays during the running of the real-time streaming job is collected, and the running status of the real-time streaming job is determined to be abnormal based on a preset empirical parameter threshold. However, the running process of a real-time streaming job has characteristics such as suddenness and diversity. In the case of occasional network congestion, message processing failures do not have to be regarded as abnormal operations, because these messages can still be retransmitted for secondary processing later.

[0004] Therefore, the existing technology has low accuracy in judging abnormal real-time streaming jobs, and human intervention in real-time streaming jobs in the case of inaccurate judgment leads to low efficiency of the real-time streaming job operation process, reducing the degree of automation of the abnormal real-time streaming job determination process. Summary of the invention

[0005] The embodiments of the present application provide a method, apparatus, device and storage medium for determining abnormal operations to improve the accuracy and efficiency of determining abnormal operations.

[0006] In a first aspect, an embodiment of the present application provides a method for determining an abnormal operation, the method comprising:

[0007] Obtain the number of message processing failures and average message processing delay of real-time stream jobs within a preset time period;

[0008] According to the number of message processing failures and the average message processing delay, based on preset fuzzy logic reasoning rules, determine the probability of abnormal operation;

[0009] It is determined whether the abnormal operation probability is equal to or greater than a preset abnormal operation probability threshold, and if so, the real-time streaming operation is determined to be an abnormal operation.

[0010] In a second aspect, an embodiment of the present application further provides an abnormal operation determination device, the device comprising:

[0011] A data acquisition module, used to obtain the number of message processing failures and the average message processing delay of the real-time stream job within a preset time period;

[0012] An abnormal operation probability determination module, used to determine the abnormal operation probability based on the number of message processing failures and the average message processing delay and based on preset fuzzy logic inference rules;

[0013] The abnormal operation determination module is used to determine whether the abnormal operation probability is equal to or greater than a preset abnormal operation probability threshold, and if so, determine that the real-time streaming operation is an abnormal operation.

[0014] In a third aspect, an embodiment of the present application further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the abnormal operation determination method as described in any one of the embodiments of the present application is implemented.

[0015] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the abnormal operation determination method as described in any one of the embodiments of the present application is implemented.

[0016] The embodiment of the present application obtains the number of message processing failures and the average message processing delay of the real-time streaming job within a preset time period; determines the probability of abnormal jobs based on the preset fuzzy logic reasoning rules according to the number of message processing failures and the average message processing delay, thereby improving the accuracy and efficiency of abnormal job determination; determines whether the probability of abnormal jobs is equal to or greater than the preset abnormal job probability threshold, and if so, determines that the real-time streaming job is an abnormal job. The above scheme realizes the automated determination of abnormal jobs. For different types of real-time streaming jobs, the probability of abnormal jobs can be determined based on fuzzy logic reasoning rules, avoiding the setting of different empirical parameters for different types of real-time streaming jobs, thereby resulting in a low degree of automation in the abnormal real-time streaming job determination process, and improving the efficiency and accuracy of abnormal job determination. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a flowchart of a method for determining an abnormal operation in Embodiment 1 of the present application;

[0018] Figure 2 It is a flowchart of a method for determining abnormal operation in Embodiment 2 of the present application;

[0019] Figure 3 It is a flowchart of a method for determining an abnormal operation in Embodiment 3 of the present application;

[0020] Figure 4It is a structural block diagram of an abnormal operation determination device in Embodiment 4 of the present application;

[0021] Figure 5 It is a structural diagram of an electronic device in Example 5 of the present application. DETAILED DESCRIPTION

[0022] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0023] Embodiment 1

[0024] Figure 1 This is a flow chart of an abnormal operation determination method provided in Embodiment 1 of the present application. This embodiment is applicable to the case of abnormal monitoring of real-time streaming operations. The method can be executed by an abnormal operation determination device, which can be implemented in software and / or hardware, such as Figure 1 As shown, the method specifically comprises the following steps:

[0025] S110: Obtain the number of message processing failures and the average message processing delay of the real-time stream job within a preset time period.

[0026] Real-time streaming jobs can be processed by streaming data processing to process received job tasks. During the processing, data does not fall to the ground and is processed as it comes. For example, job tasks can be transfers or remittances. During the operation of real-time streaming jobs, failed processing messages are usually generated. Some of these failed processing messages are errors in the messages themselves and can be considered as real abnormal messages; some may be caused by some accidental factors, such as occasional network congestion; message processing failures caused by accidental factors do not need to be regarded as abnormal jobs, because these messages can still be retransmitted for secondary processing later.

[0027] The number of message processing failures may be the number of messages that failed to be processed during the operation of the real-time stream job within a preset time period; wherein the messages that failed to be processed include messages that failed to be processed due to accidental factors, as well as true abnormal messages. The preset time period may be set by relevant technical personnel according to actual needs, for example, the preset time period may be 1 minute. The average message processing delay may be the average delay corresponding to all messages processed within the preset time period, that is, the average delay for processing each message; for example, if the number of messages processed within the preset time period is 3, and the processing delays corresponding to the 3 messages are 100ms, 140ms, and 150ms, respectively, then the average message processing delay within the preset time period is 130ms.

[0028] Exemplarily, the number of message processing failures and the average message processing delay of the real-time stream job within the preset time period can be obtained based on the interface of the real-time stream computing framework used to process the real-time stream job; for example, the real-time stream computing framework can be a Streaming framework, or other real-time stream computing frameworks, which are not limited in this embodiment. Alternatively, it can also be obtained in a customized manner, for example, a data acquisition interface of a customized real-time stream job, which can be specifically written according to actual needs. The corresponding data acquisition interface code.

[0029] S120. Determine the probability of abnormal operation according to the number of message processing failures and the average message processing delay based on preset fuzzy logic reasoning rules.

[0030] The number of message processing failures and the average message processing delay can be used as input parameters, and the probability value of the abnormal operation probability can be output based on the preset fuzzy logic reasoning rules. The fuzzy logic reasoning rules can be preset by relevant technical personnel. For example, the fuzzy logic reasoning rules can be approximate reasoning rules, fuzzy conditional reasoning rules, multi-input fuzzy reasoning rules, or multi-input multi-rule reasoning rules.

[0031] Exemplarily, corresponding fuzzy logic inference rules can be set for the number of message processing failures and the average message processing delay. For example, the fuzzy logic inference rule corresponding to the number of message processing failures can be that if the number of message failures is in the range of 0 to 100, the probability of abnormal operation can be considered to be 30%; if the number of message failures is in the range of 100 to 200, the probability of abnormal operation can be considered to be 70%; if the number of message failures is above 200, the probability of abnormal operation can be considered to be 100%.

[0032] The fuzzy logic reasoning rule corresponding to the average message processing delay can be that if the average message processing delay is in the range of 0 to 150 milliseconds, the probability of abnormal operation can be considered to be 30%; if the average message processing delay is in the range of 150 to 250 milliseconds, the probability of abnormal operation can be considered to be 70%; if the average message processing delay is above 250 milliseconds, the probability of abnormal operation can be considered to be 100%.

[0033] Optionally, a probability average weighting may be performed according to the abnormal operation probabilities corresponding to the number of message processing failures and the average message processing delay, to obtain a final abnormal operation probability.

[0034] S130: Determine whether the abnormal operation probability is equal to or greater than a preset abnormal operation probability threshold; if so, determine that the real-time streaming operation is an abnormal operation.

[0035] Determine whether the abnormal operation probability is equal to or greater than a preset abnormal operation probability threshold. If so, determine that the real-time streaming operation is an abnormal operation; if not, determine that the real-time streaming operation is a normal operation. The abnormal operation probability threshold can be preset by relevant technical personnel, for example, the abnormal operation probability threshold can be 0.68.

[0036] The embodiment of the present application obtains the number of message processing failures and the average message processing delay of the real-time streaming job within a preset time period; determines the probability of abnormal jobs based on the preset fuzzy logic reasoning rules according to the number of message processing failures and the average message processing delay, thereby improving the accuracy and efficiency of abnormal job determination; determines whether the probability of abnormal jobs is equal to or greater than the preset abnormal job probability threshold, and if so, determines that the real-time streaming job is an abnormal job. The above scheme realizes the automated determination of abnormal jobs. For different types of real-time streaming jobs, the probability of abnormal jobs can be determined based on fuzzy logic reasoning rules, avoiding the setting of different empirical parameters for different types of real-time streaming jobs, thereby resulting in a low degree of automation in the abnormal real-time streaming job determination process, and improving the efficiency and accuracy of abnormal job determination.

[0037] Embodiment 2

[0038] Figure 2 A flowchart of a method for determining abnormal operations provided in Example 2 of the present application is provided. This embodiment is optimized and improved on the basis of the above-mentioned technical solutions.

[0039] Furthermore, the step of "determining the probability of abnormal operation based on the preset fuzzy logic reasoning rules according to the number of message processing failures and the average message processing delay" is refined into "determining the algorithm according to the number of message processing failures and the preset message failure probability, obtaining the first set element in the preset message failure number probability set; determining the algorithm according to the average message processing delay and the preset average delay high and low probability set, obtaining the second set element in the preset average delay high and low probability set; determining the probability of abnormal operation based on the preset abnormal probability determination rules according to the first set element and the second set element." to improve the method of determining the probability of abnormal operation.

[0040] like Figure 2 As shown, the method comprises the following specific steps:

[0041] S210: Obtain the number of message processing failures and the average message processing delay of the real-time stream job within a preset time period.

[0042] S220. Determine an algorithm according to the number of message processing failures and a preset message failure probability, and obtain a first set element in a preset message failure number probability set.

[0043] The message failure probability determination algorithm and the message failure number probability set can be preset by relevant technical personnel; the first set element can be at least one set element in the message failure number probability set. The message processing failure number can correspond to a fuzzy semantic set, and the fuzzy semantic set is preset by relevant technical personnel; for example, the fuzzy set corresponding to the message processing failure number can be {small failure number, medium failure number, large failure number}; the small failure number in the fuzzy semantic set can be understood as the number of message failures within the preset time period is small, the medium failure number in the set element can be understood as the number of message failures within the preset time period is moderate, and the large failure number in the set element can be understood as the number of message failures within the preset time period is large. According to the preset fuzzy semantic set corresponding to the number of message processing failures, determine the first set element in the message failure number probability set. For example, if the fuzzy set corresponding to the message processing failure number is {small failure number, medium failure number, large failure number}, the first set elements in the message failure number probability set are the probability when the number of message failures is small, the probability when the number of message failures is moderate, and the probability when the number of message failures is large.

[0044] It should be noted that a small number of failures, a medium number of failures and a large number of failures only represent vague concepts of the number of message processing failures. There is no need to set specific failure number thresholds to represent a small number of failures, a medium number of failures and a large number of failures. Instead, the probability value in the first set element in the subsequent message failure number probability set represents the probability of a small number of message failures, a moderate number of message failures and a large number of message failures within a preset time period.

[0045] In an optional embodiment, obtaining a first set element in a preset message failure number probability set according to the number of message processing failures and a preset message failure probability determination algorithm includes: determining the first set element in the message failure number probability set according to the following formula:

[0046]

[0047] Among them, μ SF (x), μ MF (x) and μ LF (x) represents the first set element in the preset message failure probability set, m represents the preset average message processing failure number, x represents the message processing failure number within a preset time period, and μ1 represents the message failure probability.

[0048] SF, MF and LF can represent the set elements in the fuzzy semantic set corresponding to the number of message processing failures, where SF represents the number of small failures, MF represents the number of medium failures, and LF represents the number of large failures. The fuzzy semantic set corresponding to the number of message failures is {SF (small number of failures), MF (medium number of failures), LF (large number of failures)}. Correspondingly, the probability set of message failures is {μSF (x), μ MF (x), μ LF The average number of message processing failures m can be set by relevant technical personnel based on experimental values ​​and experience values. For example, the average number of message processing failures m can be 100.

[0049] For example, if m is preset to 100 and the number of message processing failures x within a preset time period is 60, then according to the calculation formula μ1 for the probability of message failures, μ can be determined. SF (x) = 2 / 5, μ MF (x) = 3 / 5, μ LF (x) = 0, then the message failure probability set is {2 / 5, 3 / 5, 0}.

[0050] This optional embodiment achieves accurate determination of the first set element in the message failure number probability set within a preset time period by presetting the message failure number probability set and based on a preset message failure probability determination algorithm.

[0051] S230. Obtain a second set element in a preset average delay high and low probability set according to the average message processing delay and a preset average delay high and low determination algorithm.

[0052] The average delay high and low determination algorithm and the average delay high and low probability set can be preset by relevant technical personnel; the second set element can be at least one set element in the average delay high and low probability set. The average delay of message processing can correspond to a fuzzy semantic set, and the fuzzy semantic set is preset by relevant technical personnel; for example, the fuzzy semantic set corresponding to the average delay of message processing can be {low delay, medium delay, high delay}; the low delay in the fuzzy semantic set can be understood as the average message processing delay within the preset time period is low, the number of medium failures in the set element can be understood as the average message processing delay within the preset time period is moderate, and the number of multiple failures in the set element can be understood as the average message processing delay within the preset time period is high. According to the fuzzy semantic set corresponding to the preset average delay of message processing, determine the second set element in the average delay high and low probability set. For example, if the fuzzy semantic set corresponding to the average delay of message processing is {low delay, medium delay, high delay}, the second set elements in the average delay high and low probability set are the probability when the average delay is low, the probability when the average delay is moderate, and the probability when the average delay is high.

[0053] It should be noted that low latency, medium latency and high latency only represent vague concepts of the average latency of message processing. There is no need to set a specific average latency threshold to represent low latency, medium latency and high latency. Instead, the probability values ​​in the second set elements in the subsequent average delay high and low probability sets represent the probability of a low average latency for message processing within a preset time period, the probability of a moderate average latency for message processing and the probability of a high average latency for message processing.

[0054] In an optional embodiment, obtaining a second set element in a preset average delay high and low probability set according to the message processing average delay and a preset average delay high and low determination algorithm includes: determining the second set element in the average delay high and low probability set according to the following formula:

[0055]

[0056] Among them, μ SD (y), μ MD (y) and μ HF (y) represents the second set element in the preset average delay high and low probability set, e represents the preset average message processing delay, y represents the average message processing delay within the preset time period, and μ2 represents the average delay high and low probability.

[0057] Among them, LD, MD and HD can respectively represent the set elements in the fuzzy semantic set corresponding to the average delay of message processing, among which LD represents low delay, MD represents medium delay, HD represents high delay, and the fuzzy semantic set corresponding to the high and low probability set of average delay is {LD (low delay), MD (medium delay), HD (high delay)}, and correspondingly, the high and low probability set of average delay is {μ SD (y), μ MD (y) and μ HF The average message processing delay e can be set by relevant technical personnel based on experimental values ​​and experience values. For example, the average message processing delay e can be 150 ms.

[0058] For example, if e is preset to 150ms and the average message processing delay y within the preset time period is 200ms, then μ can be determined according to the calculation formula μ2 for the high and low probability of the average delay. SD (x) = 0, μ MD (x) = 2 / 3, μ HD (x) = 1 / 3, then the probability set of message failures is {0, 2 / 3, 1 / 3}.

[0059] This optional embodiment achieves accurate determination of the second set element in the average delay high and low probability set within a preset time period by presetting the average delay high and low probability set and based on a preset average delay high and low determination algorithm.

[0060] S240. Determine the abnormal operation probability according to the first set elements and the second set elements based on a preset abnormal probability determination rule.

[0061] The abnormal probability determination rules can be pre-set by relevant technical personnel. For example, a set of abnormal operation fuzzy probabilities can be pre-set, and set elements in the set of abnormal operation fuzzy probabilities can be determined; and the abnormal operation probability can be determined according to the set elements in the set of abnormal operation fuzzy probabilities.

[0062] In an optional embodiment, the abnormal operation probability is determined based on the first set element and the second set element based on a preset abnormal probability determination rule, including: determining the third set element in the preset abnormal operation fuzzy probability set based on the first set element and the second set element based on a preset abnormal operation fuzzy probability determination algorithm; determining the abnormal operation probability of the real-time stream operation based on the preset abnormal operation probability determination algorithm.

[0063] The abnormal operation fuzzy probability determination algorithm and the abnormal operation fuzzy probability set can be preset by relevant technical personnel. The third set element can be at least one set element in the abnormal operation fuzzy probability set. The abnormal operation probability can correspond to a fuzzy semantic set, and the fuzzy semantic set is preset by relevant technical personnel; for example, the fuzzy semantic set corresponding to the abnormal operation probability can be {very small probability, small probability, moderate probability, large probability, very large probability}. According to the fuzzy semantic set corresponding to the preset abnormal operation probability, the third set element in the abnormal operation fuzzy probability set is determined. For example, if the fuzzy semantic set corresponding to the abnormal operation probability is {very small probability, moderate probability, very large probability}, the third set elements in the abnormal operation fuzzy probability set are respectively very small probability of abnormal operation, moderate probability of abnormal operation and very large probability of abnormal operation.

[0064] It should be noted that very small probability, moderate probability and very large probability only represent fuzzy concepts of the probability of abnormal operations. There is no need to set specific abnormal operation probability thresholds to represent very small probability, moderate probability and very large probability. Instead, the probability that the real-time stream operation is an abnormal operation with a very small probability, the probability that the real-time stream operation is an abnormal operation with a moderate probability and the probability that the real-time stream operation is an abnormal operation with a very large probability are represented according to the probability value in the third set element in the subsequent abnormal operation fuzzy probability set.

[0065] The abnormal operation fuzzy probability determination algorithm is used to determine the value of the third set element in the abnormal operation fuzzy probability set, and can be determined by comparing the value of the first set element with the value of the second set element.

[0066] In an optional embodiment, according to the first set elements and the second set elements, based on a preset abnormal operation fuzzy probability determination algorithm, a third set element in a preset abnormal operation fuzzy probability set is determined, including: multiplying the number of first set elements by the number of second set elements to obtain the number of third set elements; comparing the values ​​of any first set element with any second set element to determine a target value that meets the preset size comparison rules, and determining the target value as the value of the third set element.

[0067] The number of elements in the third set is related to the number of elements in the first set and the number of elements in the second set. Specifically, the number of elements in the first set is multiplied by the number of elements in the second set to obtain the number of elements in the third set. For example, if the number of elements in the first set is 3 and the number of elements in the second set is 3, then the number of elements in the third set is 9. The correlation between the probability of message failure, the probability of high and low average delay, and the probability of abnormal operation can be predetermined by relevant technical personnel.

[0068] Exemplarily, the correlation between the probability of message failure, the high and low probabilities of average delay and the probability of abnormal operation is shown in Table 2-1, where the fuzzy semantic set corresponding to the number of message processing failures is {SF (few failures), MF (medium failures), LF (many failures)}; the fuzzy semantic set corresponding to the average delay of message processing is {LD (low delay), MD (medium delay), HD (high delay)}, and the fuzzy set corresponding to the probability of abnormal operation is {VS (very small probability), S (small probability), M (moderate probability), L (large probability), VL (very large probability)}.

[0069] Table 2-1

[0070]

[0071]

[0072] If the first set element corresponding to the message failure probability set is {0, 3 / 5, 2 / 5}, and the second set element corresponding to the average delay high and low probability set is {0, 1 / 3, 2 / 3}, then compare the values ​​of any first set element with any second set element, determine the target value that meets the preset size comparison rule, and determine the target value as the value of the third set element. The preset size comparison rule can be the minimum value after comparing the values ​​of any first set element with any second set element as the target value. According to the preset size comparison rule, the correlation between the message failure probability, the average delay high and low probability, and the abnormal operation probability is determined as shown in Table 2-2.

[0073] Table 2-2

[0074]

[0075] According to the correlation between the probability of message failure, the probability of high and low average delay, and the probability of abnormal operation shown in Table 2-2, the third set element {μ M (z) = 1 / 3, μ L (z) = 3 / 5, μ VL (z) = 1 / 3, μ VL (z) = 3 / 5}. If at least one of the probability values ​​of the average delay high and low probability and the message failure number probability is 0, the probability of abnormal operation is 0, and there is no need to participate in the subsequent calculation of determining the intermediate parameters z and Z0.

[0076] Exemplarily, the abnormal operation probability can be determined based on the third set element, for example, the element with the largest value in the third set can be selected as the abnormal operation probability. It should be noted that in order to improve the accuracy of the determination of the abnormal operation probability, the abnormal operation probability of the real-time stream operation can also be determined based on a preset abnormal operation probability determination algorithm.

[0077] In an optional embodiment, according to the third set element, based on a preset abnormal operation probability determination algorithm, determining the abnormal operation probability of the real-time stream job includes: determining the abnormal operation probability of the real-time stream job according to the following formula:

[0078]

[0079]

[0080] Among them, μ VS (z), μ s (z), μ M (z), μ L (z) and μ VL (z) represents the third set element in the preset abnormal operation fuzzy probability set, z represents the intermediate parameter, μ3 represents the abnormal operation probability, and Z0 represents the clear value of the abnormal operation probability.

[0081] Exemplarily, the obtained probability value of the abnormal operation probability can be brought into the abnormal operation probability determination algorithm of the real-time streaming operation to obtain the intermediate parameter z; according to the intermediate parameter z and the abnormal operation probability μ3, the abnormal operation probability clarity value Z0 is determined. Using the clarity value Z0 of the abnormal operation probability to evaluate whether the operation is an abnormal operation can improve the accuracy of abnormal operation determination. The clarity threshold of the clarity value of the abnormal operation probability can be set in advance, for example, the clarity threshold can be 0.5. Determine whether the clarity value of the abnormal operation probability is greater than or equal to the preset clarity threshold. If so, the real-time streaming operation can be considered to be an abnormal operation; if not, the real-time streaming operation can be considered to be a normal operation.

[0082] For example, if the third set element {μ M (z) = 1 / 3, μ L (z) = 3 / 5, μ VL (z) = 1 / 3, μ VL (z)=3 / 5}. Then μ M (z) = 1 / 3, μ L (z) = 3 / 5, μ VL (z) = 1 / 3, μ VL (z) = 3 / 5 is brought into the abnormal operation probability determination algorithm, and μ is calculated respectively. M (z) = 1 / 3, μ L (z) = 3 / 5, μ VL (z) = 1 / 3, μ VL The value of z when (z) = 3 / 5. For example, μ M When (z) = 1 / 3, the corresponding value of z is z1, μ L When (z) = 3 / 5, the corresponding value of z is z2, μ VL When (z) = 1 / 3, the corresponding value of z is z3, μ VL When (z) = 3 / 5, the value of z is Z4. M (z), z2 and μ corresponding to z2 L (z), z3 and μ corresponding to z3 VL (z), and μ corresponding to z4 and z4 VL (z) is substituted into Z0, and the calculated Z0 is used as the abnormal operation probability clarity value. According to the abnormal operation probability clarity value and the clarity value threshold, it is determined whether the real-time stream operation is an abnormal operation.

[0083] This optional embodiment determines the abnormal operation probability of the real-time streaming operation based on a preset abnormal operation probability determination algorithm, thereby improving the accuracy of abnormal operation probability determination and thus improving the accuracy of abnormal operation determination.

[0084] S250: Determine whether the abnormal operation probability is equal to or greater than a preset abnormal operation probability threshold; if so, determine that the real-time streaming operation is an abnormal operation.

[0085] The scheme of this embodiment obtains the first set element in the preset message failure number probability set according to the number of message processing failures and the preset message failure probability determination algorithm, thereby realizing the determination of the first set element, and determining the first set element by the preset message failure probability determination algorithm, thereby realizing the accuracy of the value determination of the first set element. According to the average message processing delay and the preset average delay high and low determination algorithm, the second set element in the preset average delay high and low probability set is obtained; the determination of the second set element is realized, and the second set element is determined by the preset average delay high and low determination algorithm, thereby realizing the accuracy of the value determination of the second set element. According to the first set element and the second set element, based on the preset abnormal probability determination rule, the probability of abnormal operation is determined, thereby improving the accuracy of the determination of abnormal operation probability, thereby improving the accuracy of abnormal operation determination. The scheme of this embodiment realizes the automatic determination of abnormal operation. For different types of real-time stream operations, the probability of abnormal operation can be determined based on fuzzy logic reasoning rules, thereby avoiding the situation that different empirical parameters are set for different types of operations, thereby causing the low degree of automation in the abnormal real-time stream operation determination process, thereby improving the efficiency and accuracy of abnormal operation determination.

[0086] Embodiment 3

[0087] Figure 3 It is a schematic diagram of the detection process of an abnormal operation determination method provided in Example 3 of the present application. Based on the technical solutions of the above-mentioned embodiments, this embodiment of the present application provides a preferred implementation method.

[0088] This embodiment uses fuzzy logic reasoning to avoid direct judgment of large-scale online real-time streaming job parameters, thereby realizing a more general method for determining abnormal jobs. By analyzing abnormal real-time streaming jobs, it can be found that: 1. The amount of message exception processing gradually increases from a certain moment; 2. The average message processing delay continues to increase. This embodiment uses the number of message failures and the average message processing delay to construct a fuzzy system, and then infers the possibility of abnormality of the current real-time streaming job. This embodiment constructs a fuzzy system according to the following process:

[0089] S310: Determine input quantity and output quantity.

[0090] The number of message processing failures x of the real-time stream job in the last t time is defined as the input quantity, and its fuzzy semantic set is expressed as: message failure number = {SF (small number of failures), MF (medium number of failures), LF (large number of failures)}; at the same time, the average message processing delay y of the real-time stream job in the last t time is also defined as the input quantity, and its fuzzy semantic set is expressed as: average message processing delay = {LD (low delay), MD (medium delay), HD (high delay)}; accordingly, the probability that the real-time stream job is an abnormal job is taken as the output quantity, and its fuzzy semantic set is expressed as: abnormal probability = {VS (very small), S (small), M (medium), L (large), very large (VL)}.

[0091] S320, selecting a membership function.

[0092] According to the average number of message failures m (e.g., 100) and the average processing delay d (e.g., 150 milliseconds) of abnormal real-time flow operations, this embodiment uses the following membership function:

[0093] 1. Membership function of message failure number:

[0094]

[0095] 2. Membership function of average message processing delay:

[0096]

[0097] 3. Abnormal probability membership function:

[0098]

[0099] S330. Develop a fuzzy reasoning rule base.

[0100] The fuzzy reasoning rule base developed in this embodiment is shown in Table 2-1.

[0101] After comparing the value of any first set element corresponding to the average number of message processing failures with the value of any second set element corresponding to the average message delay, the minimum value obtained by comparison is used as the abnormal operation probability.

[0102] S340, defuzzification.

[0103] The corresponding z value can be determined by determining the value of μ3. This embodiment uses the centroid method to defuzzify the reasoning, and the weighted average value of μ3 of all triggered reasoning rules is the clear value of z, that is:

[0104]

[0105] By comparing Z0 with a preset abnormal job threshold θ (eg, set to: 0.68), it is determined whether the real-time streaming job is an abnormal job.

[0106] Embodiment 4

[0107] Figure 4 This is a schematic diagram of the structure of an abnormal operation determination device provided in the fourth embodiment of the present application. The abnormal operation determination device provided in the embodiment of the present application can be applied to the situation of abnormal monitoring of real-time streaming operations, and the device can be implemented in software and / or hardware. Figure 4 As shown, the device specifically includes: a data acquisition module 401, an abnormal operation probability determination module 402 and an abnormal operation determination module 403. Among them,

[0108] The data acquisition module 401 is used to acquire the number of message processing failures and the average message processing delay of the real-time stream job within a preset time period;

[0109] The abnormal operation probability determination module 402 is used to determine the abnormal operation probability based on the number of message processing failures and the average message processing delay and based on preset fuzzy logic reasoning rules;

[0110] The abnormal operation determination module 403 is used to determine whether the abnormal operation probability is equal to or greater than a preset abnormal operation probability threshold, and if so, determine that the real-time streaming operation is an abnormal operation.

[0111] The embodiment of the present application obtains the number of message processing failures and the average message processing delay of the real-time streaming job within a preset time period; determines the probability of abnormal jobs based on the preset fuzzy logic reasoning rules according to the number of message processing failures and the average message processing delay, thereby improving the accuracy and efficiency of abnormal job determination; determines whether the probability of abnormal jobs is equal to or greater than the preset abnormal job probability threshold, and if so, determines that the real-time streaming job is an abnormal job. The above scheme realizes the automated determination of abnormal jobs. For different types of real-time streaming jobs, the probability of abnormal jobs can be determined based on fuzzy logic reasoning rules, avoiding the setting of different empirical parameters for different types of real-time streaming jobs, thereby resulting in a low degree of automation in the abnormal real-time streaming job determination process, and improving the efficiency and accuracy of abnormal job determination.

[0112] Optionally, the abnormal operation probability determination module 402 includes:

[0113] A first set element determination unit, configured to obtain a first set element in a preset message failure number probability set according to the number of message processing failures and a preset message failure probability determination algorithm;

[0114] A second set element determination unit, configured to obtain a second set element in a preset average delay high and low probability set according to the message processing average delay and a preset average delay high and low determination algorithm;

[0115] The abnormal operation probability determination unit is used to determine the abnormal operation probability according to the first set elements and the second set elements based on a preset abnormal probability determination rule.

[0116] Optionally, the first set element determining unit includes:

[0117] The first set element determination subunit is used to determine the first set element in the message failure probability set according to the following formula:

[0118]

[0119] Among them, μ SF (x), μ MF (x) and μ LF (x) represents the first set element in the preset message failure probability set, m represents the preset average message processing failure number, x represents the message processing failure number within a preset time period, and μ1 represents the message failure probability.

[0120] Optionally, the second set element determination unit includes:

[0121] The second set element determination subunit determines the second set element in the average delay high and low probability set according to the following formula:

[0122]

[0123] Among them, μ SD (y), μ MD (y) and μ HF (y) represents the second set element in the preset average delay high and low probability set, e represents the preset average message processing delay, y represents the average message processing delay within the preset time period, and μ2 represents the average delay high and low probability.

[0124] Optionally, the abnormal operation probability determination unit includes:

[0125] A third set element determination subunit is used to determine a third set element in a preset abnormal operation fuzzy probability set according to the first set element and the second set element and based on a preset abnormal operation fuzzy probability determination algorithm;

[0126] The abnormal operation probability determination subunit is used to determine the abnormal operation probability of the real-time stream operation according to the third set element and based on a preset abnormal operation probability determination algorithm.

[0127] Optionally, the third set element determines the subunit, specifically for:

[0128] Multiplying the number of elements in the first set by the number of elements in the second set to obtain the number of elements in the third set;

[0129] The values ​​of any first set element and any second set element are compared to determine a target value that meets a preset size comparison rule, and the target value is determined as the value of the third set element.

[0130] Optionally, the third set element determines the subunit, specifically for:

[0131] The abnormal operation probability of the real-time stream operation is determined according to the following formula:

[0132]

[0133]

[0134] Among them, μ VS (z), μ s (z), μ M (z), μ L (z) and μ VL (z) represents the third set element in the preset abnormal operation fuzzy probability set, z represents the intermediate parameter, μ3 represents the abnormal operation probability, and Z0 represents the clear value of the abnormal operation probability.

[0135] The above-mentioned abnormal operation determination device can execute the abnormal operation determination method provided by any embodiment of the present application, and has corresponding functional modules and beneficial effects for executing each abnormal operation determination method.

[0136] Embodiment 5

[0137] Figure 5 It is a structural schematic diagram of an electronic device provided in Example 5 of the present application. Figure 5 A block diagram of an exemplary electronic device 500 suitable for implementing the embodiments of the present application is shown. Figure 5 The electronic device 500 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0138] like Figure 5 As shown, the electronic device 500 is in the form of a general computing device. The components of the electronic device 500 may include but are not limited to: one or more processors or processing units 501, a system memory 502, and a bus 503 connecting different system components (including the system memory 502 and the processing unit 501).

[0139] Bus 503 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus structures. For example, these architectures include but are not limited to Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus and Peripheral Component Interconnect (PCI) bus.

[0140] The electronic device 500 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device 500, including volatile and non-volatile media, removable and non-removable media.

[0141] System memory 502 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 504 and / or cache memory 505. Electronic device 500 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 506 may be used to read and write non-removable, non-volatile magnetic media ( Figure 5 not shown, usually called a "hard drive"). Although Figure 5 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical medium) may be provided. In these cases, each drive may be connected to bus 503 via one or more data medium interfaces. Memory 502 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of each embodiment of the present application.

[0142] A program / utility 508 having a set (at least one) of program modules 507 may be stored, for example, in the memory 502, such program modules 507 including but not limited to an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules 507 generally perform the functions and / or methods of the embodiments described herein.

[0143] The electronic device 500 may also communicate with one or more external devices 509 (e.g., keyboards, pointing devices, displays 510, etc.), may communicate with one or more devices that enable a user to interact with the electronic device 500, and / or communicate with any device that enables the electronic device 500 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication may be performed through an input / output (I / O) interface 511. Furthermore, the electronic device 500 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 512. As shown, the network adapter 512 communicates with other modules of the electronic device 500 through the bus 503. It should be understood that although Figure 5 Not shown, other hardware and / or software modules may be used in conjunction with electronic device 500, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0144] The processing unit 501 executes various functional applications and data processing by running the programs stored in the system memory 502, for example, implementing a method for determining an abnormal operation provided in an embodiment of the present application.

[0145] Embodiment 6

[0146] Embodiment 6 of the present application also provides a storage medium containing computer executable instructions, on which a computer program is stored. When the program is executed by a processor, the abnormal job determination method provided in the embodiment of the present application is implemented, including: obtaining the number of message processing failures and the average message processing delay of the real-time stream job within a preset time period; determining the abnormal job probability based on the number of message processing failures and the average message processing delay and based on preset fuzzy logic reasoning rules; judging whether the abnormal job probability is equal to or greater than a preset abnormal job probability threshold, and if so, determining that the real-time stream job is an abnormal job.

[0147] The computer storage medium of the embodiment of the present application can adopt any combination of one or more computer-readable media. Computer-readable media can be computer-readable signal media or computer-readable storage media. Computer-readable storage media can be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any combination of the above. More specific examples (non-exhaustive lists) of computer-readable storage media include: electrical connections with one or more wires, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In this document, computer-readable storage media can be any tangible medium containing or storing programs, which can be used by instruction execution systems, devices or devices or used in combination with them.

[0148] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0149] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0150] Computer program code for performing the operation of the present application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., using an Internet service provider to connect through the Internet).

[0151] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A method for determining abnormal operation, characterized in that: include: Obtain the number of message processing failures and average message processing delay of real-time stream jobs within a preset time period; According to the number of message processing failures and the average message processing delay, based on preset fuzzy logic reasoning rules, determine the probability of abnormal operation; It is determined whether the abnormal operation probability is equal to or greater than a preset abnormal operation probability threshold, and if so, the real-time streaming operation is determined to be an abnormal operation.

2. The method according to claim 1, characterized in that The determining of the abnormal operation probability according to the number of message processing failures and the average message processing delay based on preset fuzzy logic reasoning rules includes: According to the number of message processing failures and a preset message failure probability determination algorithm, a first set element in a preset message failure number probability set is obtained; According to the message processing average delay and a preset average delay high and low determination algorithm, obtain a second set element in a preset average delay high and low probability set; According to the first set elements and the second set elements, based on a preset abnormal probability determination rule, an abnormal operation probability is determined.

3. The method according to claim 2, characterized in that According to the message processing failure number and the preset message failure probability determination algorithm, a first set element in the preset message failure number probability set is obtained, including: The first set element in the message failure probability set is determined according to the following formula: Among them, SF, MF and LF represent the set elements in the fuzzy semantic set corresponding to the number of message processing failures, SF represents the number of small failures, MF represents the number of medium failures, LF represents the number of large failures, μ SF (x), μ MF (x) and μ LF (x) represents the first set element in the preset message failure probability set, μ SF (x) represents the probability when the number of message failures is small, μ MF (x) represents the probability when the number of message failures is moderate, μ LF (x) represents the probability when the number of message failures is large, m represents the preset average number of message processing failures, x represents the number of message processing failures within a preset time period, and μ1 represents the probability of message failures.

4. The method according to claim 2, characterized in that: According to the message processing average delay and the preset average delay high and low determination algorithm, a second set element in the preset average delay high and low probability set is obtained, including: The second set element in the average delay high and low probability set is determined according to the following formula: Among them, LD, MD and HD represent the set elements in the fuzzy semantic set corresponding to the average message processing delay, LD represents low delay, MD represents medium delay, HD represents high delay, μ SD (y), μ mD (y) and μ HF (y) represents the second set element in the preset average delay high and low probability set, μ SD (y) represents the probability when the average delay is low, μ MD (y) represents the probability when the average delay is moderate, μ HD (y) represents the probability when the average delay is high, e represents the preset average message processing delay, y represents the average message processing delay within the preset time period, and μ2 represents the probability of high and low average delay.

5. The method according to claim 2, characterized in that: Determining the abnormal operation probability according to the first set elements and the second set elements based on a preset abnormal probability determination rule includes: According to the first set element and the second set element, based on a preset abnormal operation fuzzy probability determination algorithm, determine a third set element in a preset abnormal operation fuzzy probability set; According to the third set element, based on a preset abnormal job probability determination algorithm, the abnormal job probability of the real-time stream job is determined.

6. The method according to claim 5, characterized in that According to the first set element and the second set element, based on a preset abnormal operation fuzzy probability determination algorithm, a third set element in a preset abnormal operation fuzzy probability set is determined, including: Multiplying the number of elements in the first set by the number of elements in the second set to obtain the number of elements in the third set; The values ​​of any first set element and any second set element are compared to determine a target value that meets a preset size comparison rule, and the target value is determined as the value of the third set element.

7. The method according to claim 5, characterized in that According to the third set element, based on a preset abnormal job probability determination algorithm, determining the abnormal job probability of the real-time stream job includes: The abnormal operation probability of the real-time stream operation is determined according to the following formula: Among them, VS, S, M, L and VL represent the set elements in the fuzzy semantic set corresponding to the probability of abnormal operation, VS represents a very small probability, S represents a small probability, M represents a moderate probability, L represents a large probability, VL represents a very large probability, μ VS (z), μ s (z), μ M (z), μ L (z) and μ VL (z) represents the third set element in the preset abnormal operation fuzzy probability set, μ VS (z) indicates that the probability of abnormal operation is very small, μ s (z) indicates that the probability of abnormal operation is small, μ M (z) indicates that the probability of abnormal operation is moderate, μ L (z) indicates that the probability of abnormal operation is high, μ VL (z) indicates that the probability of abnormal operation is very high, z indicates the intermediate parameter, μ3 indicates the probability of abnormal operation, and Z0 indicates the clear value of the probability of abnormal operation.

8. An abnormal operation determination device, characterized in that: include: A data acquisition module, used to obtain the number of message processing failures and the average message processing delay of the real-time stream job within a preset time period; An abnormal operation probability determination module, used to determine the abnormal operation probability based on the number of message processing failures and the average message processing delay and based on preset fuzzy logic inference rules; The abnormal operation determination module is used to determine whether the abnormal operation probability is equal to or greater than a preset abnormal operation probability threshold, and if so, determine that the real-time streaming operation is an abnormal operation.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the abnormal operation determination method as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the abnormal operation determination method as described in any one of claims 1 to 7 is implemented.

Images