Stream data monitoring method and device, terminal equipment and storage medium

Abstract

The application discloses a kind of flow data monitoring method, device, terminal equipment and storage medium, belong to computer monitoring technical field, the method is by obtaining the first historical data output by data source, and the first slope of first historical data is calculated;The first loss value between the first slope and each preset early warning interval is calculated;The minimum interval in each second loss value each corresponding preset early warning interval is used as target early warning interval, wherein the second loss value is the value less than preset threshold in each the first loss value;Real-time flow data output by the data source is monitored based on target early warning interval.The application is realized by using the above technical scheme, and the early warning rule parameter of automatic configuration flow data is improved the intelligence of data monitoring.

Description

Technical Field

[0001] This invention belongs to the field of computer monitoring technology, and particularly relates to a streaming data monitoring method, device, terminal equipment and storage medium. Background Technology

[0002] With the rapid development of technology, the application scenarios of Kafka streaming data as a data source are constantly increasing. Since the integrity and real-time performance of data are directly related to the system's business support capabilities, the demand for monitoring Topic data in various application scenarios is also constantly increasing.

[0003] Currently, the same topic experiences traffic peaks and troughs on the same day, and the data traffic of the same topic at the same time period shows significant differences in trend across different dates. For example, the data traffic of the same topic at the same time period shows a significant difference in trend between weekdays and weekends, and stable business growth also leads to these differences. However, traditional topic data monitoring methods cannot provide intelligent alerts based on the data characteristics of different topics due to the diverse business scenarios of different topics. This requires users to perform a lot of data analysis work in advance and to manually configure alert conditions and parameters, leading to difficulties in monitoring.

[0004] In summary, how to automatically configure the early warning rule parameters of streaming data to improve the intelligence of data monitoring has become an urgent technical problem to be solved in the field of computer monitoring technology. Summary of the Invention

[0005] The main objective of this invention is to provide a method, apparatus, terminal device, and storage medium for monitoring streaming data. The aim is to automatically configure early warning rule parameters for streaming data to improve the intelligence of data monitoring.

[0006] To achieve the above objectives, the present invention provides a streaming data monitoring method, the streaming data monitoring method comprising:

[0007] Obtain the first historical data output from the data source and calculate the first slope of the first historical data;

[0008] Calculate the first loss value between the first slope and each preset warning interval;

[0009] The minimum interval among the preset warning intervals corresponding to each second loss value is taken as the target warning interval, wherein the second loss value is the value of each first loss value that is less than a preset threshold.

[0010] The real-time streaming data output by the data source is monitored based on the target early warning interval.

[0011] Optionally, the method further includes:

[0012] Obtain multiple sets of second historical data output from the data source, and calculate the second slope of each set of second historical data;

[0013] Determine the data distribution type for each of the second slopes, and calculate the first distribution mean and first distribution standard deviation for each of the second slopes based on the data distribution type;

[0014] Each first interval is calculated based on the mean of the first distribution, the standard deviation of the first distribution, and the first preset formula;

[0015] Each of the first intervals is processed based on a preset smoothing period to obtain each of the warning intervals.

[0016] Optionally, the step of calculating the first distribution mean and first distribution standard deviation of each second slope based on the data distribution type includes:

[0017] When the data distribution type is a standard normal distribution, each of the second slopes is substituted into the second preset formula to obtain the first distribution mean and the first distribution standard deviation of each of the second slopes;

[0018] When the data distribution type is a log-normal distribution, the logarithm of each of the second slopes is substituted into the second preset formula to obtain the mean and standard deviation of the first distribution;

[0019] When the data distribution type is a mixed normal distribution, each of the second slopes is input into the expectation maximization algorithm to obtain the mean and standard deviation of the first distribution.

[0020] Optionally, the step of monitoring the real-time streaming data output by the data source based on the target early warning interval includes:

[0021] The real-time streaming data output from the data source is preprocessed to obtain the target curve;

[0022] Detect whether the slope point on the target curve is within the target warning interval;

[0023] If the slope point is detected to be outside the target warning range, a warning message will be output.

[0024] Optionally, the step of preprocessing the real-time streaming data output from the data source to obtain the target curve includes:

[0025] The real-time streaming data output from the data source is grouped according to a preset duration to obtain each data group;

[0026] Calculate the third slope corresponding to each of the data groups, and fit an initial curve formed by the third slopes;

[0027] The initial curve is smoothed based on the number of smoothing periods to obtain the target curve.

[0028] Optionally, after the step of taking the smallest interval among the preset warning intervals corresponding to each second loss value as the target warning interval, the method further includes:

[0029] Detect whether the topic of the real-time streaming data output by the data source is a newly added topic;

[0030] If it is detected that the topic of the real-time streaming data is not the newly added topic, then the step of monitoring the real-time streaming data output by the data source based on the target warning interval is executed;

[0031] If the topic of the real-time streaming data is detected to be the newly added topic, a new warning interval is determined based on multiple sets of third historical data under the newly added topic to monitor the real-time streaming data.

[0032] Optionally, the step of determining a new early warning interval based on multiple sets of third historical data under the new topic to monitor the real-time streaming data includes:

[0033] Calculate the fourth slope corresponding to each of the multiple sets of third historical data under the newly added topic;

[0034] Each second interval is calculated based on the second distribution mean and second distribution standard deviation of each of the fourth slopes;

[0035] The largest interval in each of the second intervals is taken as the new warning interval, and the real-time streaming data is monitored based on the new warning interval.

[0036] Furthermore, to achieve the above objectives, the present invention also provides a streaming data monitoring device, the streaming data monitoring device comprising:

[0037] The first calculation module is used to obtain the first historical data output by the data source and calculate the first slope of the first historical data.

[0038] The second calculation module is used to calculate the first loss value between the first slope and each preset warning interval;

[0039] The interval determination module is used to take the smallest interval among the preset warning intervals corresponding to each second loss value as the target warning interval, wherein the second loss value is the value of each first loss value that is less than a preset threshold.

[0040] The monitoring module is used to monitor the real-time streaming data output by the data source based on the target early warning interval.

[0041] In addition, to achieve the above objectives, the present invention also provides a terminal device, the terminal device comprising: a memory, a processor, and a streaming data monitoring program stored in the memory and executable on the processor, wherein the streaming data monitoring program of the terminal device, when executed by the processor, implements the steps of the streaming data monitoring method as described above.

[0042] In addition, to achieve the above objectives, the present invention also provides a computer-readable storage medium storing a streaming data monitoring program, which, when executed by a processor, implements the steps of the streaming data monitoring method as described above.

[0043] This invention provides a streaming data monitoring method, apparatus, terminal device, and storage medium. The method includes: acquiring first historical data output from a data source and calculating a first slope of the first historical data; calculating a first loss value between the first slope and each preset warning interval; taking the smallest interval among the preset warning intervals corresponding to each second loss value as a target warning interval, wherein the second loss value is a value less than a preset threshold among each first loss value; and monitoring the real-time streaming data output from the data source based on the target warning interval.

[0044] This invention improves the intelligence of data monitoring by acquiring historical data output from the current data source (i.e., the aforementioned first historical data), calculating the slope of the first historical data (i.e., the aforementioned first slope), calculating the loss value between the first slope and multiple preset warning intervals (i.e., the aforementioned first loss value), taking the value less than a preset threshold among the multiple first loss values ​​as the second loss value, and taking the smallest interval among the preset warning intervals corresponding to each second loss value as the target warning interval. Finally, the real-time streaming data output from the data source is monitored based on the target warning interval. Thus, compared to the traditional method that requires manual configuration of warning rule parameters, this invention automatically acquires historical data output from the current data source and automatically generates warning rule parameters for monitoring real-time streaming data, i.e., the aforementioned target warning interval, based on the slope of the historical data and the loss value between multiple preset warning intervals, thereby improving the intelligence of data monitoring. Attached Figure Description

[0045] Figure 1 This is a schematic diagram of the device structure of the terminal device hardware operating environment involved in the embodiments of the present invention;

[0046] Figure 2 This is a flowchart illustrating the steps of the first embodiment of the streaming data monitoring method of the present invention;

[0047] Figure 3 This is a schematic diagram illustrating the daily data growth in one embodiment of the streaming data monitoring method of the present invention;

[0048] Figure 4 This is a schematic diagram of the streaming data monitoring process according to an embodiment of the streaming data monitoring method of the present invention;

[0049] Figure 5 This is a schematic diagram of the early warning interval involved in an embodiment of the streaming data monitoring method of the present invention;

[0050] Figure 6 This is a schematic diagram illustrating the smoothing period involved in an embodiment of the streaming data monitoring method of the present invention;

[0051] Figure 7 This is a schematic diagram of the functional modules of an embodiment of the streaming data monitoring device of the present invention.

[0052] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0053] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0054] Reference Figure 1 , Figure 1 This is a schematic diagram of the hardware operating environment of the terminal device involved in the embodiment of the present invention.

[0055] The terminal device in this embodiment of the invention can be a terminal device applied in the field of computer monitoring technology. Specifically, the terminal device can be a smartphone, PC (Personal Computer), tablet computer, portable computer, etc.

[0056] like Figure 1 As shown, the terminal device may include: a processor 1001, such as a CPU; a communication bus 1002; a user interface 1003; a network interface 1004; and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or non-volatile memory, such as a disk drive. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0057] Those skilled in the art will understand that Figure 1 The terminal device structure shown does not constitute a limitation on the terminal device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0058] like Figure 1 As shown, the memory 1005, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and a streaming data monitoring program.

[0059] exist Figure 1 In the terminal shown, network interface 1004 is mainly used to connect to the backend server and communicate with it; user interface 1003 is mainly used to connect to the client and communicate with it; and processor 1001 can be used to call the streaming data monitoring program stored in memory 1005 and perform the following operations:

[0060] Obtain the first historical data output from the data source and calculate the first slope of the first historical data;

[0061] Calculate the first loss value between the first slope and each preset warning interval;

[0062] The minimum interval among the preset warning intervals corresponding to each second loss value is taken as the target warning interval, wherein the second loss value is the value of each first loss value that is less than a preset threshold.

[0063] The real-time streaming data output by the data source is monitored based on the target early warning interval.

[0064] Optionally, the processor 1001 can also be used to call the stream data monitoring program stored in the memory 1005, and perform the following operations:

[0065] Obtain multiple sets of second historical data output from the data source, and calculate the second slope of each set of second historical data;

[0066] Determine the data distribution type for each of the second slopes, and calculate the first distribution mean and first distribution standard deviation for each of the second slopes based on the data distribution type;

[0067] Each first interval is calculated based on the mean of the first distribution, the standard deviation of the first distribution, and the first preset formula;

[0068] Each of the first intervals is processed based on a preset smoothing period to obtain each of the warning intervals.

[0069] Optionally, the processor 1001 can also be used to call the stream data monitoring program stored in the memory 1005 and perform the following operations:

[0070] When the data distribution type is a standard normal distribution, each of the second slopes is substituted into the second preset formula to obtain the first distribution mean and the first distribution standard deviation of each of the second slopes;

[0071] When the data distribution type is a log-normal distribution, the logarithm of each of the second slopes is substituted into the second preset formula to obtain the mean and standard deviation of the first distribution;

[0072] When the data distribution type is a mixed normal distribution, each of the second slopes is input into the expectation maximization algorithm to obtain the mean and standard deviation of the first distribution.

[0073] Optionally, the processor 1001 can also be used to call the stream data monitoring program stored in the memory 1005 and perform the following operations:

[0074] The real-time streaming data output from the data source is preprocessed to obtain the target curve;

[0075] Detect whether the slope point on the target curve is within the target warning interval;

[0076] If the slope point is detected to be outside the target warning range, a warning message will be output.

[0077] Optionally, the processor 1001 can also be used to call the stream data monitoring program stored in the memory 1005 and perform the following operations:

[0078] The real-time streaming data output from the data source is grouped according to a preset duration to obtain each data group;

[0079] Calculate the third slope corresponding to each of the data groups, and fit an initial curve formed by the third slopes;

[0080] The initial curve is smoothed based on the number of smoothing periods to obtain the target curve.

[0081] Optionally, the processor 1001 can also be used to call the streaming data monitoring program stored in the memory 1005, and after the step of taking the smallest interval among the preset warning intervals corresponding to each second loss value as the target warning interval, it further performs the following operation:

[0082] Detect whether the topic of the real-time streaming data output by the data source is a newly added topic;

[0083] If it is detected that the topic of the real-time streaming data is not the newly added topic, then the step of monitoring the real-time streaming data output by the data source based on the target warning interval is executed;

[0084] If the topic of the real-time streaming data is detected to be the newly added topic, a new warning interval is determined based on multiple sets of third historical data under the newly added topic to monitor the real-time streaming data.

[0085] Optionally, the processor 1001 can also be used to call the stream data monitoring program stored in the memory 1005 and perform the following operations:

[0086] Calculate the fourth slope corresponding to each of the multiple sets of third historical data under the newly added topic;

[0087] Each second interval is calculated based on the second distribution mean and second distribution standard deviation of each of the fourth slopes;

[0088] The largest interval in each of the second intervals is taken as the new warning interval, and the real-time streaming data is monitored based on the new warning interval.

[0089] Based on the aforementioned terminal devices, various embodiments of the streaming data monitoring method of the present invention are proposed.

[0090] Please refer to Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of the streaming data monitoring method of the present invention. It should be noted that although the logical order is shown in the flowchart, in certain situations, the streaming data monitoring method of the present invention may execute the steps shown or described in a different order than that shown here.

[0091] In a first embodiment of the streaming data monitoring method of the present invention, the streaming data monitoring method includes:

[0092] Step S10: Obtain the first historical data output from the data source and calculate the first slope of the first historical data;

[0093] In this embodiment, the terminal device obtains historical data output from the current data source (hereinafter referred to as the first historical data for distinction), performs linear fitting on the first historical data, and calculates the slope of the first historical data (hereinafter referred to as the first slope for distinction).

[0094] It should be noted that the terminal device calculates the early warning rule parameters used for data monitoring once a day, that is, the early warning rule parameters are updated at a fixed time every day.

[0095] In one feasible implementation, if the early warning rule parameters for the next day are calculated at 24:00 every day, and the current time is 24:00 on August 8, 2023, then the terminal device obtains all the data output by the Kafka data source on the 8th, performs cluster fitting on the two-dimensional data of the data and time on the 8th, and obtains four different data categories. Specifically, the data category is used to represent the characteristics of the curve formed by the current data and time. The four data categories are unimodal curve, bimodal curve, multimodal curve, and flat curve. Each data category corresponds to multiple topics. The offset of each topic for the most recent 7 days is obtained from the Kafka platform, i.e., the first historical data mentioned above. The first historical data is divided into multiple groups of data according to a 5-minute duration. Each topic contains 12*24 groups of data per day. Linear fitting is performed on each group of data to calculate the slope of the group of data. A coordinate system is established with the time of the day as the horizontal axis and the slope of each group of data within the day as the vertical axis, as shown below. Figure 3 As shown, this is a diagram illustrating the data growth throughout the day. Figure 3 Each point in the graph represents the slope of a 5-minute data point within a day.

[0096] Step S20: Calculate the first loss value between the first slope and each preset warning interval;

[0097] In this embodiment, the terminal device calculates the loss value between the first slope and multiple preset warning intervals (hereinafter referred to as the first loss value for distinction).

[0098] It should be noted that based on the amount of data in a day, 12*24 slope points can be calculated. Each slope point corresponds to multiple preset warning intervals. For example, if the horizontal coordinate of the current slope point is from 24:00 to 00:05, which is the first 5 minutes of the day, and the vertical coordinate is 10, the preset warning interval corresponding to this slope point can be [0,20], [-10,30], etc.

[0099] In one feasible implementation, the terminal device obtains slope data for the most recent 7 days, substitutes the slope data and preset warning intervals into a preset loss function, and calculates the loss value between the slope data and each preset warning interval.

[0100] Step S30: Take the smallest interval among the preset warning intervals corresponding to each second loss value as the target warning interval, wherein the second loss value is the value of each first loss value that is less than a preset threshold.

[0101] In this embodiment, the terminal device takes the value of each first loss value that is less than a preset threshold as the second loss value, and then takes the smallest interval of the preset warning interval corresponding to each second loss value as the target warning interval.

[0102] It should be noted that the above-mentioned preset threshold is set to 5% in this invention. It should be understood that, based on different design needs of actual applications, the above-mentioned preset threshold can be any value that meets the actual needs in different feasible implementations. This invention does not limit the size of the above-mentioned preset threshold.

[0103] In one feasible implementation, the terminal device refers to the value less than 5% among a plurality of first loss values ​​as the second loss value, and determines the preset warning interval corresponding to the second loss value. For example, the loss value between the first slope and interval 1 is 2%, and the loss value between the first slope and interval 2 is 4%. These two loss values ​​are the aforementioned second loss values. If the range of interval 1 is greater than the range of interval 2, then interval 2 is taken as the target warning interval.

[0104] Step S40: Monitor the real-time streaming data output by the data source based on the target early warning interval.

[0105] In this embodiment, the terminal device monitors the real-time streaming data output by the data source based on the automatically generated target warning interval.

[0106] In one feasible implementation, the terminal device monitors the real-time streaming data output by the Kafka data source on August 9, 2023, based on the target warning interval calculated on August 8, 2023.

[0107] Furthermore, in a feasible embodiment, step S40 above may include:

[0108] Step S401: Preprocess the real-time streaming data output from the data source to obtain the target curve;

[0109] In this embodiment, the terminal device preprocesses the real-time streaming data output from the data source to obtain the target curve corresponding to the real-time streaming data.

[0110] Furthermore, in a feasible embodiment, step S401 above may include:

[0111] Step A10: Group the real-time streaming data output from the data source according to a preset duration to obtain each data group;

[0112] In this embodiment, the terminal device divides the real-time streaming data output from the data source into multiple data groups according to a preset duration.

[0113] In one feasible embodiment, the terminal device divides the real-time streaming data output by the Kafka data source on August 9, 2023, into multiple data groups in groups of 5 minutes each.

[0114] Step A20: Calculate the third slope corresponding to each of the data groups and fit the initial curve formed by the third slopes.

[0115] In this embodiment, the terminal device performs linear fitting on the data in each data group, calculates the slope corresponding to each data group (hereinafter referred to as the third slope for distinction), constructs a coordinate system with time as the horizontal axis and slope as the vertical axis, and fits multiple third slopes in this coordinate system to obtain an initial curve.

[0116] Step A30: Smooth the initial curve based on the smoothing period to obtain the target curve.

[0117] In this embodiment, the terminal device smooths the initial curve based on a preset number of smoothing periods to obtain the target curve.

[0118] In one feasible implementation, the terminal device smooths the peaks and troughs of the initial curve based on the smoothing period corresponding to the target warning interval to obtain the target curve.

[0119] Step S402: Detect whether the slope point on the target curve is within the target warning interval;

[0120] In this embodiment, the terminal device detects whether the slope point on the target curve is within the target warning range.

[0121] Step S403: If the slope point is detected to be outside the target warning range, a warning message will be output.

[0122] In this embodiment, the terminal device outputs a warning message when it detects that the slope point on the target curve is not within the target warning range.

[0123] In one feasible implementation, such as Figure 4As shown in the diagram, the streaming data monitoring process is as follows: First, the data acquisition layer records the offset of each Topic per minute from the Kafka platform, obtaining the total amount of data for each Topic per minute. A single Topic has 60*24 records per day. Then, an offline algorithm model is used to detect outliers based on linear fitting combined with multiple distribution models and moving averages. The raw data is as described above. Based on accumulated historical data, the feature extraction process adopts a segmented business period feature extraction method, refining and combining the raw data into a sequence containing multi-dimensional features. The real-time status data of the topic is input to the online alarm engine layer. Through the online real-time calculation core of the online alarm engine layer, the status data of the Topic is obtained in real time, and historical indicator data generated by the offline model is read from Redis / HBase. Alarm messages and evidence data are generated through multi-model integration. Finally, the alarm push layer includes a unified outreach service and push strategy. Specifically, the generated alarm information is pushed to end users by the unified outreach service through the Zhaohu robot, with the push strategy controlling the time period and frequency.

[0124] In this embodiment, the streaming data monitoring method of the present invention acquires the first historical data output from the current data source, performs linear fitting on the first historical data, and calculates the first slope of the first historical data; calculates the first loss value between the first slope and multiple preset warning intervals; takes the value less than a preset threshold among the first loss values ​​as the second loss value, and then takes the smallest interval among the preset warning intervals corresponding to each second loss value as the target warning interval; divides the real-time streaming data output from the data source according to a preset duration to obtain multiple data groups; performs linear fitting on the data in each data group, calculates the third slope corresponding to each data group, constructs a coordinate system with time as the horizontal axis and slope as the vertical axis, fits multiple third slopes in this coordinate system to obtain an initial curve; smooths the initial curve based on a preset smoothing period to obtain a target curve; detects whether the slope point on the target curve is within the target warning interval; and outputs warning information when the slope point on the target curve is detected to be outside the target warning interval.

[0125] Thus, this embodiment of the invention acquires historical data output from the current data source (i.e., the aforementioned first historical data), calculates the slope of the first historical data (i.e., the aforementioned first slope), then calculates the loss value between the first slope and multiple preset warning intervals (i.e., the aforementioned first loss value), takes the value less than a preset threshold among the multiple first loss values ​​as the second loss value, and takes the smallest interval among the preset warning intervals corresponding to each second loss value as the target warning interval. Finally, it monitors the real-time streaming data output from the data source based on the target warning interval. Therefore, compared to the traditional method that requires manual configuration of warning rule parameters, this invention automatically acquires historical data output from the current data source and automatically generates warning rule parameters for monitoring real-time streaming data, i.e., the aforementioned target warning interval, based on the slope of the historical data and the loss value between multiple preset warning intervals, thereby improving the intelligence of data monitoring.

[0126] Furthermore, based on the first embodiment of the streaming data monitoring method of the present invention described above, a second embodiment of the streaming data monitoring method of the present invention is proposed.

[0127] In this embodiment, the streaming data monitoring method of the present invention may further include:

[0128] Step B10: Obtain multiple sets of second historical data output by the data source, and calculate the second slope of each set of second historical data;

[0129] In this embodiment, the terminal device acquires historical data output from the current data source (hereinafter referred to as the second historical data for distinction), performs linear fitting processing on each group of second historical data, and calculates the slope corresponding to each group of second historical data (hereinafter referred to as the second slope for distinction).

[0130] In one feasible implementation, the terminal device obtains the data of the most recent 30 days from the Kafka platform, collects the aforementioned second historical data, and divides the 30 days of data into multiple 5-minute segments. Then, it calculates the slope of the data within each 5-minute segment, which is the aforementioned second slope.

[0131] Step B20: Determine the data distribution type for each of the second slopes, and calculate the first distribution mean and first distribution standard deviation for each of the second slopes based on the data distribution type;

[0132] In this embodiment, the terminal device determines the data distribution type of each second slope, and calculates the distribution mean (hereinafter referred to as the first distribution mean for distinction) and distribution standard deviation (hereinafter referred to as the first distribution standard deviation for distinction) of each second slope based on the data distribution type.

[0133] In a feasible real-time approach, the terminal device uses the slopes corresponding to the same time period (5 minutes long) each day for the past 30 days as a set of data. For example, the slopes corresponding to the data from 11:00 to 11:05 each day for the past 30 days, resulting in a set of 30 slope values. Then, the Kolmogorov-Smirnov test is used to determine whether the slopes in each time period conform to a standard normal distribution. If the set of data does not conform to a standard normal distribution, the logarithm of each slope value in the set of data is calculated to obtain 30 logarithms. The Kolmogorov-Smirnov test is then performed to determine whether the above 30 logarithms conform to a normal distribution. If they do, it is determined to be a log-normal distribution, i.e., a right-skewed distribution. If the set of data does not conform to either a standard normal distribution or a log-normal distribution, the mean, variance, and weight of each Gaussian component in the Gaussian mixture model are estimated by using an expectation-maximization (EM) algorithm.

[0134] Furthermore, in a feasible embodiment, step B20 above, the step of "calculating the first distribution mean and first distribution standard deviation of each second slope based on the data distribution type" may include:

[0135] Step B201: When the data distribution type is a standard normal distribution, substitute each of the second slopes into the second preset formula to obtain the first distribution mean and the first distribution standard deviation of each of the second slopes;

[0136] In this embodiment, when the data distribution type of each second slope is a standard normal distribution, the terminal device directly substitutes each second slope into a preset formula (hereinafter referred to as the second preset method for distinction) to obtain the first distribution mean and the first distribution standard deviation of each second slope.

[0137] It should be noted that the above preset formulas include the formulas for calculating the mean and the standard deviation. The formula for calculating the mean is: μ=(X1+X2+X3+...+X...) n The formula for calculating the standard deviation is: ) / n

[0138]

[0139] In one feasible implementation, when the terminal device determines that a set of slope data conforms to a standard normal distribution, it uses each slope as X. n The mean and standard deviation of the slope distribution are calculated using the above-mentioned mean formula and standard deviation formula.

[0140] Step B202: When the data distribution type is a log-normal distribution, substitute the logarithm of each of the second slopes into the second preset formula to obtain the mean and standard deviation of the first distribution.

[0141] In this embodiment, when the data distribution type of each second slope is a log-normal distribution, the terminal device directly substitutes the logarithm corresponding to each second slope into the second preset formula to obtain the first distribution mean and the first distribution standard deviation of each second slope.

[0142] In one feasible implementation, when the terminal device determines that a set of slope data conforms to a log-normal distribution, it uses the logarithm corresponding to each slope as X. n The mean and standard deviation of the slope distribution are calculated using the above-mentioned mean formula and standard deviation formula.

[0143] Step B203: When the data distribution type is a mixed normal distribution, input each of the second slopes into the expectation maximization algorithm to obtain the mean and standard deviation of the first distribution.

[0144] In this embodiment, when the data distribution type of each second slope is a mixed normal distribution, the terminal device inputs each second slope into the expectation maximization algorithm to obtain the first distribution mean and the first distribution standard deviation of each second slope.

[0145] In one feasible implementation, when the terminal device determines that a set of slope data conforms to a log-normal distribution, it inputs each slope into the EM (Expectation-maximization) algorithm, and uses each slope as X. n Suppose that all data Z consists of observable samples X = {X1, X2, ..., X...} n} and unobservable samples Z = {Z1, Z2, ..., Z} n If the data is composed of variables, then Y = X∪Z. The EM algorithm searches for the maximum likelihood estimate by maximizing the expected value of the likelihood function Log(L(Z;h)) for all data, where h is not a single variable but a set of parameters consisting of multiple variables. This expected value is calculated over the probability distribution followed by Z, which is determined by the unknown parameter h. However, the distribution followed by Z is unknown. The distribution of Z is estimated by replacing the actual parameter h with its current hypothesis h′.

[0146] Repeat the following two steps until convergence.

[0147] Step 1: Estimation (E) Step: Use the current hypothesis h and the observed data X to estimate the probability distribution on Y to compute Q(h′|h).

[0148] Q(h′|h)←E[In P(Y|h′)|h,X]

[0149] Step 2: Maximize (M) Step: Replace hypothesis h with hypothesis h′ that maximizes the Q function:

[0150] h←argmaxQ(h′|h)

[0151] Step B30: Calculate each first interval based on the first distribution mean, the first distribution standard deviation, and the first preset formula;

[0152] In this embodiment, the terminal device substitutes the first distribution mean and the first distribution standard deviation into a preset formula (hereinafter referred to as the first preset formula for distinction) to obtain each warning interval (hereinafter referred to as the first interval for distinction).

[0153] It should be noted that the first preset formula mentioned above includes μ+N*σ and μ-N*σ, and the warning intervals mentioned above include [μ-3*σ,μ+3*σ], [μ-4*σ,μ+4*σ], [μ-5*σ,μ+5*σ], [μ-6*σ,μ+6*σ], [μ-7*σ,μ+7*σ] and [μ-8*σ,μ+8*σ].

[0154] In one feasible implementation, such as Figure 5 As shown in the diagram of the warning intervals, the maximum and minimum values ​​corresponding to each 5-minute interval throughout the day are calculated, i.e., the maximum and minimum values ​​within each warning interval. The maximum values ​​for each 5-minute interval are fitted to one curve, and the minimum values ​​for each 5-minute interval are fitted to another curve. The range between these two curves is taken as the first interval. The formula for calculating the maximum value three times the standard deviation is μ - 3*σ, and the formula for calculating the minimum value three times the standard deviation is μ + 3*σ.

[0155] Step B40: Process each of the first intervals based on a preset smoothing period to obtain each of the warning intervals.

[0156] In this embodiment, the terminal device performs smoothing processing on each first interval based on a preset smoothing period to obtain each warning interval.

[0157] In one feasible implementation, such as Figure 6As shown in the diagram, the smoothing process is illustrated. Since the daily peaks and troughs of the data have a certain degree of shift, there is a risk of false alarms when the shift is significant. Therefore, an exponential moving average method is used to address this risk. By setting the smoothing period, exponential smoothing is performed for 6 or 12 periods to resolve the issue of peaks or troughs shifting within 30 minutes and 60 minutes. Furthermore, by statistically analyzing the shifts in historical peaks and troughs, the average shift period for each topic is estimated. Using the moving average method, the data becomes smoother.

[0158] In this embodiment, the streaming data monitoring method of the present invention acquires the second historical data output from the current data source, performs linear fitting processing on each group of second historical data, calculates the second slope corresponding to each group of second historical data, determines the data distribution type of each second slope, and, if the data distribution type of each second slope is a standard normal distribution, directly substitutes each second slope into a second preset method to obtain the first distribution mean and first distribution standard deviation of each second slope; if the data distribution type of each second slope is a log-normal distribution, directly substitutes the logarithm corresponding to each second slope into a second preset formula to obtain the first distribution mean and first distribution standard deviation of each second slope; if the data distribution type of each second slope is a mixed normal distribution, inputs each second slope into an expectation-maximization algorithm to obtain the first distribution mean and first distribution standard deviation of each second slope; substitutes the first distribution mean and first distribution standard deviation into a first preset formula to obtain each first interval; and smooths each first interval based on a preset smoothing period to obtain each warning interval.

[0159] Thus, this invention automatically acquires historical data for a certain period of time, groups the historical data according to fixed durations, calculates the slope of each group of data, and calculates the mean and standard deviation of each slope based on the distribution pattern of multiple slopes corresponding to the same time period of each day. Similarly, the mean and standard deviation of each time period throughout the day are obtained. It should be understood that if each slope value is regarded as a point, then the slope values ​​of the whole day can be connected to form a curve. The warning interval of each slope point is calculated based on the mean and standard deviation of each time period, thereby providing a selection basis for subsequent automatic adjustment of the warning interval, and smoothing the peaks and troughs on the curve corresponding to the warning interval to avoid the risk of false alarms.

[0160] Furthermore, based on the first and / or second embodiments of the streaming data monitoring method of the present invention described above, a third embodiment of the streaming data monitoring method of the present invention is proposed.

[0161] In this embodiment, the streaming data monitoring method of the present invention may further include, after step S30:

[0162] Step C10: Detect whether the topic of the real-time streaming data output by the data source is a newly added topic;

[0163] Step C20: If it is detected that the topic of the real-time streaming data is not the newly added topic, then the step of monitoring the real-time streaming data output by the data source based on the target early warning interval is executed.

[0164] Step C30: If the topic of the real-time streaming data is detected to be the newly added topic, then a new warning interval is determined based on multiple sets of third historical data under the newly added topic to monitor the real-time streaming data.

[0165] In this embodiment, the terminal device detects whether the topic of the real-time streaming data output by the current data source is a newly added topic in the current data source. If the topic of the real-time streaming data is not a newly added topic, the real-time streaming data is monitored based on the target warning interval. If the topic of the real-time streaming data is a newly added topic, a new warning interval is determined based on multiple sets of historical data under the newly added topic (hereinafter referred to as the third historical data for distinction) to monitor the real-time streaming data.

[0166] In one feasible implementation, since the determination of each preset warning interval requires relying on historical data of the most recent 30 days to achieve a good fit and judgment of the distribution, when a new Topic is added, in order to achieve a cold start, all available slope data are combined for normal fitting, the variance and standard deviation are calculated, and the outliers during the cold start period are estimated with 8 times the positive and negative standard deviations as the upper and lower limits.

[0167] Furthermore, in a feasible embodiment, step C30 above, the step of "determining a new early warning interval based on multiple sets of third historical data under the new topic to monitor the real-time streaming data" may include:

[0168] Step C301: Calculate the fourth slope corresponding to each of the multiple sets of third historical data under the newly added topic;

[0169] In this embodiment, the terminal device calculates the slope corresponding to each of the multiple sets of third historical data under the newly added topic (hereinafter referred to as the fourth slope for distinction).

[0170] In one feasible implementation, the terminal device acquires all historical data under the currently added topic, divides all historical data into multiple groups of data according to a 5-minute interval, namely the aforementioned third historical data, and then performs linear fitting on each group of data to calculate the slope corresponding to each group of data.

[0171] Step C302: Calculate each second interval based on the second distribution mean and second distribution standard deviation of each of the fourth slopes;

[0172] In this embodiment, the terminal device calculates each warning interval (hereinafter referred to as the second interval for distinction) based on the distribution mean (hereinafter referred to as the second distribution mean) and distribution standard deviation (hereinafter referred to as the second distribution standard deviation for distinction) of each fourth slope.

[0173] In one feasible implementation, the terminal device uses the Olmogorov-Smirnov test to determine the distribution type of all slope data under the newly added topic. The distribution type includes standard normal distribution, log-normal distribution, and mixed normal distribution. Then, based on the distribution type, the corresponding calculation method is selected to calculate the mean and standard deviation of all slope data to calculate the warning intervals corresponding to the current topic, i.e., the second interval.

[0174] Step C303: The largest interval in each of the second intervals is taken as the new warning interval, and the real-time streaming data is monitored based on the new warning interval.

[0175] In this embodiment, the terminal device uses the largest interval among the second intervals as the new warning interval and monitors the real-time streaming data output by the current data source based on the new warning interval.

[0176] In one feasible implementation, the terminal device determines the largest interval among multiple second intervals, namely [minimum value of 8 times standard deviation, maximum value of 8 times standard deviation], and uses the largest interval as a new early warning interval for monitoring the current real-time streaming data.

[0177] In this embodiment, the streaming data monitoring method of the present invention detects whether the topic of the real-time streaming data output by the current data source is a newly added topic in the current data source. If the topic of the real-time streaming data is not a newly added topic, the real-time streaming data is monitored based on the target warning interval. If the topic of the real-time streaming data is a newly added topic, the fourth slope corresponding to each of the multiple sets of third historical data under the newly added topic is calculated. Based on the second distribution mean and second distribution standard deviation of each fourth slope, each second interval is calculated. The largest interval among the second intervals is taken as the newly added warning interval, and the real-time streaming data output by the current data source is monitored based on the newly added warning interval.

[0178] Thus, when a new topic is added in Kafka, this invention performs normal fitting on all available slope data, calculates the variance and standard deviation, and uses 8 times the positive and negative standard deviations as upper and lower limits to estimate outliers during the cold start period, ensuring data security during the cold start.

[0179] In addition, embodiments of the present invention also provide a streaming data monitoring device.

[0180] Please refer to Figure 7 , Figure 7This is a functional module diagram of an embodiment of the streaming data monitoring device of the present invention, as shown below. Figure 7 As shown, the streaming data monitoring device of the present invention includes:

[0181] The first calculation module 10 is used to obtain the first historical data output by the data source and calculate the first slope of the first historical data.

[0182] The second calculation module 20 is used to calculate the first loss value between the first slope and each preset warning interval;

[0183] The interval determination module 30 is used to take the smallest interval among the preset warning intervals corresponding to each second loss value as the target warning interval, wherein the second loss value is the value of each first loss value that is less than a preset threshold.

[0184] The monitoring module 40 is used to monitor the real-time streaming data output by the data source based on the target early warning interval.

[0185] Optionally, the streaming data monitoring device of the present invention further includes:

[0186] The third calculation module is used to obtain multiple sets of second historical data output by the data source and calculate the second slope of each set of second historical data.

[0187] The fourth calculation module is used to determine the data distribution type of each second slope, and calculate the first distribution mean and first distribution standard deviation of each second slope based on the data distribution type;

[0188] The fifth calculation module is used to calculate each first interval based on the first distribution mean, the first distribution standard deviation and the first preset formula;

[0189] The smoothing module is used to process each of the first intervals based on a preset smoothing period to obtain each of the warning intervals.

[0190] Optionally, the fourth computing module includes:

[0191] The first calculation unit is used to substitute each of the second slopes into the second preset formula when the data distribution type is a standard normal distribution, to obtain the first distribution mean and the first distribution standard deviation of each of the second slopes;

[0192] The second calculation unit is used to substitute the logarithm of each of the second slopes into the second preset formula when the data distribution type is a log-normal distribution, to obtain the mean of the first distribution and the standard deviation of the first distribution.

[0193] The third calculation unit is used to input each of the second slopes into the expectation maximization algorithm to obtain the mean and standard deviation of the first distribution when the data distribution type is a mixed normal distribution.

[0194] Optionally, the monitoring module 40 includes:

[0195] The preprocessing unit is used to preprocess the real-time streaming data output from the data source to obtain the target curve;

[0196] The detection unit is used to detect whether the slope point on the target curve is within the target warning range;

[0197] The early warning unit is used to output early warning information if it detects that the slope point is not within the target early warning interval.

[0198] Optionally, the preprocessing unit includes:

[0199] The grouping subunit is used to group the real-time streaming data output by the data source according to a preset duration to obtain each data group;

[0200] A fitting subunit is used to calculate the third slope corresponding to each of the data groups and fit an initial curve formed by the third slopes.

[0201] The smoothing subunit is used to smooth the initial curve based on the smoothing period to obtain the target curve.

[0202] Optionally, the streaming data monitoring device of the present invention further includes:

[0203] The detection module is used to detect whether the topic of the real-time streaming data output by the data source is a newly added topic;

[0204] The first execution module is used to execute the step of monitoring the real-time stream data output by the data source based on the target warning interval if it is detected that the topic of the real-time stream data is not the newly added topic.

[0205] The second execution module is used to determine a new warning interval based on multiple sets of third historical data under the new topic if the topic of the real-time streaming data is detected to be the new topic, so as to monitor the real-time streaming data.

[0206] Optionally, the second execution module includes:

[0207] The fourth calculation unit is used to calculate the fourth slope corresponding to each of the multiple sets of third historical data under the newly added topic;

[0208] The fifth calculation unit is used to calculate each second interval based on the second distribution mean and second distribution standard deviation of each of the fourth slopes;

[0209] The monitoring unit is used to take the largest interval in each of the second intervals as a new warning interval, and monitor the real-time streaming data based on the new warning interval.

[0210] The present invention also provides a computer storage medium storing a streaming data monitoring program, wherein when the streaming data monitoring program is executed by a processor, it implements the steps of the streaming data monitoring program method as described in any of the above embodiments.

[0211] The specific embodiments of the computer storage medium of the present invention are basically the same as the embodiments of the streaming data monitoring program method of the present invention described above, and will not be repeated here.

[0212] The present invention also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the streaming data monitoring method of the present invention as described in any of the above embodiments, which will not be repeated here.

[0213] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0214] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0215] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (such as TWS earphones, etc.) to execute the methods described in the various embodiments of the present invention.

[0216] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A method for monitoring streaming data, characterized in that, The streaming data monitoring method includes: Obtain the first historical data output from the data source and calculate the first slope of the first historical data; Calculate the first loss value between the first slope and each preset warning interval; The minimum interval among the preset warning intervals corresponding to each second loss value is taken as the target warning interval, wherein the second loss value is the value of each first loss value that is less than a preset threshold. The real-time streaming data output by the data source is monitored based on the target early warning interval; The method further includes: Obtain multiple sets of second historical data output from the data source, and calculate the second slope of each set of second historical data; Determine the data distribution type for each of the second slopes, and calculate the first distribution mean and first distribution standard deviation for each of the second slopes based on the data distribution type; Each first interval is calculated based on the mean of the first distribution, the standard deviation of the first distribution, and the first preset formula; Each of the first intervals is processed based on a preset smoothing period to obtain each preset warning interval.

2. The streaming data monitoring method as described in claim 1, characterized in that, The step of calculating the first distribution mean and first distribution standard deviation of each second slope based on the data distribution type includes: When the data distribution type is a standard normal distribution, each of the second slopes is substituted into the second preset formula to obtain the first distribution mean and the first distribution standard deviation of each of the second slopes; When the data distribution type is a log-normal distribution, the logarithm of each of the second slopes is substituted into the second preset formula to obtain the mean and standard deviation of the first distribution; When the data distribution type is a mixed normal distribution, each of the second slopes is input into the expectation maximization algorithm to obtain the mean and standard deviation of the first distribution.

3. The streaming data monitoring method as described in claim 2, characterized in that, The step of monitoring the real-time streaming data output by the data source based on the target early warning interval includes: The real-time streaming data output from the data source is preprocessed to obtain the target curve; Detect whether the slope point on the target curve is within the target warning interval; If the slope point is detected to be outside the target warning range, a warning message will be output.

4. The streaming data monitoring method as described in claim 3, characterized in that, The step of preprocessing the real-time streaming data output from the data source to obtain the target curve includes: The real-time streaming data output from the data source is grouped according to a preset duration to obtain each data group; Calculate the third slope corresponding to each of the data groups, and fit an initial curve formed by the third slopes; The initial curve is smoothed based on the number of smoothing periods to obtain the target curve.

5. The streaming data monitoring method according to any one of claims 1 to 4, characterized in that, After the step of taking the smallest interval among the preset warning intervals corresponding to each second loss value as the target warning interval, the method further includes: Detect whether the topic of the real-time streaming data output by the data source is a newly added topic; If it is detected that the topic of the real-time streaming data is not the newly added topic, then the step of monitoring the real-time streaming data output by the data source based on the target warning interval is executed; If the topic of the real-time streaming data is detected to be the newly added topic, a new warning interval is determined based on multiple sets of third historical data under the newly added topic to monitor the real-time streaming data.

6. The streaming data monitoring method as described in claim 5, characterized in that, The step of determining a new early warning interval based on multiple sets of third historical data under the newly added topic to monitor the real-time streaming data includes: Calculate the fourth slope corresponding to each of the multiple sets of third historical data under the newly added topic; Each second interval is calculated based on the second distribution mean and second distribution standard deviation of each of the aforementioned fourth slopes; The largest interval in each of the second intervals is taken as the new warning interval, and the real-time streaming data is monitored based on the new warning interval.

7. A streaming data monitoring device, characterized in that, The streaming data monitoring device includes: The first calculation module is used to obtain the first historical data output by the data source and calculate the first slope of the first historical data. The second calculation module is used to calculate the first loss value between the first slope and each preset warning interval; The interval determination module is used to take the smallest interval among the preset warning intervals corresponding to each second loss value as the target warning interval, wherein the second loss value is the value of each first loss value that is less than a preset threshold. The monitoring module is used to monitor the real-time streaming data output by the data source based on the target early warning interval; The streaming data monitoring device further includes: The third calculation module is used to obtain multiple sets of second historical data output by the data source and calculate the second slope of each set of second historical data. The fourth calculation module is used to determine the data distribution type of each second slope, and calculate the first distribution mean and first distribution standard deviation of each second slope based on the data distribution type; The fifth calculation module is used to calculate each first interval based on the first distribution mean, the first distribution standard deviation and the first preset formula; The smoothing module is used to process each of the first intervals based on a preset smoothing period to obtain each of the preset warning intervals.

8. A terminal device, characterized in that, The terminal device includes: a memory, a processor, and a streaming data monitoring program stored in the memory and executable on the processor, wherein the streaming data monitoring program, when executed by the processor, implements the steps of the streaming data monitoring method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a streaming data monitoring program, which, when executed by a processor, implements the steps of the streaming data monitoring method as described in any one of claims 1 to 6.

Images